I’m interested in doing in-depth dialogues to find cruxes. Message me if you are interested in doing this.
I do alignment research, mostly stuff that is vaguely agent foundations. Currently doing independent alignment research on ontology identification. Formerly on Vivek’s team at MIRI. Most of my writing before mid 2023 is not representative of my current views about alignment difficulty.
would hopefully include many people who understand that understanding constraints is key and that past research understood some constraints.
Good point, I’m convinced by this.
build on past agent foundations research
I don’t really agree with this. Why do you say this?
That’s my guess at the level of engagement required to understand something. Maybe just because when I’ve tried to use or modify some research that I thought I understood, I always realise I didn’t understand it deeply enough. I’m probably anchoring too hard on my own experience here, other people often learn faster than me.
(Also I’m confused about the discourse in this thread (which is fine), because I thought we were discussing “how / how much should grantmakers let the money flow”.)
I was thinking “should grantmakers let the money flow to unknown young people who want a chance to prove themselves.”
I agree this would be a great program to run, but I want to call it a different lever to the one I was referring to.
The only thing I would change is that I think new researchers need to understand the purpose and value of past agent foundations research. I spent too long searching for novel ideas while I still misunderstood the main constraints of alignment. I expect you’d get a lot of wasted effort if you asked for out-of-paradigm ideas. Instead it might be better to ask for people to understand and build on past agent foundations research, then gradually move away if they see other pathways after having understood the constraints. Now I see my work as mostly about trying to run into constraints for the purpose of better understand them.
Maybe that wouldn’t help though, it’s really hard to make people see the constraints.
The main thing I’m referring to are upskilling or career transition grants, especially from LTFF, in the last couple of years. I don’t have stats, I’m assuming there were a lot given out because I met a lot of people who had received them. Probably there were a bunch given out by the ftx future fund also.
Also when I did MATS, many of us got grants post-MATS to continue our research. Relatively little seems to have come of these.
How are they falling short?
(I sound negative about these grants but I’m not, and I do want more stuff like that to happen. If I were grantmaking I’d probably give many more of some kinds of safety research grant. But “If a man has an idea just give him money and don’t ask questions” isn’t the right kind of change imo).
I think I disagree. This is a bandit problem, and grantmakers have tried pulling that lever a bunch of times. There hasn’t been any field-changing research (yet). They knew it had a low chance of success so it’s not a big update. But it is a small update.
Probably the optimal move isn’t cutting early-career support entirely, but having a higher bar seems correct. There are other levers that are worth trying, and we don’t have the resources to try every lever.
Also there are more grifters now that the word is out, so the EV is also declining that way.
(I feel bad saying this as someone who benefited a lot from early-career financial support).
My first exposure to rationalists was a Rationally Speaking episode where Julia recommended the movie Locke.
It’s about a man pursuing difficult goals under emotional stress using few tools. For me it was a great way to be introduced to rationalism because it showed how a ~rational actor could look very different from a straw Vulcan.
It’s also a great movie.
Nice.
Similar rule of thumb I find handy is 70 divided by growth rate to get doubling time implied by a growth rate. I find it way easier to think about doubling times than growth rates.
E.g. 3% interest rate means 70⁄3 ≈ 23 year doubling time.
I get the feeling that I’m still missing the point somehow and that Yudkowsky would say we still have a big chance of doom if our algorithms were created by hand with programmers whose algorithms always did exactly what they intended even when combined with their other algorithms.
I would bet against Eliezer being pessimistic about this, if we are assuming the algorithms are deeply-understood enough that we are confident that we can iterate on building AGI. I think there’s maybe a problem with the way Eliezer communicates that gives people the impression that he’s a rock with “DOOM” written on it.
I think the pessimism comes from there being several currently-unsolved problems that get in the way of “deeply-understood enough”. In principle it’s possible to understand these problems and hand-build a safe and stable AGI, it just looks a lot easier to hand-build an AGI without understanding them all, and even easier than that to train an AGI without even thinking about them.
I call most of these “instability” problems. Where the AI might for example learn more, or think more, or self-modify, and each of these can shift the context in a way that causes an imperfectly designed AI to pursue unintended goals.
Here are some descriptions of problems in that cluster: optimization daemons, ontology shifts, translating between our ontology and the AI’s internal ontology in a way that generalizes, pascal’s mugging, reflectively stable preferences & decision algorithms, reflectively stable corrigibility, and correctly estimating future competence under different circumstances.
Some may be resolved by default along the way to understanding how to build AGI by hand, but it isn’t clear. Some are kinda solved already in some contexts.
Intelligence/IQ is always good, but not a dealbreaker as long as you can substitute it with a larger population.
IMO this is pretty obviously wrong. There are some kinds of problem solving that scales poorly with population, just as there are some computations that scale poorly with parallelisation.
E.g. project euler problems.
When I said “problems we care about”, I was referring to a cluster of problems that very strongly appear to not scale well with population. Maybe this is an intuitive picture of the cluster of problems I’m referring to.
I buy that such an intervention is possible. But doing it requires understanding the internals at a deep level. You can’t expect SGD to implement the patch in a robust way. The patch would need to still be working after 6 months on an impossible problem, in spite of it actively getting in the way of finding the solution!
I’d be curious about why it isn’t changing the picture quite a lot, maybe after you’ve chewed on the ideas. From my perspective it makes the entire non-reflective-AI-via-training pathway not worth pursuing. At least for large scale thinking.
I was probably influenced by your ideas! I just (re?)read your post on the topic.
Tbh I think it’s unlikely such a sweet spot exists, and I find your example unconvincing. The value of this kind of reflection for difficult problem solving directly conflicts with the “useful” assumption.
I’d be more convinced if you described the task where you expect an AI to be useful (significantly above current humans), and doesn’t involve failing and reevaluating high-level strategy every now and then.
Extremely underrated post, I’m sorry I only skimmed it when it came out.
I found 3a,b,c to be strong and well written, a good representation of my view.
In contrast, 3d I found to be a weak argument that I didn’t identify with. In particular, I don’t think internal conflicts are a good way to explain the source of goal misgeneralization. To me it’s better described as just overfitting or misgeneralization.[1] Edge cases in goals are clearly going to be explored by a stepping back process, if initial attempts fail. In particular if attempted pathways continue to fail. Whereas thinking of the AI as needing to resolve conflicting values seems to me to be anthropomorphizing in a way that doesn’t seem to transfer to most mind designs.
You also used the word coherent in a way that I didn’t understand.
Human intelligence seems easily useful enough to be a major research accelerator if it can be produced cheaply by AI
I want to flag this as an assumption that isn’t obvious. If this were true for the problems we care about, we could solve them by employing a lot of humans.
humans provides a pretty strong intuitive counterexample
It’s a good observation that humans seem better at stepping back inside of low-level tasks than at high-level life-purposes. For example, I got stuck on a default path of finishing a neuroscience degree, even though if I had reflected properly I would have realised it was useless for achieving my goals a couple of years earlier. I got got by sunk costs and normality.
However, I think this counterexample isn’t as strong as you think it is. Firstly because it’s incredibly common for people to break out of a default-path. And secondly because stepping back is usually proceeded by some kind of failure to achieve the goal using a particular approach. Such failures occur often at small scales. They occur infrequently in most people’s high-level life plans, because such plans are fairly easy and don’t often raise flags that indicate potential failure. We want difficult work out of an AI. This implies frequent total failure, and hence frequent high-level stepping back. If it’s doing alignment research, this is particularly true.
Like for reasons given in section 4 of the misalignment and catastrophe doc.
Trying to write a new steelman of Matt’s view. It’s probably incorrect, but seems good to post as a measure of progress:
You believe in agentic capabilities generalizing, but also in additional high-level patterns that generalize and often overpower agentic behaviour. You expect training to learn all the algorithms required for intelligence, but also pick up patterns in the data like “research style”, maybe “personality”, maybe “things a person wouldn’t do” and also build those into the various-algorithms-that-add-up-to-intelligence at a deep level. In particular, these patterns might capture something like “unwillingness to commandeer some extra compute” even though it’s easy and important and hasn’t been explicitly trained against. These higher level patterns influence generalization more than agentic patterns do, even though this reduces capability a bit.
One component of your model that reinforces this: Realistic intelligence algorithms rely heavily on something like caching training data and this has strong implications about how we should expect them to generalize. This gives an inductive-bias advantage to the patterns you mention, and a disadvantage to think-it-through-properly algorithms (like brute force search, or even human-like thinking).
We didn’t quite get to talking about reflection, but this is probably the biggest hurdle in the way of getting such properties to stick around. I’ll guess at your response: You think that an intelligence that doesn’t-reflect-very-much is reasonably simple. Given this, we can train chain-of-thought type algorithms to avoid reflection using examples of not-reflecting-even-when-obvious-and-useful. With some effort on this, reflection could be crushed with some small-ish capability penalty, but massive benefits for safety.
Not much to add, I haven’t spent enough time thinking about structural selection theorems.
I’m a fan of making more assumptions. I’ve had a number of conversations with people who seem to make the mistake of not assuming enough. Sometimes leading them to incorrectly consider various things impossible. E.g. “How could an agent store a utility function over all possible worlds?” or “Rice’s theorem/halting problem/incompleteness/NP-hardness/no-free-lunch theorems means it’s impossible to do xyz”. The answer is always nah, it’s possible, we just need to take advantage of some structure in the problem.
Finding the right assumptions is really hard though, it’s easy to oversimplify the problem and end up with something useless.
Good point.
What I meant by updatelessness removes most of the justification is the reason given here at the very beginning of “Against Resolute Choice”. In order to make a money pump that leads the agent in a circle, the agent has to continue accepting trades around a full preference loop. But if it has decided on the entire plan beforehand, it will just do any plan that involves <1 trip around the preference loop. (Although it’s unclear how it would settle on such a plan, maybe just stopping its search after a given time). It won’t (I think?) choose any plan that does multiple loops, because they are strictly worse.
After choosing this plan though, I think it is representable as VNM rational, as you say. And I’m not sure what to do with this. It does seem important.
However, I think Scott’s argument here satisfies (a) (b) and (c). I think the independence axiom might be special in this respect, because the money pump for independence is exploiting an update on new information.
I think the problem might be that you’ve given this definition of heuristic:
A heuristic is a local, interpretable, and simple function (e.g., boolean/arithmetic/lookup functions) learned from the training data. There are multiple heuristics in each layer and their outputs are used in later layers.
Taking this definition seriously, it’s easy to decompose a forward pass into such functions.
But you have a much more detailed idea of a heuristic in mind. You’ve pointed toward some properties this might have in your point (2), but haven’t put it into specific words.
Some options: A single heuristic is causally dependent on <5 heuristics below and influences <5 heuristics above. The inputs and outputs of heuristics are strong information bottlenecks with a limit of 30 bits. The function of a heuristic can be understood without reference to >4 other heuristics in the same layer. A single heuristic is used in <5 different ways across the data distribution. A model is made up of <50 layers of heuristics. Large arrays of parallel heuristics often output information of the same type.
Some combination of these (or similar properties) would turn the heuristics intuition into a real hypothesis capable of making predictions.
If you don’t go into this level of detail, it’s easy to trick yourself into thinking that (2) basically kinda follows from your definition of heuristics, when it really really doesn’t. And that will lead you to never discover the value of the heuristics intuition, if it is true, and never reject it if it is false.
I’ve only skimmed this post, but I like it because I think it puts into words a fairly common model (that I disagree with). I’ve heard “it’s all just a stack of heuristics” as an explanation of neural networks and as a claim that all intelligence is this, from several people. (Probably I’m overinterpreting other people’s words to some extent, they probably meant a weaker/nuanced version. But like you say, it can be useful to talk about the strong version).
I think you’ve correctly identified the flaw in this idea (it isn’t predictive, it’s unfalsifiable, so it isn’t actually explaining anything even if it feels like it is). You don’t seem to think this is a fatal flaw. Why?
You seem to answer
However, the key interpretability-related claim is that heuristics based decompositions will be human-understandable, which is a more falsifiable claim.
But I don’t see why “heuristics based decompositions will be human-understandable” is an implication of the theory. As an extreme counterexample, logic gates are interpretable, but when stacked up into a computer they are ~uninterpretable. It looks to me like you’ve just tacked an interpretability hypothesis onto a heuristics hypothesis.
Trying to think this through, I’ll write a bit of a braindump just in case that’s useful:
The futachy hack can be split into two parts. The first is that is that conditioning on untaken actions makes most probabilities ill-defined. Because there are no incentives to get it right, the market can can settle to many equilibria. The second part is that there are various incentives for traders to take advantage of this for their own interests.
With your technique, I think approach would be to duplicate each trader into two traders with the same knowledge, and make their joint earnings zero sum.[1]
This removes one explicit incentive for a single trader to manipulate a value to cause a different action to happen. But only if it’s doing so to make the distribution easier to predict and thereby improving their score. Potentially there are still other incentives i.e. if the trader has preferences over the world, and these aren’t eliminated.
Why doesn’t this happen in LI already? LI is zero sum overall, because there is a finite pool of wealth. But this is shared among traders with different knowledge. If there is a wealthiest trader that has a particular piece of knowledge, it should manipulate actions to reduce variance to get a higher score. So the problem is that it’s not zero-sum with respect to each piece of knowledge.
But, the first issue is entirely unresolved. The probabilities that condition on untaken actions will be path-dependent leftovers from the convergence procedure of LI, when the market was more uncertain about which action will be taken. I’d expect these to be fairly reasonable, but they don’t have to be.
This reasonableness is coming from something though, and maybe this can be formalized.
You’d have to build a lot more structure into the LI traders to guarantee they can’t learn to cooperate and are myopic. But that seems doable. And its the sort of thing I’d want to do anyway.
These seem right, but more importantly I think it would eliminate investing in new scalable companies. Or dramatically reduce it in the 50% case. So there would be very few new companies created.
(As a side note: Maybe our response to this proposal was a bit cruel. It might have been better to just point toward some econ reading material).