I liked most thoughts in this post even though I have quite opposite intutions about timelines.
I agree timeline intuitions feel largely vibes based, so I struggle to form plausible guesses about the exact source behind our different intuitions.
I thought this passage was interesting in that respect:
Or, maybe there will be a sudden jump. Maybe learning sequential reasoning is a single trick, and now we can get from 4 to 1000 in two more years.
What you write as an afterthought is what I would have thought immediately. Sure, good chance I’m wrong. Still, to me, it feels like there’s probably no big difference between “applying reasoning for 3 steps” and “applying reasoning for 1,000 steps.” Admittedly, (using your terminology,) you probably need decent philosophical intelligence to organize your 1,000 steps with enough efficiency and sanity checks along the way so you don’t run the risk of doing tons of useless work. But I don’t see why that’s a thing that cannot also be trained, now that we have the chain of thought stuff and can compare, over many answering runs, which thought decompositions more reliably generate accurate answers. Humans must have gotten this ability from somewhere and it’s unlikely the brain has tons of specialized architecture for it. The thought generator seems more impressive/fancy/magic-like to me. Philosophical intelligence feels like something that heavily builds on a good thought generator plus assessor, rather than being entirely its own thing. (It also feels a bit subjective—I wouldn’t be surprised if there’s no single best configuration of brain parameters for making associative leaps through thought space, such that different reasoners will be better at different problem domains. Obviously smart reasoners should then pick up where they’re themselves experts and where they should better defer.)
Your blue and red graphs made me think about how AI appears to be in a thought generator overhang, so any thought assessor progress should translate into particularly steep capability improvements. If sequential reasoning is mostly a single trick, things should get pretty fast now. We’ll see soon? :S
Humans must have gotten this ability from somewhere and it’s unlikely the brain has tons of specialized architecture for it.
This is probably a crux; I think the brain does have tons of specialized architecture for it, and if I didn’t believe that, I probably wouldn’t think thought assessment was as difficult.
The thought generator seems more impressive/fancy/magic-like to me.
Notably people’s intuitions about what is impressive/difficult tend to be inversely correlated with reality. The stereotype is (or at least used to be) that AI will be good at rationality and reasoning but struggle with creativity, humor, and intuition. This stereotype contains information since inverting it makes better-than-chance predictions about what AI has been good at so far, especially LLMs.
I think this is not a coincidence but roughly because people use “degree of conscious access” an inverse proxy for intuitive difficulty. The more unconscious something is, the more it feels like we don’t know how it works, the more difficult it intuitively seems. But I suspect degree of conscious access positively correlates with difficulty.
If sequential reasoning is mostly a single trick, things should get pretty fast now. We’ll see soon? :S
Yes; I think the “single trick” view might be mostly confirmed or falsified in as little as 2-3 years. (If I introspect I’m pretty confident that I’m not wrong here, the scenario that frightens me is more that sequential reasoning improves non-exponentially but quickly, which I think could still mean doom, even if it takes 15 years. Those feel like short timelines to me.)
This is probably a crux; I think the brain does have tons of specialized architecture for it, and if I didn’t believe that, I probably wouldn’t think thought assessment was as difficult.
I think this is also a crux.
IMO, I think the brain is mostly cortically uniform, ala Steven Byrnes, and in particular I think that the specialized architecture for thought assessment was pretty minimal.
The big driver of human success is basically something like the bitter lesson applied to biological brains, combined with humans being very well optimized for tool use, such that they can over time develop technology that is used to dominate the world (it’s also helpful that humans can cooperate reasonably below 100 people, which is more than almost all social groups, though I’ve become much more convinced that cultural learning is way less powerful than Henrich et al have said).
(There are papers which show that humans are better at scaling neurons than basically everyone else, but I can’t find them right now).
I liked most thoughts in this post even though I have quite opposite intutions about timelines.
I agree timeline intuitions feel largely vibes based, so I struggle to form plausible guesses about the exact source behind our different intuitions.
I thought this passage was interesting in that respect:
What you write as an afterthought is what I would have thought immediately. Sure, good chance I’m wrong. Still, to me, it feels like there’s probably no big difference between “applying reasoning for 3 steps” and “applying reasoning for 1,000 steps.” Admittedly, (using your terminology,) you probably need decent philosophical intelligence to organize your 1,000 steps with enough efficiency and sanity checks along the way so you don’t run the risk of doing tons of useless work. But I don’t see why that’s a thing that cannot also be trained, now that we have the chain of thought stuff and can compare, over many answering runs, which thought decompositions more reliably generate accurate answers. Humans must have gotten this ability from somewhere and it’s unlikely the brain has tons of specialized architecture for it. The thought generator seems more impressive/fancy/magic-like to me. Philosophical intelligence feels like something that heavily builds on a good thought generator plus assessor, rather than being entirely its own thing. (It also feels a bit subjective—I wouldn’t be surprised if there’s no single best configuration of brain parameters for making associative leaps through thought space, such that different reasoners will be better at different problem domains. Obviously smart reasoners should then pick up where they’re themselves experts and where they should better defer.)
Your blue and red graphs made me think about how AI appears to be in a thought generator overhang, so any thought assessor progress should translate into particularly steep capability improvements. If sequential reasoning is mostly a single trick, things should get pretty fast now. We’ll see soon? :S
This is probably a crux; I think the brain does have tons of specialized architecture for it, and if I didn’t believe that, I probably wouldn’t think thought assessment was as difficult.
Notably people’s intuitions about what is impressive/difficult tend to be inversely correlated with reality. The stereotype is (or at least used to be) that AI will be good at rationality and reasoning but struggle with creativity, humor, and intuition. This stereotype contains information since inverting it makes better-than-chance predictions about what AI has been good at so far, especially LLMs.
I think this is not a coincidence but roughly because people use “degree of conscious access” an inverse proxy for intuitive difficulty. The more unconscious something is, the more it feels like we don’t know how it works, the more difficult it intuitively seems. But I suspect degree of conscious access positively correlates with difficulty.
Yes; I think the “single trick” view might be mostly confirmed or falsified in as little as 2-3 years. (If I introspect I’m pretty confident that I’m not wrong here, the scenario that frightens me is more that sequential reasoning improves non-exponentially but quickly, which I think could still mean doom, even if it takes 15 years. Those feel like short timelines to me.)
I think this is also a crux.
IMO, I think the brain is mostly cortically uniform, ala Steven Byrnes, and in particular I think that the specialized architecture for thought assessment was pretty minimal.
The big driver of human success is basically something like the bitter lesson applied to biological brains, combined with humans being very well optimized for tool use, such that they can over time develop technology that is used to dominate the world (it’s also helpful that humans can cooperate reasonably below 100 people, which is more than almost all social groups, though I’ve become much more convinced that cultural learning is way less powerful than Henrich et al have said).
(There are papers which show that humans are better at scaling neurons than basically everyone else, but I can’t find them right now).