Anyone want to guess how capable Claude system level 2 will be when it is polished? I expect better than o3 by a small amt.
RussellThor
Karma: 463
Yes the human brain was built using evolution, I have no disagreement that give us 100-1000 years with just tinkering etc we would likely get AGI. Its just that in our specific case we have bio to copy and it will get us there much faster.
Types of takeoff
When I first heard and thought about AI takeoff I found the argument convincing that as soon as an AI passed IQ 100, takeoff would become hyper exponentially fast. Progress would speed up, which would then compound on itself etc. However there other possibilities.
AGI is a barrier that requires >200 IQ to pass unless we copy biology?
Progress could be discontinuous, there could be IQ thresholds required to unlock better methods or architectures. Say we fixed our current compute capability, and with fixed human intelligence we may not be able to figure out the formula for AGI, in a similar way that the combined human intelligence hasn’t cracked many hard problems even with decades and the worlds smartest minds working on them (maths problems, Quantum gravity...). This may seem unlikely for AI, but to illustrate the principle, say we only allowed IQ<90 people to work on AI. Progress would stall. So IQ <90 software developers couldn’t unlock IQ>90 AI. Can IQ 160 developers with our current compute hardware unlock >160 AI?
To me the reason we don’t have AI now is that the architecture is very data inefficient and worse at generalization than say the mammalian brain, for example a cortical column. I expect that if we knew the neural code and could copy it, then we would get at least to very high human intelligence quickly as we have the compute.
From watching AI over my career it seems to be that even the highest IQ people and groups cant make progress by themselves without data, compute and biology to copy for guidance, in contrast to other fields. For example Einstein predicted gravitational waves long before they where discovered, but Turing or Von Neumann didn’t publish the Transformer architecture or suggest backpropagation. If we did not have access to neural tissue, would we still not have artificial NN? In a related note, I think there is an XKCD cartoon that says something like the brain has to be so complex that it cannot understand itself.
(I believe now that progress in theoretical physics and pure maths is slowing to a stall as further progress requires intellectual capacity beyond the combined ability of humanity. Without AI there will be no major advances in physics anymore even with ~100 years spent on it.)
After AGI is there another threshold?
Lets say we do copy biology/solve AGI and with our current hardware can get >10,000 AGI agents with >= IQ of the smartest humans. They then optimize the code so there is 100K agents with the same resources. but then optimization stalls. The AI wouldn’t know if it was because it had optimized as much as possible, or because it lacked the ability to find a better optimization.
Does our current system scale to AGI with 1GW/1 million GPU?
Lets say we don’t copy biology, but scaling our current systems to 1GW/1 million GPU and optimizing for a few years gets us to IQ 160 at all tasks. We would have an inferior architecture compensated by a massive increase in energy/FLOPS as compared to the human brain. Progress could theoretically stall at upper level human IQ for a time rather then takeoff. (I think this isn’t very likely however) There would of course be a significant overhang where capabilities would increase suddenly when the better architecture was found and applied to the data center hosting the AI.
Related note—why 1GW data centers won’t be a consistent requirement for AI leadership.
Based on this, then a 1GW or similar data center isn’t useful or necessary for long. If it doesn’t give a significant increase in capabilities, then it won’t be cost effective. If it does, then it would optimize itself so that such power isn’t needed anymore. Only in a small range of capability increase does it actually stay around.
To me its not clear the merits of the Pause movement and training compute caps. Someone here made the case that compute caps could actually speed up AGI as people would then pay more attention to finding better architectures rather than throwing resources into scaling existing inferior ones. However all things considered I can see a lot of downsides from large data centers and little upside. I see a specific possibility where they are build, don’t give the economic justification, decrease in value a lot, then are sold to owners that are not into cutting edge AI. Then when the more efficient architecture is discovered, they are suddenly very powerful without preparation. Worldwide caps on total GPU production would also help reduce similar overhang possibilities.
I am also not impressed with the pause AI movement and am concerned about AI safety. To me focusing on AI companies and training FLOPS is not the best way to do things. Caps on data center sizes and worldwide GPU production caps would make more sense to me. Pausing software but not hardware gives more time for alignment but makes a worse hardware overhang. I don’t think thats helpful. Also they focus too much on OpenAI from what I’ve seen. xAI will soon have the largest training center for a start.
I don’t think this is right or workable https://pauseai.info/proposal—figure out how biological intelligence learns and you don’t need a large training run. There’s no guarantee at all that a pause at this stage can help align super AI. I think we need greater capabilities to know what we are dealing with. Even with a 50 year pause to study GPT4 type models I wouldn’t be confident we could learn enough from that. They have no realistic way to lift the pause, so its a desire to stop AI indefinitely.
“There will come a point where potentially superintelligent AI models can be trained for a few thousand dollars or less, perhaps even on consumer hardware. We need to be prepared for this.”
You can’t prepare for this without first having superintelligent models running on the most capable facilities then having already gone through a positive Singularity. They have no workable plan for achieving a positive Singularity, just try to stop and hope.
OK fair point. If we are going to use analogies, then my point #2 about a specific neural code shows our different positions I think.
Lets say we are trying to get a simple aircraft of the ground and we have detailed instructions for a large passenger jet. Our problem is that the metal is too weak and cannot be used to make wings, engines etc. In that case detailed plans for aircraft are no use, a single minded focus on getting better metal is what its all about. To me the neural code is like the metal and all the neuroscience is like the plane schematics. Note that I am wary of analogies—you obviously don’t see things like that or you wouldn’t have the position you do. Analogies can explain, but rarely persuade.
A more single minded focus on the neural code would be trying to watch neural connections form in real time while learning is happening. Fixed connectome scans of say mice can somewhat help with that, more direct control of dishbrain, watching the zebra fish brain would all count, however the details of neural biology that are specific to higher mammals would be ignored.
Its possible also that there is a hybrid process, that is the AI looks at all the ideas in the literature then suggests bio experiments to get things over the line.
Yes you have a point.
I believe that building massive data centers are the biggest risk atm and in the near future. I don’t think open AI/Anthropic will get to AGI, but rather someone copying biology will. In that case probably the bigger the datacenter around when that happens, the bigger the risk. For example a 1million GPU with current tech doesn’t get super AI, but when we figure out the architecture, it suddenly becomes much more capable and dangerous. That is from IQ 100 up to 300 with a large overhang. If the data center was smaller, then the overhang is smaller. The scenario I have in mind is someone figures AGI out, then one way or another the secret gets adopted suddenly by the large data center.
For that reason I believe focus on FLOPS for training runs is misguided, its hardware concentration and yearly worldwide HW production capacity that is more important.
Perhaps LLM will help with that. The reason I think that is less likely is
Deep mind etc is already heavily across biology from what I gather from interview with Demis. If the knowledge was there already there’s a good chance they would have found it
Its something specific we are after, not many small improvements, i.e. the neural code. Specifically back propagation is not how neurons learn. I’m pretty sure how they actually do is not in the literature. Attempts have been made such as the forward-forward algorithm by Hinton, but that didn’t come to anything as far as i can tell. I havn’t seen any suggestion that even with too much detail on biology we know what it is. i.e. can a very detailed neural sim with extreme processing power learn as data efficiently as biology?
If progress must come from a large jump rather than small steps, then LLM have quite a long way to go, i.e. LLM need to speed up coming up ideas as novel as the forward-forward algo to help much. If they are still below that threshold in 2026 then those possible insights are still almost entirely done by people.
Even the smartest minds in the past have been beaten by copying biology in AI. The idea for neural nets came from copying biology. (Though the transformer arch and back prop didn’t)
I think it is clear that if say you had a complete connectome scan and knew everything about how a chimp brain worked you could scale it easily to get human+ intelligence. There are no major differences. Small mammal is my best guess, mammals/birds seem to be able to learn better than say lizards. Specifically the https://en.wikipedia.org/wiki/Cortical_column is important to understand, once you fully understand one, stacking them will scale at least somewhat well.
Going to smaller scales/numbers of neurons, it may not need to be as much as a mammal, https://cosmosmagazine.com/technology/dishbrain-pong-brain-on-chip-startup/, perhaps we can learn enough of the secrets here? I expect not, but only weakly confident.
Going even simpler, we have the connectome scan of a fly now, https://flyconnecto.me/ and that hasn’t led to major AI advances. So its somewhere between fly/chimp I’d guess mouse that gives us the missing insight to get TAI
Putting down a prediction I have had for quite some time.
The current LLM/Transformer architecture will stagnate before AGI/TAI (That is the ability to do any cognitive task as effectively and cheaper than a human)From what I have seen, Tesla autopilot learns >10,000 slower than a human datawise.
We will get AGI by copying nature, at the scale of a simple mammal brain, then scaling up, like this kind of project:
https://x.com/Andrew_C_Payne/status/1863957226010144791
https://e11.bio/news/roadmap
I expect AGI to be 0-2 years after a mammal brain is mapped. In terms of cost-effectiveness I consider such a connectome project to be far more cost effective per $ than large training runs or building a 1GW data center etc if you goal is to achieve AGI.That is TAI by about 2032 assuming 5 years to scan a mammal brain. In this case there could be a few years when Moores law has effectively stopped, larger data centers are not being built and it is not clear where progress will come from.
In related puzzles I did hear something a while ago now, Bostrom perhaps. You have say 6 challenging events to achieve to get from no life to us. They are random and some of those steps are MUCH harder than the others, but if you look at the successful runs, you cant in hindsight see what they are. For life its say no life to life, simple single cell to complex cell and perhaps 4 other events that aren’t so rare.
A run is a sequence of 100 steps where you either don’t achieve the end state (all 6 challenging events achieved in order, or you do)
There is a 1 in a million chance that a run is successful.
Now if you look at the successful runs, you cant then in hindsight see what events were really hard and which weren’t. The event with 1⁄10,000 chance at each step may have taken just 5 steps in the successful run, it couldn’t take 10,000 steps because only 100 are allowed etc.
A good way to resolve the paradox to me is to modify the code to combine both the functions into one function and record the sequences of the 10,000, In one array you store the sequences where there are two consecutive 6′s and in the second you store the one where they are not consecutive. That makes it a bit clearer.
For a run of 10,000 I get 412 runs where the first two 6′s are consecutive (sequences_no_gap), and 192 where they are not (sequences_can_gap). So if its just case A you get 412 runs, but for case B you get 412+192 runs. Then you look at the average sequence length of sequences_no_gap and compare it to sequences_can_gap. If the average sequence length in sequences_can_gap > than sequences_no_gap, then that means the expectation will be higher, and thats what you get.
mean sequence lengths
sequences_can_gap: 3.93
sequences_no_gap: 2.49
Examples:
sequences_no_gap
[[4, 6, 6], [6, 6], [6, 6], [4, 6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [6, 6], [4, 6, 6], [4, 6, 6], …]
sequences_can_gap
[[6, 4, 4, 6], [6, 4, 6], [4, 6, 4, 6], [2, 2, 6, 2, 6], [6, 4, 2, 6], [6, 4, 6], [6, 2, 6], [6, 2, 4, 6], [6, 2, 2, 4, 6], [6, 4, 6], [6, 4, 6], [2, 4, 6, 2, 6], [6, 4, 6], [6, 4, 6], …]
The many examples such as [6 4 4 6] which are excluded in the first case make the expected number of rolls higher for the case where they are allowed.
(Note GPT o-1 is confused by this problem and gives slop)
I read the book, it was interesting, however a few points.
Rather than making the case, it was more a plea for someone else to make the case. It didn’t replace the conventional theory with its own one, it was far too short and lacking on specifics for that. If you throw away everything, you then need to recreate all our knowledge from your starting point and also explain how what we have still works so well.
He was selective about quantum physics—e.g. if reality is only there when you observe, then the last thing you would expect is a quantum computer to exist and have the power to do things outside what is possible with conventional computers. MWI predicts this much better if you can correlate an almost infinite amt of world lines to do your computation for you. Superposition/entanglement should just be lack of knowledge rather than part of an awesome computation system.
He claims that consciousness is fundamental, but then assumes Maths is also. So which is fundamental? You cant have both.
If we take his viewpoint then we can still derive principles, such as the small/many determine the big. e.g. if you get the small wrong (chemical imbalance in the brain) then the big (macro behavior) is wrong. It doesn’t go the other way—you can’t IQ away your Alzheimer’s. So its just not clear even if you try to fully accept his point of view what how you should even take things.
In a game theoretic framework we might say that the payoff matrices for the birds and bees are different, so of course we’d expect them to adopt different strategies.
Yes somewhat, however it would still be best for all birds if they had a better collective defense. In a swarming attack, none would have to sacrifice their life so its unconditionally better for both the individual and the collective. I agree that inclusive fitness is pretty hard to control for, however perhaps you can only get higher inclusive fitness the simpler you go? e.g. all your cells have exactly the same DNA, ants are very similar, birds are more different. The causation could be simpler/less intelligent organisms → more inclusive fitness possible/likely → some cooperation strategies opened up.
Cool, that was my intuition. GPT was absolutely sure in the golf ball analogy however that it couldn’t happen. That is the ball wouldn’t “reflect” off the low friction surface. Tempted to try and test somehow
Yes that does sound better, and is there an equivalent to total internal refraction where the wheels are pushed back up the slope?
Another analogy is with a ball rolling on two surfaces crossing the boundary. The first very little friction, then second a bit more.
From AI:
“The direction in which the ball veers when moving from a smooth to a rough surface depends on several factors, especially the initial direction of motion and the orientation of the boundary between the two surfaces. Here’s a general outline of how it might behave:
If Moving at an Angle to the Boundary:
Suppose the ball moves diagonally across the boundary between the smooth and rough surfaces (i.e., it doesn’t cross perpendicularly).
When it hits the rough surface, frictional resistance increases more on the component of motion along the boundary line than on the perpendicular component.
This causes the ball to veer slightly toward the rougher surface, meaning it will change direction in a way that aligns more closely with the boundary.
This is similar to a light ray entering water. So is the physics the same? (on second reading, its not so clear, if you put a golf ball from a smooth surface to a rough one, what happens to the angle at the boundary?)
Well in this case, the momentum of the ball clearly won’t increase, instead it will be constantly losing momentum and if the second surface was floating it would be pushed so as to conserve momentum. Unlike for light however if it then re-enters the smooth surface it will be going slower. It seems the ball would lose momentum at both transition boundary. (however if the rough surface was perfectly floating, then perhaps it would regain it)
Anyway for a rough surface that is perfectly floating, it seems the ball gives some momentum to the rough surface when it enters it, (making it have velocity) then recovers it and returns the rough surface to zero velocity when it exits it. In that case the momentum of the ball decreases while travelling over the rough surface.
Not trying to give answers here, just add to the confusion lol.
Not quite following—your possibilities.
1. Alignment is almost impossible, then there is say 1e-20 chance we survive. Yes surviving worlds have luck and good alignment work etc. Perhaps you should work on alignment or still bednets if the odds really are that low.2. Alignment is easy by default, but there is nothing like 0.999999 we survive, say 95% because AGI that is not TAI superintelligence could cause us to wipe ourselves out first, among other things. (This is a slow takeoff universe(s))
#2 has much more branches in total where we survive (not sure if that matters) and the difference between where things go well and badly is almost all about stopping ourself killing ourselves with non TAI related things. In this situation, shouldn’t you be working on those things?
If you average 1,2 then you still get a lot of work on non-alignment related stuff.
I believe its somewhere closer to 50⁄50 and not so overdetermined one way or the other, but we are not considering that here.
OK for this post. “smart”. A response is smart/intelligent if
Firstly there is an assumed goal and measure. I don’t think it matters whether we are talking about the bees/birds as individuals or as part of the hive/flock. In this case the bee defense is effective both for the individual bee and hive. If a bee was only concerned about its survival, swarming the scout would still be beneficial, and of course such behavior is for the hive. Similarly for birds, flocks with large numbers of birds with swarming behavior would be better both for the flock, and individual birds in such a flock.
There is a force multiplier effect, the benefit of the behavior is much greater than the cost. This is obvious for the bees, a tiny expenditure of calories saves the hive. Likewise for birds, they waste a huge amount of calories both individually and collectively evading the hawk etc.
There is a local optimum (or something close) for the behavior—that is half measures don’t give half the benefit. So it seems like the result of foresight. There is perhaps more of a distinct and distant local optimum for the bee behavior “identify the scout, send warning chemicals to the hive, then swarm it” then the possible bird behavior “call then attack the attacker” as the scout isn’t the actual attack in the bees case.
The change is behavioral, rather than a physical adaptation.
This fits into the intuitive feeling of what intelligent is also. A characteristic of what people feel is intelligent is to imagine a scenario, then make it real. The bees havn’t done that, but the outcome is as if they had. “Imagine if we took out the scout, then there would be no later invasion”
You look at the birds and think “how do they miss this—ant colonies swarm to see off a larger attacker, herding animals do too, why do they miss such a simple effective strategy?” In that situation they are not intuitively intelligent.
Yes agree, unclear what you are saying that is different to me? The new solution is something unique and powerful when done well like language etc.
That’s some significant progress, but I don’t think will lead to TAI.
However there is a realistic best case scenario where LLM/Transformer stop just before and can give useful lessons and capabilities.
I would really like to see such an LLM system get as good as a top human team at security, so it could then be used to inspect and hopefully fix masses of security vulnerabilities. Note that could give a false sense of security, unknown unknown type situation where it would’t find a totally new type of attack, say a combined SW/HW attack like Rowhammer/Meltdown but more creative. A superintelligence not based on LLM could however.