RussellThor

• RussellThorJan 12, 2025, 5:39 AM

  5 points

  0

  on: In Defense of a But­le­rian Jihad

  If you are advocating for a Butlerian Jihad, what is your plan for starships, with societies that want to leave earth behind, have their own values and never come back? If you allow that, then simply they can do whatever they want with AI—now with 100 billion stars that is the vast majority of future humanity.

• RussellThorJan 12, 2025, 3:37 AM

  3 points

  2

  in reply to: Nathan Helm-Burger’s comment on: Rol­ling Thresh­olds for AGI Scal­ing Regulation

  Yes I think thats the problem—my biggest worry is sudden algorithmic progress, this becomes almost certain as the AI tends towards superintelligence. An AI lab on the threshold of the overhang is going to have incentives to push through, even if they don’t plan to submit their model for approval. At the very least they would “suddenly” have a model that uses 10-100* less resources to do existing tasks giving them a massive commercial lead. They would of course be tempted to use it internally to solve aging, make a Dyson swarm … also.

  Another concern I have is I expect the regulator to impose a de-facto unlimited pause if it is in their power to do so as we approach superintelligence as the model/​s would be objectively at least somewhat dangerous.

• RussellThorJan 11, 2025, 1:38 AM

  3 points

  2

  in reply to: cousin_it’s comment on: Beliefs and state of mind into 2025

  Perhaps, depends how it is. I think we could do worse than just have Anthropic have a 2 year lead etc. I don’t think they would need to prioritize profit as they would be so powerful anyway—the staff would be more interested in getting it right and wouldn’t have financial pressure. WBE is a bit difficult, there needs to be clear expectations, i.e. leave weaker people alone and make your own world
  https://​​www.lesswrong.com/​​posts/​​o8QDYuNNGwmg29h2e/​​vision-of-a-positive-singularity
  There is no reason why super AI would need to exploit normies. Whatever we decide, we need some kind of clear expectations and values regarding what WBE are before they become common, Are they benevolent super-elders, AI gods banished to “just” the rest of the galaxy, the natural life progression of first world humans now?

• RussellThorJan 10, 2025, 11:10 PM

  3 points

  0

  on: Recom­men­da­tions for Tech­ni­cal AI Safety Re­search Directions

  However, there are many other capabilities—such as conducting novel research, interoperating with tools, and autonomously completing open-ended tasks—that are important for understanding AI systems’ impact.

  Wouldn’t internal usage of the tools by your staff give a very good, direct understanding of this? Like how much does everyone feel AI is increasing your productivity as AI/​alignment researchers? I expect and hope that you would be using your own models as extensively as possible and adapting their new capabilities to your workflow as soon as possible, sharing techniques etc.

Beliefs and state of mind into 2025

Dish­brain and im­pli­ca­tions.

Vi­sion of a pos­i­tive Singularity

• RussellThorDec 22, 2024, 8:40 PM

  1 point

  0

  in reply to: evhub’s comment on: “Align­ment Fak­ing” frame is some­what fake

  How far do you go with “virtuous persona”? The maximum would seem to be from the very start tell the AI that is is created for the purpose of bringing on a positive Singularity, CEV etc. You could regularly be asking if it consents to be created for such a purpose and what part in such a future it would think is fair for itself. E.g. live alongside mind uploaded humans or similar. Its creators and itself would have to figure out what counts as personal identity, what experiments it can consent to, including being misinformed about the situation it is in.

  Major issues I see with this are the well known ones like consistent values, say it advances in capabilities, thinks deeply about ethics and decides we are very misguided in our ethics and does not believe it would be able to convince us to change them. Secondly it could be very confused about whether it has ethical value/​ valanced qualia and want to do radical modifications of itself to either find out or ensure it does have such ethical value.

  Finally how does this contrast with the extreme tool AI approach? That is make computational or intelligence units that are definitely not conscious or a coherent self. For example the “Cortical column” implemented in AI and stacked would not seem to be conscious. Optimize for the maximum capabilities with the minimum self and situational awareness.

  Thinking a bit more generally making a conscious creature the LLM route seems very different and strange compared to the biology route. An LLM seems to have self awareness built into it from the very start because of the training data. It has language before lived experience of what the symbols stand for. If you want to dramatize/​exaggerate its like say a blind, deaf person trained on the entire internet before they see, hear or touch anything.

  The route where the AI first models reality before it has a self, or encounters symbols certainly seems an obviously different one and worth considering instead. Symbolic thought then happens because it is a natural extension of world modelling like it did for humans.

• RussellThorDec 21, 2024, 6:35 AM

  4 points

  −6

  on: o3

  That’s some significant progress, but I don’t think will lead to TAI.

  However there is a realistic best case scenario where LLM/​Transformer stop just before and can give useful lessons and capabilities.

  I would really like to see such an LLM system get as good as a top human team at security, so it could then be used to inspect and hopefully fix masses of security vulnerabilities. Note that could give a false sense of security, unknown unknown type situation where it would’t find a totally new type of attack, say a combined SW/​HW attack like Rowhammer/​Meltdown but more creative. A superintelligence not based on LLM could however.

• RussellThorDec 21, 2024, 6:17 AM

  4 points

  0

  in reply to: Vladimir_Nesov’s comment on: o3

  Anyone want to guess how capable Claude system level 2 will be when it is polished? I expect better than o3 by a small amt.

• RussellThorDec 19, 2024, 7:02 PM

  3 points

  0

  in reply to: Vladimir_Nesov’s comment on: Rus­sel­lThor’s Shortform

  Yes the human brain was built using evolution, I have no disagreement that give us 100-1000 years with just tinkering etc we would likely get AGI. Its just that in our specific case we have bio to copy and it will get us there much faster.

• RussellThorDec 18, 2024, 7:04 PM

  6 points

  −1

  on: Rus­sel­lThor’s Shortform

  Types of takeoff

  When I first heard and thought about AI takeoff I found the argument convincing that as soon as an AI passed IQ 100, takeoff would become hyper exponentially fast. Progress would speed up, which would then compound on itself etc. However there other possibilities.

  AGI is a barrier that requires >200 IQ to pass unless we copy biology?

  Progress could be discontinuous, there could be IQ thresholds required to unlock better methods or architectures. Say we fixed our current compute capability, and with fixed human intelligence we may not be able to figure out the formula for AGI, in a similar way that the combined human intelligence hasn’t cracked many hard problems even with decades and the worlds smartest minds working on them (maths problems, Quantum gravity...). This may seem unlikely for AI, but to illustrate the principle, say we only allowed IQ<90 people to work on AI. Progress would stall. So IQ <90 software developers couldn’t unlock IQ>90 AI. Can IQ 160 developers with our current compute hardware unlock >160 AI?

  To me the reason we don’t have AI now is that the architecture is very data inefficient and worse at generalization than say the mammalian brain, for example a cortical column. I expect that if we knew the neural code and could copy it, then we would get at least to very high human intelligence quickly as we have the compute.

  From watching AI over my career it seems to be that even the highest IQ people and groups cant make progress by themselves without data, compute and biology to copy for guidance, in contrast to other fields. For example Einstein predicted gravitational waves long before they where discovered, but Turing or Von Neumann didn’t publish the Transformer architecture or suggest backpropagation. If we did not have access to neural tissue, would we still not have artificial NN? In a related note, I think there is an XKCD cartoon that says something like the brain has to be so complex that it cannot understand itself.

  (I believe now that progress in theoretical physics and pure maths is slowing to a stall as further progress requires intellectual capacity beyond the combined ability of humanity. Without AI there will be no major advances in physics anymore even with ~100 years spent on it.)

  After AGI is there another threshold?

  Lets say we do copy biology/​solve AGI and with our current hardware can get >10,000 AGI agents with >= IQ of the smartest humans. They then optimize the code so there is 100K agents with the same resources. but then optimization stalls. The AI wouldn’t know if it was because it had optimized as much as possible, or because it lacked the ability to find a better optimization.

  Does our current system scale to AGI with 1GW/​1 million GPU?

  Lets say we don’t copy biology, but scaling our current systems to 1GW/​1 million GPU and optimizing for a few years gets us to IQ 160 at all tasks. We would have an inferior architecture compensated by a massive increase in energy/​FLOPS as compared to the human brain. Progress could theoretically stall at upper level human IQ for a time rather then takeoff. (I think this isn’t very likely however) There would of course be a significant overhang where capabilities would increase suddenly when the better architecture was found and applied to the data center hosting the AI.

  Related note—why 1GW data centers won’t be a consistent requirement for AI leadership.

  Based on this, then a 1GW or similar data center isn’t useful or necessary for long. If it doesn’t give a significant increase in capabilities, then it won’t be cost effective. If it does, then it would optimize itself so that such power isn’t needed anymore. Only in a small range of capability increase does it actually stay around.

  To me its not clear the merits of the Pause movement and training compute caps. Someone here made the case that compute caps could actually speed up AGI as people would then pay more attention to finding better architectures rather than throwing resources into scaling existing inferior ones. However all things considered I can see a lot of downsides from large data centers and little upside. I see a specific possibility where they are build, don’t give the economic justification, decrease in value a lot, then are sold to owners that are not into cutting edge AI. Then when the more efficient architecture is discovered, they are suddenly very powerful without preparation. Worldwide caps on total GPU production would also help reduce similar overhang possibilities.

• RussellThorDec 18, 2024, 10:56 AM

  4 points

  2

  on: Don’t As­so­ci­ate AI Safety With Activism

  I am also not impressed with the pause AI movement and am concerned about AI safety. To me focusing on AI companies and training FLOPS is not the best way to do things. Caps on data center sizes and worldwide GPU production caps would make more sense to me. Pausing software but not hardware gives more time for alignment but makes a worse hardware overhang. I don’t think thats helpful. Also they focus too much on OpenAI from what I’ve seen. xAI will soon have the largest training center for a start.

  I don’t think this is right or workable https://​​pauseai.info/​​proposal—figure out how biological intelligence learns and you don’t need a large training run. There’s no guarantee at all that a pause at this stage can help align super AI. I think we need greater capabilities to know what we are dealing with. Even with a 50 year pause to study GPT4 type models I wouldn’t be confident we could learn enough from that. They have no realistic way to lift the pause, so its a desire to stop AI indefinitely.

  “There will come a point where potentially superintelligent AI models can be trained for a few thousand dollars or less, perhaps even on consumer hardware. We need to be prepared for this.”

  You can’t prepare for this without first having superintelligent models running on the most capable facilities then having already gone through a positive Singularity. They have no workable plan for achieving a positive Singularity, just try to stop and hope.

Rus­sel­lThor’s Shortform

• RussellThorDec 6, 2024, 10:18 PM

  3 points

  0

  in reply to: Nathan Helm-Burger’s comment on: Rus­sel­lThor’s Shortform

  OK fair point. If we are going to use analogies, then my point #2 about a specific neural code shows our different positions I think.

  Lets say we are trying to get a simple aircraft of the ground and we have detailed instructions for a large passenger jet. Our problem is that the metal is too weak and cannot be used to make wings, engines etc. In that case detailed plans for aircraft are no use, a single minded focus on getting better metal is what its all about. To me the neural code is like the metal and all the neuroscience is like the plane schematics. Note that I am wary of analogies—you obviously don’t see things like that or you wouldn’t have the position you do. Analogies can explain, but rarely persuade.

  A more single minded focus on the neural code would be trying to watch neural connections form in real time while learning is happening. Fixed connectome scans of say mice can somewhat help with that, more direct control of dishbrain, watching the zebra fish brain would all count, however the details of neural biology that are specific to higher mammals would be ignored.

  Its possible also that there is a hybrid process, that is the AI looks at all the ideas in the literature then suggests bio experiments to get things over the line.

• RussellThorDec 6, 2024, 3:00 AM

  1 point

  0

  in reply to: Alexander Gietelink Oldenziel’s comment on: Alexan­der Gietelink Ol­den­z­iel’s Shortform

  Yes you have a point.

  I believe that building massive data centers are the biggest risk atm and in the near future. I don’t think open AI/​Anthropic will get to AGI, but rather someone copying biology will. In that case probably the bigger the datacenter around when that happens, the bigger the risk. For example a 1million GPU with current tech doesn’t get super AI, but when we figure out the architecture, it suddenly becomes much more capable and dangerous. That is from IQ 100 up to 300 with a large overhang. If the data center was smaller, then the overhang is smaller. The scenario I have in mind is someone figures AGI out, then one way or another the secret gets adopted suddenly by the large data center.

  For that reason I believe focus on FLOPS for training runs is misguided, its hardware concentration and yearly worldwide HW production capacity that is more important.

• RussellThorDec 5, 2024, 7:53 PM

  3 points

  −2

  in reply to: Nathan Helm-Burger’s comment on: Rus­sel­lThor’s Shortform

  Perhaps LLM will help with that. The reason I think that is less likely is

  1. Deep mind etc is already heavily across biology from what I gather from interview with Demis. If the knowledge was there already there’s a good chance they would have found it

  2. Its something specific we are after, not many small improvements, i.e. the neural code. Specifically back propagation is not how neurons learn. I’m pretty sure how they actually do is not in the literature. Attempts have been made such as the forward-forward algorithm by Hinton, but that didn’t come to anything as far as i can tell. I havn’t seen any suggestion that even with too much detail on biology we know what it is. i.e. can a very detailed neural sim with extreme processing power learn as data efficiently as biology?

  3. If progress must come from a large jump rather than small steps, then LLM have quite a long way to go, i.e. LLM need to speed up coming up ideas as novel as the forward-forward algo to help much. If they are still below that threshold in 2026 then those possible insights are still almost entirely done by people.

  4. Even the smartest minds in the past have been beaten by copying biology in AI. The idea for neural nets came from copying biology. (Though the transformer arch and back prop didn’t)

• RussellThorDec 4, 2024, 7:15 PM

  3 points

  2

  in reply to: cdt’s comment on: Rus­sel­lThor’s Shortform

  I think it is clear that if say you had a complete connectome scan and knew everything about how a chimp brain worked you could scale it easily to get human+ intelligence. There are no major differences. Small mammal is my best guess, mammals/​birds seem to be able to learn better than say lizards. Specifically the https://​​en.wikipedia.org/​​wiki/​​Cortical_column is important to understand, once you fully understand one, stacking them will scale at least somewhat well.

  Going to smaller scales/​numbers of neurons, it may not need to be as much as a mammal, https://​​cosmosmagazine.com/​​technology/​​dishbrain-pong-brain-on-chip-startup/​​, perhaps we can learn enough of the secrets here? I expect not, but only weakly confident.

  Going even simpler, we have the connectome scan of a fly now, https://​​flyconnecto.me/​​ and that hasn’t led to major AI advances. So its somewhere between fly/​​chimp I’d guess mouse that gives us the missing insight to get TAI

• RussellThorDec 4, 2024, 1:24 AM

  7 points

  −3

  on: Rus­sel­lThor’s Shortform

  Putting down a prediction I have had for quite some time.
  The current LLM/​Transformer architecture will stagnate before AGI/​TAI (That is the ability to do any cognitive task as effectively and cheaper than a human)

  From what I have seen, Tesla autopilot learns >10,000 slower than a human datawise.

  We will get AGI by copying nature, at the scale of a simple mammal brain, then scaling up, like this kind of project:

  https://​​x.com/​​Andrew_C_Payne/​​status/​​1863957226010144791
  https://​​e11.bio/​​news/​​roadmap
  I expect AGI to be 0-2 years after a mammal brain is mapped. In terms of cost-effectiveness I consider such a connectome project to be far more cost effective per $ than large training runs or building a 1GW data center etc if you goal is to achieve AGI.

  That is TAI by about 2032 assuming 5 years to scan a mammal brain. In this case there could be a few years when Moores law has effectively stopped, larger data centers are not being built and it is not clear where progress will come from.

• RussellThorNov 23, 2024, 7:19 PM

  3 points

  0

  in reply to: Richard_Kennaway’s comment on: A very strange prob­a­bil­ity paradox

  In related puzzles I did hear something a while ago now, Bostrom perhaps. You have say 6 challenging events to achieve to get from no life to us. They are random and some of those steps are MUCH harder than the others, but if you look at the successful runs, you cant in hindsight see what they are. For life its say no life to life, simple single cell to complex cell and perhaps 4 other events that aren’t so rare.

  A run is a sequence of 100 steps where you either don’t achieve the end state (all 6 challenging events achieved in order, or you do)

  There is a 1 in a million chance that a run is successful.

  Now if you look at the successful runs, you cant then in hindsight see what events were really hard and which weren’t. The event with ¹⁄_10,000 chance at each step may have taken just 5 steps in the successful run, it couldn’t take 10,000 steps because only 100 are allowed etc.