2024-12-12

My AI timelines

DISCLAIMER

I’m not a deep learning expert. I understand theory of LLMs, RNNs, CNNs, etc. but I don’t have experience training large models or curating large datasets or doing original DL research.
Please consider getting the opinion of many people other than me. Many of the signatories on this list have made individual statements, search for their podcast interviews, blogposts, research publications and so on. Lesswrong is also a good forum.
I strongly recommend you form your own independent opinion on this subject, and not blindly copy the views of others, no matter how smart or trustworthy they seem. This field would make a lot more sense if more people formed independent opinions. (And yes forming a good opinion takes a lot of time and effort, and you have to decide whether the time investment is worth it for you personally.)

This document is mainly aimed at lesswrong (LW) rationalists / effective altruists / adjacent people, since a lot of my work is culturally downstream of theirs, and a lot of my potential research collaborators exist in their communities. This doc will make less sense if you haven’t encountered their writings before.

Most of this doc is guesswork rather than models I have a lot of confidence in. Small amounts of evidence could upend my entire view of these topics.

If you have evidence that my view is wrong, please tell me. Not having to spend any more time thinking about AI will improve my quality of life. I am being completely serious when I say I might thank you till the day I die, if you persuade me either way. I can also pay you atleast $1000 for convincing me, although we’ll have to discuss the details if you really wanna be paid.

The last time I read papers on this topic was early-2023, it’s possible I’m not up-to-speed on any of the latest research. Feel free to send me anything that’s relevant.

I have ~15% probability humanity will invent artificial superintelligence (ASI) by 2030.

If it happens, this will be the most significant event in ~10,000 years of human history on many metrics of what counts as significant. (If the intelligence gap is sufficiently large it might even be the most important event in the ~14 billion year history of the universe. My probability for this is smaller.)
This will more-likely-than-not use transformers + data and compute scaling.
I define artificial superintelligence to be AI superior to the best humans (not median human) at basically every task humans can do on laptop with internet access.
See my usual disclaimer on predictions shaping reality. If you are involved in building ASI, please consider *not* doing that or atleast talking to me once about it. I don’t want ASI to be built by 2030 unless many aspects of society change first.

Conditional on ASI being invented by 2030, I expect ~30% probability it will kill everyone on Earth soon after. In total that’s ~5% probability of humanity killed by ASI by 2030.

I have chosen not to work on this problem myself. This is downstream of ~15% not being large enough and the problem not capturing my curiosity enough. I am not a utilitarian who blindly picks the biggest problem to solve, although I do like picking bigger problems over smaller ones. If you are working on safety of AGI/ASI, I think you are doing important work and I applaud you for the same.

Reasons for my beliefs

Mildly relevant rant on consensus-building: Both these probabilities seem to involve dealing with what I’d call deep priors. “Assume I take a random photo of Mars and a random pixel from that photo, what is the probability its RGB value has higher Green than Blue?” Humans tend to agree better on prior probabilities when there’s enough useful data around the problem to form a model of it. Humans tend to agree less on prior probabilities when they’re given a problem with very little obviously useful data, and need to rely on a lifetime of almost-useless-but-not-completely-useless data instead. The “deepest” prior in this ontology is the universal prior.

~15% is lower than what many people in EA/LW communities assign, because I reject a lot of the specific models they use to forecast higher likelihood of ASI.

Total amount of neurons in human brain has nothing to do with compute required to build ASI, as evolution produced human brains and evolution used a lot more compute than an individual brain.
Nobody has a proposal that’s IMO likely to work at replicating biological evolution in-silico. How to capture initial conditions? How much compute do you require to simulate the environment? I haven’t seen convincing arguments for why these details aren’t important.
I have a prior that most STEM research that promises bold outcomes fails. This is the default not the exception. To study this prior you have to go back in time to each of the failed research agendas of the past 100 years, and notice what it would have **felt like** if you were born then and were hyped up by a research agenda you didn’t know would succeed or fail.

Discontinuous progress in human history by Katja Grace is the closest thing I could find to work that tries evaluating this prior probability. I have not spent a lot of time searching though. Convincing me either way will require publishing or pointing me to a lot more work of this type. Alternatively you have to provide me a gears-level model that explains why the LLM scaling laws empirically hold.

~15% is higher than what many AI researchers assign, because I reject a lot of the specific reasons they give for why LLM scaling cannot possibly achieve ASI

I have read some arguments for why specific unsolved problems are *hard* when compared with already solved problems, and why there’s no specific reason LLM will crack them.
However most of the problems LLMs have already cracked also got cracked without anyone having an actual model for why LLMs will or won’t crack them. LLM scaling has consistently proven itself (to me) to be black magic; both its supporters and its detractors fail to accurately predict which problems it will or won’t crack. Predicting loss curve doesn’t tell which problems will or won’t be cracked.
Some people cite architectural limitations of LLM as a bottleneck. For instance LLM has a theoretical upperbound on number of calculations per forward pass, and a certain ratio of “memory access” to “computation” steps. But solving a real world problem can use multiple forward passes + non-LLM scaffolding + non-self-attention layers. For example, you can often use layers from two different architectures (let’s say self-attention layers and CNN layers) in a single model, train this model and get good performance.
Some people say we will run out of data required as per empirical scaling law but my guess is this is more likely than not solveable. I’m very unsure on this. (And I lack sufficient expertise to evaluate this argument tbh.) I’m guessing you can teach a model simple reasoning using a smaller dataset, use this model to fill holes in a bigger dataset, and then teach the model more difficult reasoning using this data.

~30% probability for extinction conditional on ASI invention by 2030 is because I am more optimistic about boxing an ASI than some LW rationalists. I do believe misalignment happens by default with high probability.

The intelligence difference between Einstein or Churchill or Hitler, and a hyperpersuasive AI is large, relative to the intelligence jumps shown by scaling so far. I understand there’s no natural scale for intelligence. I am open to the idea a hyperpersuasive AI can exist in theory. (In this context, a hyperpersuasive AI is an AI that can gain complete control over your mind with near 100% probability simply by talking to you.)
I am assuming a high level of competence among the people boxing the ASI. I have low probability of ASI coming completely by surprise. A lab building ASI in this decade will almost certainly employ many people who take the prospect of ASI seriously.
I am assuming that pausing ASI research will be politically possible, even obvious, once we have built a boxed ASI with empirical evidence of a lack of safety. I think I have better intuitions on politics than the median LW poster.
I have some confusion on how the AI will reason about the hardware-software abstraction. At what point does an AI translate “maximise XYZ assembly variable” into “maximise the number of electrons in positions in semiconductor underlying XYZ”? My guess is whether an AI wants to “break out” of its machine depends on how it reasons about this. I accept that an ASI could understand that its variables are made up of positions of electrons in a semicondutor, I’m just unsure what it’ll do once it knows this.