Personal website: https://andrewtmckenzie.com/
Andy_McKenzie
Karma: 1,516
Sounds good, I’m happy with that arrangement once we get these details figured out.
Regarding the human programmer formality, it seems like business owners would have to be really incompetent for this to be a factor. Plenty of managers have coding experience. If the programmers aren’t doing anything useful then they will be let go or new companies will start that don’t have them. They are a huge expense. I’m inclined to not include this since it’s an ambiguity that seems implausible to me.
Regarding the potential ban by the government, I wasn’t really thinking of that as a possible option. What kind of ban do you have in mind? I imagine that regulation of AI is very likely by then, so if the automation of all programmers hasn’t happened by Jan 2027, it seems very easy to argue that it would have happened in the absence of the regulation.
Regarding these and a few of the other ambiguous things, one way we could do this is that you and I could just agree on it in Jan 2027. Otherwise, the bet resolves N/A and you don’t donate anything. This could make it an interesting Manifold question because it’s a bit adversarial. This way, we could also get rid of the requirement for it to be reported by a reputable source, which is going to be tricky to determine.
Understandable. How about this?
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
I guess that there might be some disagreements in these terms, so I’d be curious to hear your suggested improvements.
Caveat: I don’t have much disposable money right now, so it’s not much money, but perhaps this is still interesting as a marker of our beliefs. Totally ok if it’s not enough money to be worth it to you.
I’m wondering if we could make this into a bet. If by remote workers we include programmers, then I’d be willing to bet that GPT-5/6, depending upon what that means (might be easier to say the top LLMs or other models trained by anyone by 2026?) will not be able to replace them.
These curves are due to temporary plateaus, not permanent ones. Moore’s law is an example of a constraint that seems likely to plateau. I’m talking about takeoff speeds, not eventual capabilities with no resource limitations, which I agree would be quite high and I have little idea of how to estimate (there will probably still be some constraints, like within-system communication constraints).
Does anyone know of any AI-related predictions by Hinton?
Here’s the only one I know of—“People should stop training radiologists now. It’s just completely obvious within five years deep learning is going to do better than radiologists because it can get a lot more experience. And it might be ten years but we got plenty of radiologists already.” − 2016, slightly paraphrased
This seems like still a testable prediction—by November 2026, radiologists should be completely replaceable by deep learning methods, at least other than regulatory requirements for trained physicians.
Thanks! I agree with you about all sorts of AI alignment essays being interesting and seemingly useful. My question was more about how to measure the net rate of AI safety research progress. But I agree with you that an/your expert inside view of how insights are accumulating is a reasonable metric. I also agree with you that the acceptance of TAI x-risk in the ML community as a real thing is useful and that—while I am slightly worried about the risk of overshooting, like Scott Alexander describes—this situation seems to be generally improving.
Regarding (2), my question is why algorithmic growth leading to serious growth of AI capabilities would be so discontinuous. I agree that RL is much better in humans than in machines, but I doubt that replicating this in machines would require just one or a few algorithmic advances. Instead, my guess, based on previous technology growth stories I’ve read about, is that AI algorithmic progress is likely to occur due to the accumulation of many small improvements over time.
Good essay! Two questions if you have a moment:
1. Can you flesh out your view of how the community is making “slow but steady progress right now on getting ready”? In my view, much of the AI safety community seems to be doing things that have unclear safety value to me, like (a) coordinating a pause in model training that seems likely to me to make things less safe if implemented (because of leading to algorithmic and hardware overhangs) or (b) converting to capabilities work (quite common, seems like an occupational hazard for someone with initially “pure” AI safety values). Of course, I don’t mean to be disparaging, as plenty of AI safety work does seem useful qua safety to me, like making more precise estimates of takeoff speeds or doing cybersecurity work. Just was surprised by that statement and I’m curious about how you are tracking progress here.
2. It seems like you think there are some key algorithmic insights, that once “unlocked”, will lead to dramatically faster AI development. This suggests that not many people are working on algorithmic insights. But that doesn’t seem quite right to me—isn’t that a huge group of researchers, many of whom have historically been anti-scaling? Or maybe you think there are core insights available, but the field hasn’t had (enough of) its Einsteins or von Neumanns yet? Basically, I’m trying to get a sense of why you seem to have very fast takeoff speed estimates given certain algorithmic progress. But maybe I’m not understanding your worldview and/or maybe it’s too infohazardous to discuss.
I didn’t realize you had put so much time into estimating take-off speeds. I think this is a really good idea.
This seems substantially slower than the implicit take-off speed estimates of Eliezer, but maybe I’m missing something.
I think the amount of time you described is probably shorter than I would guess. But I haven’t put nearly as much time into it as you have. In the future, I’d like to.
Still, my guess is that this amount of time is enough that there are multiple competing groups, rather than only one. So it seems to me like there would probably be competition in the world you are describing, making a singleton AI less likely.
Do you think that there will almost certainly be a singleton AI?
Thanks for writing this up as a shorter summary Rob. Thanks also for engaging with people who disagree with you over the years.
Here’s my main area of disagreement:
General intelligence is very powerful, and once we can build it at all, STEM-capable artificial general intelligence (AGI) is likely to vastly outperform human intelligence immediately (or very quickly).
I don’t think this is likely to be true. Perhaps it is true of some cognitive architectures, but not for the connectionist architectures that are the only known examples of human-like AI intelligence and that are clearly the top AIs available today. In these cases, I expect human-level AI capabilities to grow to the point that they will vastly outperform humans much more slowly than immediately or “very quickly”. This is basically the AI foom argument.
And I think all of your other points are dependent on this one. Because if this is not true, then humanity will have time to iteratively deal with the problems that emerge, as we have in the past with all other technologies.
My reasoning for not expecting ultra-rapid takeoff speeds is that I don’t view connectionist intelligence as having a sort of “secret sauce”, that once it is found, can unlock all sorts of other things. I think it is the sort of thing that will increase in a plodding way over time, depending on scaling and other similar inputs that cannot be increased immediately.
In the absence of some sort of “secret sauce”, which seems necessary for sharp left turns and other such scenarios, I view AI capabilities growth as likely to follow the same trends as other historical growth trends. In the case of a hypothetical AI at a human intelligence level, it would face constraints on its resources allowing it to improve, such as bandwidth, capital, skills, private knowledge, energy, space, robotic manipulation capabilities, material inputs, cooling requirements, legal and regulatory barriers, social acceptance, cybersecurity concerns, competition with humans and other AIs, and of course value maintenance concerns (i.e. it would have its own alignment problem to solve).
I guess if you are also taking those constraints into consideration, then it is really just a probabilistic feeling about how much those constraints will slow down AI growth. To me, those constraints each seem massive, and getting around all of them within hours or days would be nearly impossible, no matter how intelligent the AI was.
As a result, rather than indefinite and immediate exponential growth, I expect real-world AI growth to follow a series of sigmoidal curves, each eventually plateauing before different types of growth curves take over to increase capabilities based on different input resources (with all of this overlapping).
One area of uncertainty: I am concerned about there being a spectrum of takeoff speeds, from slow to immediate. In faster takeoff speed worlds, I view there as being more risk of bad outcomes generally, such as a totalitarian state using an AI to take over the world, or even the x-risk scenarios that you describe.
This is why I favor regulations that will be helpful in slower takeoff worlds, such as requiring liability insurance, and will not cause harm by increasing take-off speed. For example, pausing AGI training runs seems likely to make takeoff speed more discontinuous, due to creating hardware, algorithmic, and digital autonomous agent overhangs, thereby making the whole situation more dangerous. This is why I am opposed to it and dismayed to see so many on LW in favor of it.
I also recognize that I might be wrong about AI takeoff speeds not being fast. I am glad people are working on this, so long as they are not promoting policies that seem likely to make things more dangerous in the slower takeoff scenarios that I consider more likely.
Another area of uncertainty: I’m not sure what is going to happen long-term in a slow takeoff world. I’m confused. While I think that the scenarios you describe are not likely because they are dependent upon there being a fast takeoff and a resulting singleton AI, I find outcomes in slow takeoff worlds extraordinarily difficult to predict.
Overall I feel that AI x-risk is clearly the most likely x-risk of any in the coming years and am glad that you and others are focusing on it. My main hope for you is that you continue to be flexible in your thinking and make predictions that help you to decide if you should update your models.
Here are some predictions of mine:
Connectionist architectures will remain the dominant AI architecture in the next 10 years. Yes, they will be hooked up in larger deterministic systems, but humans will also be able to use connectionist architectures in this way, which will actually just increase competition and decrease the likelihood of ultra-rapid takeoffs.
Hardware availability will remain a constraint on AI capabilities in the next 10 years.
Robotic manipulation capabilities will remain a constraint on AI capabilities in the next 10 years.
I can see how both Yudkowsky’s and Hanson’s arguments can be problematic because they either assume fast or slow takeoff scenarios, respectively, and then nearly everything follows from that. So I can imagine why you’d disagree with every one of Hanson’s paragraphs based on that. If you think there’s something he said that is uncorrelated with the takeoff speed disagreement, I might be interested, but I don’t agree with Hanson about everything either, so I’m mainly only interested if it’s also central to AI x-risk. I don’t want you to waste your time.
I guess if you are taking those constraints into consideration, then it is really just a probabilistic feeling about how much those constraints will slow down AI growth? To me, those constraints each seem massive, and getting around all of them within hours or days would be nearly impossible, no matter how intelligent the AI was. Is there any other way we can distinguish between our beliefs?
If I recall correctly from your writing, you have extremely near-term timelines. Is that correct? I don’t think that AGI is likely to occur sooner than 2031, based on this criteria: https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/
Is this a prediction that we can use to decide in the future whose model of the world today was more reasonable? I know it’s a timelines question, but timelines are pretty correlated with takeoff speeds I guess.
To clarify, when I mentioned growth curves, I wasn’t talking about timelines, but rather takeoff speeds.
In my view, rather than indefinite exponential growth based on exploiting a single resource, real-world growth follows sigmoidal curves, eventually plateauing. In the case of a hypothetical AI at a human intelligence level, it would face constraints on its resources allowing it to improve, such as bandwidth, capital, skills, private knowledge, energy, space, robotic manipulation capabilities, material inputs, cooling requirements, legal and regulatory barriers, social acceptance, cybersecurity concerns, competition with humans and other AIs, and of course safety concerns (i.e. it would have its own alignment problem to solve).
I’m sorry you resent that implication. I certainly didn’t mean to offend you or anyone else. It was my honest impression, for example, based on the fact that there hadn’t seemed to be much if any discussion of Robin’s recent article on AI on LW. It just seems to me that much of LW has moved past the foom argument and is solidly on Eliezer’s side, potentially due to selection effects of non-foomers like me getting heavily downvoted like I was on my top-level comment.
Here’s a nice recent summary by Mitchell Porter, in a comment on Robin Hanson’s recent article (can’t directly link to the actual comment unfortunately):
Robin considers many scenarios. But his bottom line is that, even as various transhuman and posthuman transformations occur, societies of intelligent beings will almost always outweigh individual intelligent beings in power; and so the best ways to reduce risks associated with new intelligences, are socially mediated methods like rule of law, the free market (in which one is free to compete, but also has incentive to cooperate), and the approval and disapproval of one’s peers.
The contrasting philosophy, associated especially with Eliezer Yudkowsky, is what Robin describes with foom (rapid self-enhancement) and doom (superintelligence that cares nothing for simpler beings). In this philosophy, the advantages of AI over biological intelligence are so great, that the power differential really will favor the individual self-enhanced AI, over the whole of humanity. Therefore, the best way to reduce risks is through “alignment” of individual AIs—giving them human-friendly values by design, and also a disposition which will prefer to retain and refine those values, even when they have the power to self-modify and self-enhance.
Eliezer has lately been very public about his conviction that AI has advanced way too far ahead of alignment theory and practice, so the only way to keep humanity safe is to shut down advanced AI research indefinitely—at least until the problems of alignment have been solved.
ETA: Basically I find Robin’s arguments much more persuasive, and have ever since those heady days of 2008 when they had the “Foom” debate. A lot of people agreed with Robin, although SIAI/MIRI hasn’t tended to directly engage with those arguments for whatever reason.
This is a very common outsider view of LW/SIAI/MIRI-adjacent people, that they are “foomers” and that their views follow logically from foom, but a lot of people don’t agree that foom is likely because this is not how growth curves have worked for nearly anything historically.
AIs can potentially trade with humans too though, that’s the whole point of the post.
Especially if the AI’s have architectures/values that are human brain-like and/or if humans have access to AI tools, intelligence augmentation, and/or whole brain emulation.
Also, it’s not clear why AIs will find it easier to coordinate with one another than humans and humans or humans and AIs. Coordination is hard for game theoretic reasons.
These are all standard points, I’m not saying anything new here.
When you write “the AI” throughout this essay, it seems like there is an implicit assumption that there is a singleton AI in charge of the world. Given that assumption, I agree with you. But if that assumption is wrong, then I would disagree with you. And I think the assumption is pretty unlikely.
No need to relitigate this core issue everywhere, just thought this might be useful to point out.
I agree this is a very important point and line of research. This is how humans deal with sociopaths, after all.
Here’s me asking a similar question and Rob Bensinger’s response: https://www.lesswrong.com/posts/LLRtjkvh9AackwuNB/on-a-list-of-lethalities?commentId=J42Fh7Sc53zNzDWCd
One potential wrinkle is that in a very fast take off world AI’s could potentially coordinate very well because they would basically be the same, or close branches of the same AI.
“Science advances one funeral at a time” → this seems to be both generally not true as well as being a harmful meme (because it is a common argument used to argue against life extension research).
https://www.lesswrong.com/posts/fsSoAMsntpsmrEC6a/does-blind-review-slow-down-science
Interesting, thanks. All makes sense and no need to apologize. I just like it when people write/think about schizophrenia and want to encourage it, even as a side project. IMO, it’s a very important thing for our society to think about.
A lot of the quotes do find decreased connectivity, but some of them find increased connectivity between certain regions. It makes me think that there’s a probability there might be something more complicated than just “increased or decreased”, but rather specific types of connections. But that’s just a guess, and I think an explanation across all cortical connections is more parsimonious and therefore more likely a priori.
Of your criteria of “things to explain”, here are some thoughts:
4.1 The onset of schizophrenia is typically in the late-teens-to-twenties, 4.2 Positive symptoms—auditory hallucinations (hearing voices), “distortions of self-experience”, etc. 4.3 Negative symptoms—yes these are all critical to explain.
4.4 Creativity—hm, this is tricky and probably needs to be contextualized. Some people disagree that schizophrenia is associated with increased creativity in relatives, although I personally agree with it. I don’t think it’s a core aspect.
4.5 Anticorrelation with autism—I don’t think this is a core aspect. I’m not even sure it’s true.
4.6 Relation to myelination—I think this is likely true, but I think it’s too low level to call a core aspect of the disease per se. I agree with your point about two terms always yielding search results, this is true of Alzheimer’s disease as well.
4.7 Schizophrenia and blindness—I don’t think this is a core aspect, I agree with you it’s probably not true.
Other core aspects I think should be explained:
1. Specific types of gene pathways that are altered in people with schizophrenia being related to the development/function of whatever the physiologic thing being hypothesized is. Genetics are causal, so this is usually pretty helpful, albeit quite complex.
2. Cognitive deficits: These include impairments in executive function, working memory, and other cognitive domains. These are usually considered distinct from negative symptoms (anhedonia, blunted affect, etc), and usually involve a decline from functioning premorbid/earlier in life.
3. Why nicotine is helpful.
4. Why antipsychotics/neuroleptics seem to be helpful (at least in certain circumstances).
5. Why there is so much variability in the disorder? Why do some people end up with predominantly delusions, hallucinations, or negative symptoms as the core part of their experience with schizophrenia?
Just some thoughts. As I said, I’m glad you’re focused on this!
Interesting theory and very important topic.
I think the best data source here is probably neuroimaging. Here’s a recent review: https://www.frontiersin.org/articles/10.3389/fnins.2022.1042814/full. Here are some quotes from that:
For functional studies, be they fluorodeoxyglucose positron emission tomography (FDG PET), rs-fMRI, task-based fMRI, diffusion tensor imaging (DTI) or MEG there generally is hypoactivation and disconnection between brain regions. …
Histologically this gray matter reduction is accompanied by dendritic and synaptic density decreases which likely signals a lack of communication (disconnection theory) across selected neural networks…
According to Orliac et al. (2013), patients with schizophrenia have reduced functional connectivity in the default mode network and salience network. Furthermore, decreased connectivity in the paracingulate cortex is associated with difficulties with abstract thought, whereas decreased connectivity in the left striatum is associated with delusions and depression. Longer memory response time for face recognition was also associated with functional connectivity abnormalities in early-schizophrenia, centered in the anterior cingulate...
This is in line with the frontotemporoparietal network disruption theory in schizophrenia that is well-known (Friston and Frith, 1995). …
In a study that conducted by Lottman et al., patients with schizophrenia showed an increased connectivity between auditory and subcortical networks …
Both increased and decreased functional connectivity has been observed in patients with schizophrenia vs. controls, in resting state and during various tasks
Mondino et al. (2016) found that transcranial direct current stimulation can decrease negative symptoms course and the severity of auditory verbal hallucination in patients with schizophrenia. This improvement was associated with reduction in functional connectivity between the left anterior insula and left temporoparietal junction (middle and superior temporal gyri and Wernicke’s area)
Overall I think it’s pretty complicated. I imagine that when you wrote explaining “everything” was tongue in cheek, but I think there are a lot of things that need to be explained about schizophrenia beyond the seven that you wrote about. I hope you keep doing some research in this area and continue to refine your theory.
Those sound good to me! I donated to your charity (the Animal Welfare Fund) to finalize it. Lmk if you want me to email you the receipt. Here’s the manifold market:
Bet
Andy will donate $50 to a charity of Daniel’s choice now.
If, by January 2027, there is not a report from a reputable source confirming that at least three companies, that would previously have relied upon programmers, and meet a defined level of success, are being run without the need for human programmers, due to the independent capabilities of an AI developed by OpenAI or another AI organization, then Daniel will donate $100, adjusted for inflation as of June 2023, to a charity of Andy’s choice.
Terms
Reputable Source: For the purpose of this bet, reputable sources include MIT Technology Review, Nature News, The Wall Street Journal, The New York Times, Wired, The Guardian, or TechCrunch, or similar publications of recognized journalistic professionalism. Personal blogs, social media sites, or tweets are excluded.
AI’s Capabilities: The AI must be capable of independently performing the full range of tasks typically carried out by a programmer, including but not limited to writing, debugging, maintaining code, and designing system architecture.
Equivalent Roles: Roles that involve tasks requiring comparable technical skills and knowledge to a programmer, such as maintaining codebases, approving code produced by AI, or prompting the AI with specific instructions about what code to write.
Level of Success: The companies must be generating a minimum annual revenue of $10 million (or likely generating this amount of revenue if it is not public knowledge).
Report: A single, substantive article or claim in one of the defined reputable sources that verifies the defined conditions.
AI Organization: An institution or entity recognized for conducting research in AI or developing AI technologies. This could include academic institutions, commercial entities, or government agencies.
Inflation Adjustment: The donation will be an equivalent amount of money as $100 as of June 2023, adjusted for inflation based on https://www.bls.gov/data/inflation_calculator.htm.
Regulatory Impact: In January 2027, Andy will use his best judgment to decide whether the conditions of the bet would have been met in the absence of any government regulation restricting or banning the types of AI that would have otherwise replaced programmers.