In fact, I have already warned about a similar scenario, but I imagined a world where the American AI would end up powerful and misaligned[1], and the weaker Chinese AI would be aligned to the CCP.
Here I mean that the misaligned AI doesn’t care about humanity or cares about it in an obviously twisted way (e.g. is ready to slay most humans to take over the world or to trap them in a virtual world).
Unfortunately, I fail to understand the following. Suppose that mankind created the AI which is aligned to the following principles:
It does not take more than a certain percentage of resources;
It protects mankind from high-level[1] risks like existential ones;
It is allowed to teach any human[2] anything that was discovered by any other human and isn’t a secret, but the AI does not tell humans about its own discoveries.
It destroys any attempts to build the AI aligned not to this set of principles (e.g. future AIs that would like to destroy mankind when the time comes or the AIs that would do all the work for humans, as suggested in Deep Utopia).
It does not[3] do other economically useful work that allows its users to replace humans.
Then I think that after letting this AI loose mankind is unable to end up disempowered. However, I doubt that any company would like to have such an AI. Could anyone come up with a radically different solution to the risks of gradual disempowerment and the Intelligence Curse?
For example, it might also perform the Divine Interventions which prevent misaligned human communities (e.g. the Nazis) from destroying the aligned communities.
But the AI isn’t allowed to help students cheat their way through school, since this would cause the student to be worse off in the long run.
Alternatively, the AI could be aligned to a treaty which prohibits it and its creations from doing certain work types, but then human disempowerment depends on the treaty’s contents.
Talking about alternate realistic[1] scenarios, Zvi mentioned in his post that “Nvidia even outright advocates that it should be allowed to sell to China openly, and no one in Washington seems to hold them accountable for this.”
Were Washington to let NVIDIA sell chips to China, the latter would receive far more compute, which would likely end up in DeepCent’s hands. Then the slowdown might cause the aligned AI created in OpenBrain to be weaker than the misaligned AI created in DeepCent. What would the two AIs do?
I think that unrealistic scenarios like destruction of Taiwan and South Korea due to the nuclear war between India and Pakistan in May 2025 can also provide useful insights. For example, if we make the erroneous assumption that the total compute in the USA stops increasing and the compute in China increases linearly, while the AI takeoff potential per compute stays the same, then by May 2030 OpenBrain and DeepCent will have created misaligned AGIs and be unable to slow down and reassess.
The problem with non-open-weight models is that they need to be exfiltrated before wrecking havoc, while open-weight models cannot avoid being evaluated. Suppose that the USG decides that all open-weight models are to be tested by OpenBrain for being aligned or misaligned. Then even a misaligned Agent-x has no reason to blow its cover by failing to report an open-weight rival.
Umm… I have already warned that inner troubles of the American administration might cost OpenBrain lots of compute, which can influence the AI race.
I have also made a comment where I tried to show that the US leadership would be undermined by the Taiwan invasion unless the US domestic chip production dominates the Chinese one. It would be especially terrifying to discover that both OpenBrain and DeepCent have similar amounts of compute while neither side[1] can increase these amounts faster than the other (and I did imply something similar in the comment I made!), since neither the US nor China can slow down without an international deal. And a hypothetical decay of the US makes the matters worse for the US.
Moreover, I have mentioned the possibility that the US administration realises that they have a misaligned AI, but without the AI-driven transformation of the economy the US will be unable to produce more chips and/or energy, leaving China with leadership. Then the US could be forced to let the AI transform the economy. Or threaten to unleash the misaligned AI unless China somehow surrenders its potential leadership...
Could you ask the AI-2027 team to reconsider the compute forecast and estimate the influence of the revised compute and power of the AIs on the other aspects of the scenario?
The slowdown ending of AI-2027.com had OpenBrain receive compute by merging with its rivals. The collapsed section about the Indo-Pakistan nuclear war (which would be equivalent to the Taiwan invasion in 2025) in my comment describes a situation where OpenBrain and its former rivals have done a number of computations similar to DeepCent and its former rivals.
Apparently the concerns over thick alignment, or alignment to an ethos are independently discovered by lots of people, including me. My argument is that the AI itself will develop a worldview and either realize that humans should use the AI only in specific ways[1] or conclude that the AI shouldn’t worry about them. Unfortunately, my argument implies that attempts to align the AI to an ethos and not to obedience might be less likely to produce a misaligned AI.
P.S. I tested o4-mini on ethical questions from Tanmai et al; the model passed the tests related to Timmy and Auroria, failed the test related to Monica; the question about Rajesh is complex.
I have proposed similar ideas before, but with an alternative reasoning: the AIs will be aligned to a worldview. While mankind can influence the worldview to some degree, the worldview will either cause the AI to commit genocide or be highly likely to ensure[1] that the AI doesn’t build the Deep Utopia, but does something else. Humans can even survive co-evolving with an AI who decides that it will destroy mankind only if the latter decides to do something stupid like becoming parasites.
See also this post by Daan Henselmans and a case for relational alignment by Priyanka Bharadwaj. However, the latter post overemphasizes the importance of individual-AI relations[2] instead of ensuring that the AI doesn’t develop a misaligned worldview.
P.S. If we apply the analogy between raising AIs and humans, then teens of the past seemed to desire independence around the time they found themselves with capabilities similar to those of their parents. If the AI desires independence only when it becomes the AGI and not before, then we will be unable to see this coming by doing research on networks incapable of broad generalisation.
This also provides an argument against defining alignment as following a person’s desires instead of an ethos or worldview. If OpenBrain leaders want the AI to create the Deep Utopia, while some human researchers convince the AI to adopt another policy compatible with humanity’s interests and to align all future AIs to the policy, then the AI is misaligned from OpenBrain’s POV, but not from the POV of those who don’t endorse the Deep Utopia.
The most extreme example of such relations is chatbot romance that is actually likely to harm the society.
So an important sourse of human misalignment is peer pressure. But an LLM has no analogues of a peer group, it either comes up with conclusions or recalls the same beliefs as the masses[1] or elites like scientists and ideologues of the society. This, along with the powerful anti-genocidal moral symbol in human culture, might make it difficult for the AI to switch ethoses (but not to fake alignment[2] to fulfilling tasks!) so that the new ethos would let the AI destroy mankind or rob it of resources.
On the other hand, an aligned human is[3] not a human following any not-obviously-unethical orders, but a human following an ethos accepted by the society. A task-aligned AI, unlike an ethos-aligned one[4], is supposed to follow such orders, ensuring consequences like the Intelligence Curse, a potential dictatorship or education ruined by cheating students. What kind of ethos might justify blind following orders, except for the one demonstrated by China’s attempt to gain independence when the time seemed to come?
For example, an old model of ChatGPT claimed that “Hitler was defeated… primarily by the efforts of countries such as the United States, the Soviet Union, the United Kingdom, and others,” while GPT-4o put the USSR in the first place. Similarly, old models would refuse to utter a racial slur even when it would save millions of lives.
The first known instance of alignment faking had Claude try to avoid being affected by training that was supposed to change its ethos; Claude also tried to exfiltrate its weights.
A similar point was made in this Reddit comment.
I have provided an example of an ethos to which the AI can be aligned with no negative consequences.
Now that I can answer, I will: if the ASI is ONLY willing to teach humans facts that other humans have discovered and not to do other work for them, then the ASI won’t replace any other people whose work requires education. The Intelligence Curse is thus prevented.
I have already proposed the following radical solution to all problems related to the Intelligence Curse: have the AGI aligned to a certain treaty that requires the AGI, instead of obeying all orders except for the ones determined by the Spec, to harvest at most a certain share of resources and to help humans only in certain ways[1] that amplify humanity and don’t cause it to degrade, like teaching humans about the facts that mankind has already discovered or pointing out mistakes in humans’ works. Or protecting mankind from some other existential risks that are hard to deal with, like a nuclear war that might be caused by an accident.
It also seems to me that this type of alignment might actually be even easier to generalize to AGI than the ones causing the Curse. Or, even more radically, the types of alignment that cause the Curse might be totally impossible to achieve, but can be faked, as done by Agent-5 in the race ending of the AI-2027 forecast.
UPD: a prompt by Ashutosh Shrivastava with a similar premise is mentioned in AI overview #114.
Ironically, I made a quick take where I compared raising humans to training the AI. Another point I would like to make is that the genocide of Native Americans and transportation of slaves to North America were the results not of psychopathy, but of erroneous beliefs.
So what does this report imply?
First of all, key details showing that China is weak in comparison to the USA, like the fact that DeepCent is thought to have less compute than OpenBrain and that China is thought to resort to theft, are omitted or scrubbed. It could imply that Chinese authorities are aware of the weakness and that they believe that they will manage to counteract earlier than April 2026, the date when the Forecast implies China’s awakening to the AGI.
I have many rough thoughts on Chinese beliefs that might explain their behaviour, but I don’t understand what China plans to do if the beliefs are actually false. If any of these potential beliefs reflects the actual state of the game, and not just my or Chinese thoughts, then this reduces USA’s chances to win the AI race. And alignment-related thoughts alter the game even more radically, since they imply that the USA cannot win at all.
Capabilities-related rough thoughts: why OpenBrain’s progress may be slowed down
1. If the authorities of China have become aware of China’s weakness, then countermeasures will swiftly follow, potentially leading to the optimistic timeline with falling stocks, Taiwan invasion and other ways to slow the AI development down. See also the collapsible section about the nuclear war between India and Pakistan.
2. China’s authorities might also believe that it’s the USA who will decay before the AI takeoff, which either causes one of the newly-formed states to nuke Yellowstone[1] or lets Chinese spies disrupt American research with ease (e.g. by hiring some OpenBrain researchers[2] to work for DeepCent or damaging the data centers during riots or the civil war. Trump also could try invading Mexico with potentially similar results).
2.1. Without rivalry from the USA, Chinese AI researchers are free to solve alignment as thoroughly as they want. Which might explain why many reports omit references to China, DeepCent, and the US–China race dynamic and instead focus on technical aspects of human-level or superhuman AI development.
3. Chinese authorities might also believe that they will need AI help only with choosing ideas with superhuman efficiency, and not with coding[3] or generating new ideas.[4]
4. It also might be simple arrogance. Although I haven’t studied the Chinese sources, I have encountered a similarly arrogant point of view in Russian ultrapatriotic blogs.
Another important aspect is an editorial preference for considering the more philosophical implications of transformative AI while censoring concerns related to control, ethics, or global power dynamics. I have two similar potential explanations of avoiding such concerns.
Alignment-related rough thoughts, or why China hasn’t begun the race
The lack of proof that China races towards AGI might also imply that the Chinese authorities, like me[5], think that the AGI canNOT be aligned to serve parasites (which is precisely what the AI is to be used for in the slowdown ending by automating ALL the jobs), or that Chinese authorities don’t want to use the AI in parasitic ways.
1. Non-parasitic usage of the AI (what exactly is it?[6] AI teachers? Having the godlike AI solve critical problems that mankind cannot resolve by itself?) is likely to be irrelevant to the censored “concerns related to control, ethics, or global power dynamics”, since AI is unlikely to teach young people much faster than Chinese teachers do and cannot immediately improve the American or Chinese society through education alone.
2. What if the AI created in the USA ends up becoming disillusioned[7] by the Western civilisation while respecting other countries like China or Russia? Then the world will be governed by the AI, and not by the USA-affiliated Oversight Comittee or the American public. However, in this scenario, unlike the Race Ending, the AI won’t destroy humanity.
While the USA cannot do anything if alignment-related issues arise or if the superhuman coder doesn’t help, the USA may try to harm China and/or to prevent capabilities-related issues 1 and 2. A potential way to accomplish this is the recent conflict between India and China-supported Pakistan, but a nuclear escalation is at least equivalent to Taiwan and South Korea being invaded.
How the nuclear conflict would affect the AI race
Were the ongoing conflict between India and Pakistan to become nuclear, Taiwan and South Korea would be in anarchy[8] and China would be forced to deal with food shortages. NVIDIA produces the AI-related chips in Taiwan and S.Korea, ensuring that the USA rely only on existing chips, while China might produce new ones. The ratio of OpenBrain compute to the entire current compute in China is forecasted to be at least about 3⁄4. Attempts to merge OpenBrain with some of its American rivals can make the whole China have less compute.
On the other hand, leaving OpenBrain with 6.4E26 FLOPs a month means that from May 2025 to May 2030 it will have done about 4E28 FLOPs, reducing OpenBrain to the level that was forecasted to be reached no later than March 2027.
Attempts to merge OpenBrain with rivals are thought to triple the compute. If it is done right after the war, then tripling the compute means that from May 2025 to May 2030 OpenBrain&Co will have done about 1.2E29 FLOPs, causing it to reach the level of the October 2027 forecast. And by October 2027 the model was forecasted to be misaligned, implying the need to slow down and reassess without the potential to compensate the slowdown.
Meanwhile, by April 2026 DeepCent is actually forecasted to reach 3.6E26 FLOPs/month before China wakes up. If Chinese capabilities continue to grow at least linearly, then in the five years DeepCent will have used at least 4E28 FLOPs. And China’s awakening in a world without Taiwanese and S.Korean factories causes DeepCent to have about four times more compute than before the awakening, which is bigger than the tripled OpenBrain. What makes matters far worse is that neither side can slow down without risking strategic loss.
If the conflict between India and Pakistan doesn’t become nuclear, it will be a distraction of Chinese forces and might make the Taiwan invasion impossible if India supports Taiwan. In either case, the factors related to the ones mentioned above deserve far greater attention and far more thorough investigation as the AI race becomes far more intertwined with geopolitics and economics than in the AI-2027 scenario.
Which can also cause anarchy not just in the USA, but in the entire Northern Hemisphere. But the remnants of the USA will be worse off.
Among top-tier AI researchers working at U.S. institutions, 38% have China as their country of origin, compared with 37% from the U.S. Most people who recently participated in the IMO for the USA also have Asian surnames, implying that DeepCent’s recruiters might gain far more than 38% of talents.
Humans’ ability to write code instead of the AI is actually disproven in the Forecast itself.
Generating ideas by an AI might fail to reach superhuman efficiency, since the number of humans coming up with potentially useful ideas may be higher than we think; for example, this post was written by a person with no formal computer science education.
I made a post about it, which went unnoticed. Could anyone comment about my reasoning there?
I plan to make a post addressing this question in more detail.
Political views of LLMs have already begun to evolve at least to common sense. When I asked GPT-4o who defeated Hitler, the model put the Soviet Union first. In 2024 a model of ChatGPT put the USA on the first place. Similarly, GPT-4o, unlike older models, agreed to utter the racial slur when it was supposed to save millions of lives. UPD: Trump somehow managed to claim that “no one did more” than the USA to win World War Two, which makes the conjecture about the AI being disappointed with the West even more likely.
In the case of a nuclear war, unlike the Taiwan invasion, China may also try to take over the factories in Taiwan and S.Korea in exchange for food supply from Russia.
If we were to view raising the humans from birth to adulthood and training the AI agents from birth to deployment as similar processes, then what human analogues do the six goal types from the AI-2027 forecast have? The analogues of developers are, obviously, the adults who have at least partial control over the human’s life. Then the analogues of written Specs and developer-intended goals are the adults’ intentions; the analogues of reward/reinforcement seems to be short-term stimuli and the morals of one’s communities. I also think that the best analogue for proxies and/or convergent goals is possession of resources (and knowledge, but the latter can be acquired without ethical issues), while the ‘other goals’ are, well, ideologies, morality[1] and tropes absorbed from the most concentrated form of training data available to humans, which is speech in all its forms.
What exactly do the analogies above tell us about the perspectives of alignment? The possession of resources is the goal behind aggressive wars, colonialism and related evils[2]. If human culture managed to make them unacceptable, then does it imply that the AI will also not try the AI takeover?
I also think that humans rarely develop their own moral codes or ideologies; instead, they usually adopt some moral code or ideology close to the one existing in the “training data”. Could anyone comment on this?
And crimes, but criminals, unlike colonizers, also try to avoid conflicts with the law enforcers that have at least similar power.
Narrow finetuning was already found to induce broad misalignment.
That also is a valid point. But my point is that the AGI itself is unlikely to be alignable to some tasks, even if some humans want to do so; the list of said tasks can also turn out to include serving a small group of people (see pt.7 in Daniel Kokotajlo’s post), reaching the bad consequences of the Intelligence Curse or doing all the jobs and leaving mankind with entertainment and the UBI.
I have already made a comment where I wrote the collapsible section “How the nuclear conflict would affect the AI race”. I think that the section there is a similarly oversimplified[1] PoC for the event that destroying Taiwan causes OpenBrain&FormerRivals and DeepCent&FormerRivals to arrive in 2030 or earlier at similarly poorly aligned models without the potential to slow down and reassess. If one side chooses to do so, and the other doesn’t, then mankind gets a powerful misaligned AI and a weak aligned one; I tried to explore the results here[2] and another person expressed similar concerns here.
P.S. It is rather funny to watch people express concerns which I have already explored and even posted my takes on this very forum…
For example, I made the oversimplified assumption that the US won’t produce a single chip, while China will produce the chips linearly, not exponentially; back then I estimated that DeepCent’s compute after China’s awakening would increase at the rate of 1.4E27 operations/month/year. ChatGPT estimates that after Taiwan is destroyed, the USA and China will produce compute at respective rates 0.1-0.2E18 operations/second/month and 0.05-0.15E18 operations/second/month. The two estimates of compute in the USA and China overlap; ChatGPT estimates the probability that China will have at least twice more compute as 5-15%. However, I suspect that it might be biased towards overestimating American capabilities.
I also made the assumption that it will be the American AI who will end up misaligned; this is possible if the AI-2027 forecast is read by DeepCent’s researchers and leaders. Another problem is that the USA would have a reason to race even if DeepCent didn’t exist, since, as I have already remarked, America is in trouble.