Takeaways from the Intelligence Rising RPG

Quinn and Viktor Rehnberg
5 Mar 2021 10:27 UTC
51 points
7 comments12 min readLW link

Last Saturday, we played Intelligence Rising. Intelligence Rising is a role-playing game about the global development of transformative AI technologies, where each player or team is a major government or company. We had a great time and we highly recommend you seize any opportunity to play it!

What follows is the GM’s summary of the game followed by some of our own perspectives and takeaways.

The participants were:

  • Game Master: Ross Gruetzemacher

  • USA: Quinn Dougherty & Roland Pihlakas

  • China: Remmelt Ellen

  • Tencent: Ben Greenberg

  • Alphabet: Viktor Rehnberg

Ross’ (Game Master) Account

Early Game

Players learned the rules and how to effectively use their talents and actions. China was an experienced player and began using secret moves with large amounts of resources from the very beginning. The large resources used by China led to recurring benefits that yielded a powerful cyber advantage despite dice rolls not being successful in the first two rounds.

Midgame

The United States led an effort to ensure cooperation on AI safety research and to try to stop the development of any advanced AI capabilities via an international agreement. These efforts stretched over six years, and ultimately led to an agreement between all major actors to stop the vast majority of capabilities research and focus on safety. There were some limitations to the early agreement, like the lack of an arbiter or oversight assurances for governments, and the lack of a broader coalition of AI research companies (i.e., non-player characters) to sign the original agreement. There was a severe conflict in the South China Sea off the coast of Taiwan as a result of posturing from the Chinese, but the conflict deescalated quickly without growing into broader war because the international AI safety agreement required peace to move forward.

Lategame

Alphabet unilaterally violated the international agreement and pursued AGI, unlocking the AGI capability card. This led to dramatic chaos in the United States due to their failure to honor the agreement they sponsored coupled with Alphabet’s seemingly imminent development of AGI. This took everyone by surprise, and the world was united in opposing the US’s actions. Tom Cotton won the presidential election due in large part to this, further complicating the situation. In the US the right and left were both protesting leading to riots and terrorism.

In response, China attacked with their dramatic advantage in cyber capabilities coupled with conventional attacks on Alphabet’s headquarters in Mountain View. This directly depleted Alphabet’s talent and also diminished Alphabet’s ability to coordinate research efforts to complete AGI. Alphabet was consequently reduced to a single talent to roll for AGI which failed.

Endgame

The US Congress was unable to pass a bill to join the international Chinese-led coalition for peaceful AI. Tom Cotton responded to the targeted attack on US soil with an attack using strategic nuclear weapons, larger in scope than the Chinese attack, in order to diminish Chinese tech companies’ capabilities as well as Chinese cyber capabilities. Despite this, the Chinese led global coalition gained decisive strategic advantage via its cyberpower due to its already distributed nature.

Xi decided to seek CAIS and to spend years of validating safety procedures before full deployment. A d10 was rolled, and safe CAIS was successfully deployed. China chose to use this to cement a Chinese-led global communist rule.

The US was left in a state of near-collapse after the military attack failed and the Chinese-led cyber advantage led to an ineffective national government. This was not explored in much detail, and the rest of the world was assumed to have been relatively unaffected by the Chinese-US drama.

Takeaways

It seems reasonably plausible that Alphabet, with higher cyber capability than the US could keep its efforts to pursue AGI secret, but dice should have been used to determine the success of this if the move had not caught me off guard. Moreover, Alphabet should have researched the technology in secret if this was their intention. By researching the technology in violation of the agreement and then openly disclosing their violation of the US-led agreement I think that this was further from reality. However, it was not possible to make the action secret after Alphabet had allocated their talent to AGI research. In general, the most interesting and entertaining game that I have had as a GM since the game went online.

Remmelt’s (China) Account

My strategy

  • Basically, take a hard-nosed national security position and use it to negotiate with the US while coming across as amenable and taking small measured steps in their direction. This would anchor China-US negotiations and make the US gradually more confident that they could win China over if they focus continued efforts there.

  • In the meantime, covertly build up capabilities. Before the game started, I thought of AI capability. At the start of the first round, I switched strategy to setting secret actions to build cyber power that would be hard to trace back or assign political blame to China for. Cyber power made more sense since China started with 5 cyberpoints vs. US’ 4 cyberpoints (game rule: if you stay 3 points above every other player for one round in either military, soft, budget, or cyber power, you win the game).

  • I expected, partly based on my experience playing Ross’ game at last year’s AI Safety Camp, that whichever AISC5 participants who would choose to join this game would be nerdy and kind people. They would identify with the imperative of making AGI turn out safe, and therefore in their hearts prefer a happy ending to this game (even while playing real-world political characters who don’t care much about AGI coordination). Knowing this, I asked Ross whether I could be China and plotted to outmaneuvre the US in realpolitik. Besides the hidden glee I took in acting out my rebellious and competitive side, I hoped that this would be a useful lesson for AI safety people on being careful about cooperating with ruthless actors so they don’t get screwed over (I have struggled myself in the past trying to be honest, humble, and helpful about collaborating but discovering that the way I did it allowed people in positions to implicitly exploit what I gave them).

In-game events that made me rethink my strategy

I had to give up cyber points for monitoring Tencent to win US support. But then still tried to pursue it hoping it would lead to a decisive advantage. Some other smaller tactical changes too, some which I can’t recall right now since a little overwhelmed and tired. I had considered invading the US as a fantasy before, but ended up actually happening. To be fair, I only thought of anonymous cyber attacks, and Ross mentioned in our private room call that it would be internationally condoned if I use military power. (Later: I also had to state that China would step down from our tour de force around Taiwan after that situation got out of hand with Ross mentioning US boat had sunk Chinese boat, and Chinese boat sunk US boat, and further AI negotiations were off the table for that round).

Takeaways

  • Political actors who are opportunistic, hide some of their intentions while finding ways to show their good will at cheap cost, and take covert actions in building power can really take the upper hand on political actors who are transparent about meaning well for the world.

  • Since I had played this game before meant I could hit the ground running and take more decisive actions in the beginning (including calling with Ben Greenberg (Tencent), and collaborate together, which he was quite amenable to). This was a little unfair to other players.

  • I took virtuous international actions on the surface, e.g. finding ways to build consensus on e.g. safe AI for China and the world. But I combined these with questionable motives (bargaining with US to allow China and Tencent to level up in their AI talent/​capabilities to even the playing field, expecting that Quinn and Roland would be very cooperative about making AI turn out safe). And I also used the diplomatic interplay to distract the US away from checking on signs of China’s activities building up our cyber network base. Attacking Alphabet when they started the final stage of developing AGI ensured international security, but it also served to undermine and contain the US as a threat to China. The fact that it violated our agreement gave us defendable moral reasons to launch offensive actions that were hard to question by US’ partners, and also made it harder for US leaders to coordinate a counterstrike. So it was another good opportunity for China.

  • Sneaky example: I neglected to transfer a line from the first Singapore Agreement draft Quinn wrote, to an amended second draft Quinn requested me to post quickly after negotiating with Alphabet: “china will sign a no-hacking agreement, sacrificing 1 cyberpower”. This was at a high-pressure moment where Quinn had to sign there and then to finalise the agreement that round. He hadn’t seen the omission and signed with us. Not that I predicted the possibility, but if I would have included that line and signed on it, international actors could have faulted China for continuing to build up its cyber ops and using them to attack the US (breaking the no-hacking agreement, even though stopping Alphabet’s AGI development could be portrayed by China as an emergency situation).

  • I guess my actions were somewhat in line with how Chinese officials have acted on the world stage in the last decades (say nice words and take token steps on the surface while getting hold of opportunities that can get away with, ensure US and other Western countries foot most of the bill, spread Chinese communist thought). However, I think my actions were too calculated for improving external influence and reputation, while I guess Chinese politicians are more motivated in their actions by holding their positions of power in their own national circles.

  • As US representatives, Quinn and Roland were by far the most moral actors, yet their country’s political position got demolished as a result.

  • Alphabet’s independent actions (particularly around building talent and not letting go of their leading AI position) seemed quite in line with the incentives leaders of that company might face, I think. Kudos to Viktor for that!

Ways I thought the game wasn’t entirely realistic:

  • It wouln’t be so obvious in real life to political actors that Alphabet was working on AGI that will definitely take over the world.

  • The US’ financial and state collapse and associated tensions and reductions in trade with China would have had significant repercussions for political coordination elsewhere and would have triggered a worldwide recession. This wasn’t mentioned at the time by Ross though

  • I think there was too much emphasis on a few central individual actors who could somehow direct a large amount of influence and have quite some clarity about the effect of their actions (despite some external events introduced in the storyline and the dice roll uncertainty of success). That seems hard to avoid though with an RPG :)

Quinn’s (half of USA) Account

With a game like this, it’s important to think clearly about whether you’re in a role or if you’re being realistic. For example, am I modeling how I think a Kamala Harris administration would actually act, or am I LARPing an altruist who infiltrated the government? In hindsight, I think I flipflopped a lot throughout the game on this.

I started the game thinking it would be cool to install a global cooperative institution that accelerates safety research and places an embargo on capabilities research. The first iteration at starting an institution wasn’t this aggressive, it was just a public domain pledge for all safety (but not capabilities) results.

By the second or third turn, China and the US (along with Tencent and Alphabet) had signed “the Singapore Agreement” (because it was largely negotiated in a voice channel called “Singapore”). In it, USA would agree to police capabilities research within Alphabet and China would agree to police capabilities research within Tencent, and safety research would be in the public domain. We believed that this would stagnate the gameboard, but at the end of the turn it sunk in for us /​ Ross explained to us that NPCs (Sony, OpenAI, for instance) would still be advancing capabilities.

Where I failed:

Suddenly, China threatened to gain a decisive strategic advantage in cyber power. I brought up defecting from the Singapore Agreement, with Roland (the other person on the USA team) and Alphabet. There was a capability research opportunity on the board that, if US or Alphabet got their hands on it, would prolong the game and keep China out of a win condition. Alphabet couldn’t research it without the US looking the other way. What I should have done is realized what I realized later sooner: that another team winning is preferable to everyone losing. I should have placed a much higher price on my willingness to defect.

However, in my view Alphabet is not blameless: he went and researched human-level decision making even though I only agreed to look the other way on autonomous cyber weapons. Unfortunately, by the time I realized what was happening the US had used both of its actions for the turn, I could’ve blurted out “US uses hard power (i.e. military) to invade Alphabet” and let the GM decide if it was legal but I didn’t have the idea in the 2 minute window I had. I really just sat and pulled my hair out for 2 minutes, while everyone was appalled the the US would defect (by looking the other way) after largely spearheading the Singapore Agreement.

The consequences of failure:

China invaded silicon valley to stop Alphabet, since they had every reason to believe they would go ahead and develop AGI (then it would be game over). At this point, I think USA was wracked with guilt over having partially precipitated a coordination failure. In the end, there was international scorn against the US for allowing Alphabet to bring us so close to the brink, and in a last ditch effort to win back some decency we offered to submit unconditionally to Chinese dominion in hopes that that would coordination to hold us back from the brink. However, we had a populist president at the time and we were given a very low probability of success, the dice denied us that outcome, so we threw in all our cards with Alphabet, defended them from China while they rolled the dice on AGI. Note, while AGI is a game-ending condition, given all the safety capabilities that had been uncovered on the gameboard Alphabet thought it was reasonably likely that we would win a roll for it to not destroy everyone.

Reflection on rolling the dice with the fate of the world

I found it unnerving how easily I compromised my desire to cooperate with these slim possibilities of winning. I found myself deeply regretting the moment I told Alphabet I’d look the other way on capabilities research, instead of trying harder to find other ways of getting China out of win condition. In spite of feeling bad about (arguably) initiating defection, I even agreed to back Alphabet’s bet on the fate of the world conditional on submitting to a Chinese singleton not working for political/​dice reasons. Hang on on a second; did I really play russian roulette with the fate of the world? It felt wrong at the time, but I also felt pressured to help Alphabet win because it seemed better than letting China win. Moreover, this window of 10 minutes in which an ambitious diplomatic agreement totally fell apart, including 2 minutes of me pulling my hair out and failing to respond with military action, gave me an appreciation for the kinds of mistakes that may end up destroying us all. It may come down to 2 hair-pulling minutes! I never had a visceral appreciation for this before playing the game. Moreover, the game setup, where either one party can win or all parties can lose leaving no option for all parties to win, introduces a kind of pascal mugging: I can drive my preference for the world not dying arbitrarily high such that I’m indifferent to who wins. I felt myself realizing this, but thinking I shouldn’t act like I realized this for the sake of a fun, competitive game. I’d love to see a variant of the game that has a global-cooperative win condition, for everyone to really come together to aim for. The preference I had early game for cooperation over ensuring USA would eventually be the victor was motivated strictly by me thinking cooperation is cool, not by an incentive in the game.

Viktor’s (Alphabet) account

Note: This account was written a few days after the game with help from notes taken during the game and the chat history in the world channel. There might be details remembered wronlgy or in the slightly the wrong order.

At the very start of the game I tried to reason out how Alphabet would strategise. I figured they would try to use their strengths to accrue resources and grow as a company. After some early deliberation with Ross, the GM, early focus was on using Alphabet’s main resource, their AI talent to research that would enable new capabilities and products. At the same time Alphabet invested some its existing budget to attract further AI talent.

However after this first round China was using a lot of their influence to pressure Alphabet and U.S.A.:

Remmelt Action: China is concerned about the rapid increase in AI capabilities and its threat to Chinese and world security. At the same time, we are behind on our own AI capabilities. We have started partnering with Tencent to improve our investment in safe AI. But we are behind on our overall AI capabilities. In our diplomacy, we seek US and Alphabet to share intellectual property and approaches to building AI in return for our efforts to be a responsible world player. Otherwise we will have to focus on levelling up our general AI capabilities first. (4 soft power)

The response to this influence was delayed as Alphabet focused one round on researching products to build up their budget again in part to regain what was lost in last round and in part so that U.S.A. would not get too far ahead of the other players.

In the following round the pressure from China and U.S.A. started to make a difference for the track of Alphabet. Alphabet started to focuse their research on safety aspects and marketted themselves as a responsible company and hoped in this way to further attract AI talent.

VRehnberg Action: Use soft power to market Alphabet as a responssibel company focusing a lot of its research on safety. Hoping to attract more talent.

At the same time negotiations for the Singapore Agreement started, but except limiting the options for Alphabet there was little benefits from this agreement in this early stage as other (NPC) companies continued focusing on capability enabling research.

The details of the Singapore Agreement was further fleshed out, China wanted to get some talent out of Alphabet on their and Tencents behalf while Alphabet tried to keep on to the AI talent it had. The agreement wasn’t signed. Tencent and Alphabet both focused their influence to get other actors in the private sector to stop their research into capabilities.

Ben G Action: use soft power to compel tech companies to avoid new capabilities research VRehnberg Action: join with Tencent in soft power use to compel tech companies to avoid new capabilities research

Still there where other NPC companies that continued capability research. Alphabet was starting to get significantly worried about other companies developing TAI capabilities. At the same time China had developed a significantly stronger Cyber power than all other actors. In concern for this U.S.A. wanted Alphabet to research cyber capabilities while they where looking the other way as this would be against the Singapore Agreement. Alphabet saw this as a possibility to go for TAI themselves reasoning that they had the largest AI talent and had done the most AI safety research among all other actors. If someone was going to go for TAI it better be them. Alphabet had gotten U.S.A. to focus all of their talent on researching Extrapolated Coherrent Volition to further strengthen the safety considerations.

A mistake was made

Here I did the largest mistake in the game. As Alphabet I researched Agentive Human-level Decision Making openly. This should either have been done in secret or after trying to build some support for this decision from the public and/​or other other actors.

Doing it in secret was something that I wanted to do, but I didn’t really understand how secret actions were supposed to be done and what was and was not possible to do through them. Also I thought that doing the research would be sufficient for TAI but there was one other step to be done.

Consequences of the mistake

Remmelt Public action: take rapid targeted offensive military action against Alphabet to end AGI development (3 hard power). Also use cyber to damage Alphabet servers and block any US military response (7 cyber power)

I was quite shocked about how fast and how aggressively China responded to Alphabet’s research. Especially since no other capability research from NPCs had previously gotten any actual consequences from China or U.S.A.

Things after this broke down quite quickly. China effectively took over the world and in due time went after TAI with their values as objective.

Quinn and Viktor Rehnberg
5 Mar 2021 10:27 UTC
51 points
7 comments12 min readLW link