Human-level Diplomacy was my fire alarm
Epistemic status: AI is my hobby, not my job.
Meta just released their new Diplomacy AI, Cicero. Sure, it was several different modules kludged together in a somewhat clever way, with no fundamentally new ideas or massive scaling, but it was damn impressive.
What surprises me most about Cicero is its ability to go all the way from data input to natural language output with relatively high reliability. The model makes a battle plan, uses natural language to interface with humans, then uses the information gained from said interface to alter its plan. And it does this consistently enough to outmatch the average human Diplomacy player. Maybe it doesn’t represent anything truly new in terms of architecture or scaling. But that’s exactly why I’m so impressed.
It doesn’t have the “human spark”. But the lesson that we’ve been learning over the past few decades is that very impressive tasks, from human level Chess to Go to art, can be accomplished without said spark. I’m starting to suspect that human-level AGI is entirely possible with something like Cicero. If systems like Cicero can remain coherent over longer time scales, we could be seeing the automation of many complex tasks, including the holy grail that is AI research. Writing scientific papers? I’m already starting to use automated hypothesis discovery. Let’s break down how the median academic paper actually gets written.
Somewhere in the universe, an academic feels publication pressure. They start looking at their connections, resources, and potential funding.
They start bouncing ideas and try to figure out how to write the highest impact paper they can with the lowest use of resources. Analysis of other peoples’ data is very tempting at this point. AI can probably be used to search through public data for underutilized but useful datasets.
Funding applications are written. This part can be automated.
They start pulling threads, using their specialty knowledge to come up with the best novel hypotheses they can, and picks the one with the lowest p-value. No multiplicity testing, of course. Automated hypothesis discovery can help greatly here. A language model can probably also be used to estimate future citations for each hypothesis-paper.
Hypothesis + data → set of core ideas that is then expanded into an abstract. I think current transformer models can already do this.
The paper is actually written. Fine-tuned models have published short papers already. A rough template, multiple generations, and a bit of human help can probably crank out a full-length paper without suspicion.
The paper sent to an impact-factor 3 journal, gets bounced a few times between their editor and the academic and is finally accepted after the recommended citations get added. Editing a paper according to a series of recommendations sounds like something that specialized transformer models are doing already.
Publication! Remember that the median paper has 0 citations.
Recommendations for follow-up experiments can also be obtained with AI analysis.
I fully anticipate an explosion of low-impact papers in the next few years, especially from China. With even worse English than normal. But how long until this starts improving? A bad paper is one with a boring/flawed hypothesis and bad results (p=0.050, after messing with outliers). The same data set and methods, but a more insightful hypothesis, can generate something a good (actually contributes to human knowledge, acts as a stepping stone for future researchers, etc.) paper. I have crawled down a rabbit hole of an interesting idea, realized it didn’t pan out halfway through, and then had to keep working with it since I still had to finish that paper and it was too late to start over. In scientific research, quantity is quality, since you can just chuck the bad ideas in the trash and keep the gems. AI systems tend to go from very subhuman to superhuman performance very fast. I expect a significant increase in scientific productivity over the next 5 years, much of which is attributable to AI.
And of course, the same applies to AI research. Some say that the last 30 years of AI research have been researchers stumbling around in the dark, trying different paradigms with trial and error until they hit upon DL and scaling. I don’t think that’s quite true, but surely the testing of increasingly experimental architectures can be automated.
Part of the reason I’m writing this is that I promised myself I would “pull my fire alarm” when AI achieved human-level open press Diplomacy. But I also want it on the record that I’m short on AGI timelines (50% by 2030). I’ve also shifted my investments toward AI stocks accordingly.
Predictions:
Fully automated cab service in most tier-1 Chinese cities, with 80%+ coverage. I will also encounter one at least once a week. 80% by 2026
50%+ of a paper I’ve published (by word count) is written by an AI. 80% by 2025, conditional on my continued work in academic-adjacent fields.
An AI paradigm as performance-enhancing as transformers is discovered by AI search. 30% by 2030.
Good news is that the median academic paper contributes approximately 0 to scientific progress; if all we were automating was the median academic paper then I wouldn’t expect things to accelerate much at all.
(Bad news is that actually important research, such as what gets done in industry & by some academics, will accelerate too. And that acceleration will beget further acceleration, and so on until we blow right past human level abilities.)
The median academic paper is a good hypothesis away from actually making incremental scientific progress. The top 10% of papers written in such a manner, let’s call it “paper milled”, inspire exploration and follow-up in other scientists. They can become highly cited. They can save other people a LOT of time.
Let me give an example of something I’m working on. My team was looking at Clonal Hematopoiesis of Indeterminate Origin (CHIP), non-disease-causing mutations in cancer-driving genes that appear in white blood cells as people age. They were causing interference in our cancer panels. We were trying to disentangle the effects of selection pressures and mutation hotspots on the appearance of CHIP. How much was due to higher replication rates, and how much was due to the higher likelihood of mutation in specific locations? We were going to do a time-consuming analysis of our data before I found several twin studies, which found no significant difference between identical and fraternal twins (same sex) in terms of CHIP correlation. Turns out, mutation hotspots didn’t play a major role (there are a lot of caveats here, obviously; I’m oversimplifying).
These papers, which were simple analyses done on an existing data set, probably saved us a hundred man-hours. They may have been a formulaic analysis of other peoples’ data, but are still fundamentally good and useful papers that advance science forward. Soon, I’ll be using AI to sort through the enormity of papers to find useful ones like these. I’ll also be using better hypotheses with the aid of AI.
AI is going to matter. It’s already mattering. This is important for the average scientist.
I wonder how it would update its strategies if you negotiated in an unorthodox way:
“If you help me win, I will donate £5000 across various high-impact charities”
“If you don’t help me win, I will kill somebody”
Think about what happens in the dataset of human games where such conversations take place. It probably adds more uncertainty to the predicted actions of players who say these things.
I mean, what would you do if you saw such messages in a game you’re playing? Probably assume they’re mentally unstable and adjust accordingly.
Or: What if it finds out that all the humans are doing a strategy of “interrogate everyone to find out who the bot is, then gang up on the bot.” How does it react?
It was reported that high level diplomacy players have a different game-theoretical situation, because they all know eachother by (user)name. So if DiplomacyGrandmaster69 goes up against TheDiplomancer, they know their games will be publicly streamed, and the other high level players will see how honest they really are. Whereas casual players are playing a single-shot prisoner’s dilemma, the pros are playing an iterated prisoner’s dilemma, and that makes a difference.
I wonder what would happen if CICERO were placed in repeated 6-human-one-AI showmatches where everyone know which one was the AI. How would it fair?
I can’t remember the exact source, but I believe that CICERO was optimized with the expectation of anonymity. In fact, all players in the games CICERO played were anonymous. CICERO was optimized with the assumption that other players would have no knowledge of its past history (and hence expected behavior). Versions of CICERO that were optimized with the assumption that other players would treat it according to its past history were explicitly noted as being more vindictive.
Now that I think about it, this probably gave CICERO a significant advantage. Most human games are played with player names visible. Anonymous play is thus a deviation from the standard metagame. The Meta team noted that players played more vindictively than optimal and that CICERO got an advantage for being less vindictive. Since these were top players, it implies that the human players simply didn’t fully adjust to the anonymous format. I don’t recall any CICERO games with public names in the paper; maybe the results were less impressive?
imagenet was my fire alarm. and alphago. and alphazero. or maybe gpt3. actually, the fire alarm hadn’t gone off until alphafold, at which time it really started ringing. sorry, I mean alphafold 2. actually, PaLM was what really convinced me agi was soon. well, I mean, not really soon, but hey, maybe if they scale RWKV or S4 or NPT and jam them into a MuZero it somehow won’t be agi, despite that it obviously would be. I wonder how the EfficientZero followups are looking these days? don’t worry, agi can’t happen, they finally convinced me language models aren’t real intelligence because they can’t do real causal reasoning. they’re not good enough at using a causal information bottleneck and they don’t have the appropriate communication patterns of real physics. they’re prone to stereotyping and irrational, unlike real intelligence,
at this point if people aren’t convinced it’s soon, they’re not going to be convinced until after it happens. there’s no further revelation that could occur. [it’ll be here within the year] → {edit 1yr later: well, some folks have included gpt4 in “agi”, and gpt4 was released after I said this, but I think actually what I was really expecting didn’t happen quite as I expected it}, and I don’t know why it’s been so hard for people to see. I guess the insistence on yudkowskian foom has immunized people against real life slow takeoff? but that “slow” is speeding up, hard.
anyway, I hope y’all are using good ai tools. I personally most recommend metaphor.systems, summarize.tech, and semanticscholar.
I disagree, in fact I actually think you can argue this development points the opposite direction, when you look at what they had to do to achieve it and the architecture they use.
I suggest you read Ernest Davis’ overview of Cicero. Cicero is a special-purpose system that took enormous work to produce—a team of multiple people labored on it for three years. They had to assemble a massive dataset from 125,300 online human games. They also had to get expert annotations on thousands of preliminary outputs. Even that was not enough.. they had to generate synthetic datasets as well to fix issues with the system! Even then, the dialogue module required a specialized filter to remove nonsense. This is a break from the scaling idea that says to solve new problems you just need to scale existing architectures to more parameters (and train on a large enough dataset).
Additionally, they argue that this system appears very unlikely to generalize to other problems, or even to slight modifications of the game of Diplomacy. It’s not even clear how well it would generalize to non-blitz games. If the rules were modified slightly, the entire system would likely have to be retrained.
I also want to point out that scientific research is not easy as you make it sound. Professors spend the bulk of their time writing proposals, so perhaps AI could help there by summarizing existing literature. Note though a typical paper, even a low-value one, generally takes a graduate student with specialized training about a year to complete, assuming the experimental apparatus and other necessary infrastructure are all in place. Not all science is data-driven either, science can also be observation-driven or theory-driven.
How is...
...consistent with...
...?
Doesn’t AGI imply the latter?
I’m using a weird definition of AI here, basically “an AI that can do my job”. I’m imagining a cobbled-together system of transformers that individually automates everything I can do, thereby replacing most of the information jobs like coding, scientific research, and advertising. So in a lot of the worlds where AGI happens, there’s no hard takeoff. AIs are helping do AI research, and maybe labor isn’t a major limiting factor in AI development anymore. But there isn’t a >1 OOM increase in AI research output from AI.
This also means that I think in most of the 30% there is no hard takeoff. Some low-hanging fruit is picked by machines, but not enough for a FOOM.
Thanks for bringing up the contradiction, though. I really need to go back and clarify a lot of my statements.
It’s not my fire alarm (in part because I don’t think that’s a good metaphor). But it has caused me to think about updating timelines.
My initial reaction was to update timelines, but this achievement seems less impressive than I thought at first. It doesn’t seem to represent an advance in capabilities; instead it is (another) surprising result of existing capabilities.
Isn’t yet another surprising result of existing capabilities evidence that general intelligence is itself a surprising result of existing capabilities?
That is too strong a statement. I think that it is evidence that general intelligence may be easier to achieve than commonly thought. But, past evidence has already shown that over the last couple of years and I am not sure that this is significant additional evidence in that regard.
On one hand, I agree that nothing really special and novel is happening in Cicero. On the other hand, something about it makes me feel like it’s important. I think it’s the plan->communicate->alter plan->repeat cycle partly taking place in English that intuitively makes me think “oh shit, that’s basically how I think. If we scale this up, it’ll do everything I can do”. I don’t know how true this actually is.
I vaguely recall feeling something like this when a general-purpose model learned to play all the Atari games. But I’m feeling it a lot more now. Maybe it’s the fact that if you showed the results of this to pre-GPT2 me, I’d think it’s an AGI, with zero doubt or hesitation.