I think that DeepMind realized they’d need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory.
Is this mildly good news on the existential risk front (because the state of the field isn’t actually as advanced as it looks), or extremely bad news (because we live in a world of all-pervasive information warfare where no one can figure out what’s going on because even the reports of people whose job it is to understand what’s going on are distorted by marketing pressures)?
In what sense is this information warfare or even misleading? The second sentence of the blog post says: “AlphaStar was ranked above 99.8% of active players,” which seems quite clear. They seem to have done a pretty good job of making that comparison as fair as you could expect. What do they say or even imply which is highly misleading?
Perhaps they say “Grandmaster level,” and it’s possible that this gives a misleading impression to people who don’t know what that term means in Starcraft? Though I think chess grandmaster also means roughly “better than 99.8% of ladder players,” and the competitive player pools have similar size. So while it might be misleading in the sense that Chess has a larger player pool a smaller fraction of whom are competitive, it seems fairly straightforward.
Sorry, let me clarify: I was specifically reacting to the OP’s characterization of “throw in the towel while making it look like they were claiming victory.” Now, if that characterization is wrong, then my comment becomes either irrelevant (if you construe it as a conditional whose antecedent turned out to be false: “If DeepMind decided to throw in the towel while making it look like …, then is that good news or bad news”) or itself misleading (if you construe it as me affirming and propagating the misapprehension that DeepMind is propagating misapprehensions—and if you think I’m guilty of that, then you should probably downvote me and the OP so that the Less Wrong karma system isn’t complicit with the propagation of misapprehensions).
I agree that the “Grandmaster-level”/”ranked above 99.8% of active players” claims are accurate. But I also think it’s desirable for intellectuals to aspire to high standards of intent to inform, for which accuracy of claims is necessary but not sufficient, due to the perils of selective reporting.
Imagine that, if you spoke to the researchers in confidence (or after getting them drunk), they would agree with the OP’s commentary that “AlphaStar doesn’t really do the ‘strategy’ part of real-time strategy [...] because there’s no representation of causal thinking.” (This is a hypothetical situation to illustrate the thing I’m trying to say about transparency norms; maybe there is some crushing counterargument to the OP that I’m not aware of because I’m not a specialist in this area.) If that were the case, why not put that in the blog post in similarly blunt language, if it’s information that readers would consider relevant? If the answer to that question is, “That would be contrary to the incentives; why would anyone ‘diss’ their own research like that?” … well, the background situation that makes that reply seem normative is what I’m trying to point at with the “information warfare” metaphor: it’s harder to figure out what’s going on with AI in a world in which the relevant actors are rewarded and selected for reporting impressive-seeming capability results subject to the constraint of not making any false statements, than a world in which actors are directly optimizing for making people more informed about what’s going on with AI.
For me, it is evidence for AGI, as it says that we only just one step, may be even one idea, behind it: we need to solve “genuine causal reasoning”. Something like “train a neural net to recognise patterns in in AI’s plans, corresponding to some strategic principles”.
My personal estimate is 10 per cent in 10 years. If it is distributed linearly, it is around 0.2 per cent until the end of 2019, most likely from unknown secret project.
My model of the world doesn’t find this kind of thing very surprising, due to previous reports like this and this, and just on theoretical grounds. I do wonder if this causes anyone who is more optimistic about x-risk to update though.
On the other hand, the information warfare seems to be pitched at a level below what people like us can ultimately rise above. So for example the misleading APM comparison was quickly detected (and probably wasn’t aimed at people like us in the first place) and this analysis of AlphaStar eventually came out (and many of us probably already had similar but unarticulated suspicions). So maybe that’s a silver lining, depending on how you expect the world to be “saved”?
Thanks for asking. The reason artificial general intelligence is an existential risk is because agentic systems that construct predictive models of their environment can use those models to compute what actions will best achieve their goals (and most possible goals kill everyone when optimized hard enough because people are made of atoms that can be used for other things).
The “compute what actions will best achieve goals” trick doesn’t work when the models aren’t accurate! This continues to be the case when the agentic system is made out of humans. So if our scientific institutions systematically produce less-than-optimally-informative output due to misaligned incentives, that’s a factor that makes the “human civilization” AI dumber, and therefore less good at not accidentally killing itself.
I see. In that case, I don’t think it makes much sense to model scientific institutions or the human civilization as an agent. You can’t hope to achieve unanimity in a world as big as ours.
I mean, yes, but we still usually want to talk about collections of humans (like a “corporation” or “the economy”) producing highly optimized outputs, like pencils, even if no one human knows everything that must be known to make a pencil. If someone publishes bad science about the chemistry of graphite, which results in the people in charge of designing a pencil manufacturing line making a decision based on false beliefs about the chemistry of graphite, that makes the pencils worse, even if the humans never achieve unanimity and you don’t want to use the language of “agency” to talk about this process.
Is this mildly good news on the existential risk front (because the state of the field isn’t actually as advanced as it looks), or extremely bad news (because we live in a world of all-pervasive information warfare where no one can figure out what’s going on because even the reports of people whose job it is to understand what’s going on are distorted by marketing pressures)?
In what sense is this information warfare or even misleading? The second sentence of the blog post says: “AlphaStar was ranked above 99.8% of active players,” which seems quite clear. They seem to have done a pretty good job of making that comparison as fair as you could expect. What do they say or even imply which is highly misleading?
Perhaps they say “Grandmaster level,” and it’s possible that this gives a misleading impression to people who don’t know what that term means in Starcraft? Though I think chess grandmaster also means roughly “better than 99.8% of ladder players,” and the competitive player pools have similar size. So while it might be misleading in the sense that Chess has a larger player pool a smaller fraction of whom are competitive, it seems fairly straightforward.
Sorry, let me clarify: I was specifically reacting to the OP’s characterization of “throw in the towel while making it look like they were claiming victory.” Now, if that characterization is wrong, then my comment becomes either irrelevant (if you construe it as a conditional whose antecedent turned out to be false: “If DeepMind decided to throw in the towel while making it look like …, then is that good news or bad news”) or itself misleading (if you construe it as me affirming and propagating the misapprehension that DeepMind is propagating misapprehensions—and if you think I’m guilty of that, then you should probably downvote me and the OP so that the Less Wrong karma system isn’t complicit with the propagation of misapprehensions).
I agree that the “Grandmaster-level”/”ranked above 99.8% of active players” claims are accurate. But I also think it’s desirable for intellectuals to aspire to high standards of intent to inform, for which accuracy of claims is necessary but not sufficient, due to the perils of selective reporting.
Imagine that, if you spoke to the researchers in confidence (or after getting them drunk), they would agree with the OP’s commentary that “AlphaStar doesn’t really do the ‘strategy’ part of real-time strategy [...] because there’s no representation of causal thinking.” (This is a hypothetical situation to illustrate the thing I’m trying to say about transparency norms; maybe there is some crushing counterargument to the OP that I’m not aware of because I’m not a specialist in this area.) If that were the case, why not put that in the blog post in similarly blunt language, if it’s information that readers would consider relevant? If the answer to that question is, “That would be contrary to the incentives; why would anyone ‘diss’ their own research like that?” … well, the background situation that makes that reply seem normative is what I’m trying to point at with the “information warfare” metaphor: it’s harder to figure out what’s going on with AI in a world in which the relevant actors are rewarded and selected for reporting impressive-seeming capability results subject to the constraint of not making any false statements, than a world in which actors are directly optimizing for making people more informed about what’s going on with AI.
For me, it is evidence for AGI, as it says that we only just one step, may be even one idea, behind it: we need to solve “genuine causal reasoning”. Something like “train a neural net to recognise patterns in in AI’s plans, corresponding to some strategic principles”.
In other words, there is a non-trivial chance we could get to AGI literally this year?
My personal estimate is 10 per cent in 10 years. If it is distributed linearly, it is around 0.2 per cent until the end of 2019, most likely from unknown secret project.
My model of the world doesn’t find this kind of thing very surprising, due to previous reports like this and this, and just on theoretical grounds. I do wonder if this causes anyone who is more optimistic about x-risk to update though.
On the other hand, the information warfare seems to be pitched at a level below what people like us can ultimately rise above. So for example the misleading APM comparison was quickly detected (and probably wasn’t aimed at people like us in the first place) and this analysis of AlphaStar eventually came out (and many of us probably already had similar but unarticulated suspicions). So maybe that’s a silver lining, depending on how you expect the world to be “saved”?
Could you elaborate why it’s “extremely bad” news? In what sense is it “better” for DeepMind to be more staightforward with their reporting?
Thanks for asking. The reason artificial general intelligence is an existential risk is because agentic systems that construct predictive models of their environment can use those models to compute what actions will best achieve their goals (and most possible goals kill everyone when optimized hard enough because people are made of atoms that can be used for other things).
The “compute what actions will best achieve goals” trick doesn’t work when the models aren’t accurate! This continues to be the case when the agentic system is made out of humans. So if our scientific institutions systematically produce less-than-optimally-informative output due to misaligned incentives, that’s a factor that makes the “human civilization” AI dumber, and therefore less good at not accidentally killing itself.
I see. In that case, I don’t think it makes much sense to model scientific institutions or the human civilization as an agent. You can’t hope to achieve unanimity in a world as big as ours.
I mean, yes, but we still usually want to talk about collections of humans (like a “corporation” or “the economy”) producing highly optimized outputs, like pencils, even if no one human knows everything that must be known to make a pencil. If someone publishes bad science about the chemistry of graphite, which results in the people in charge of designing a pencil manufacturing line making a decision based on false beliefs about the chemistry of graphite, that makes the pencils worse, even if the humans never achieve unanimity and you don’t want to use the language of “agency” to talk about this process.