I’m broadly interested in the question, what physical limits if any, will a superintelligence face? What problems will it have to solve and which ones will it struggle with?
Eliezer Yudkowsky has made the claim
“A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.”
I can’t see how this is true. It isn’t obvious to me that one could conclude anything from a video like that without a substantial prior knowledge of mathematical physics. Seeing a red, vaguely circular object, move across a screen tells me nothing unless I already know an enormous amount.
We can put absolute physical limits on the energy cost of a computation, at least in classical physics. How many computations would we expect an AI to need in order to do X or Y. Can we effectively box an AI by only giving it a 50W power supply?
I think there are some interesting questions at the intersection of information theory/physics/computer science that seem like they would be relevant for the AI discussion that I haven’t seen addressed anywhere. There’s a lot of hand-waving, and arguments about things that seem true, but “seem true” is a pretty terrible argument. Unlike math, “seem true” pretty reliably yields whatever you wanted to believe in the first place.
I’m making slow progress on some of these questions, and I’ll eventually write it up, but encouragement, suggestions, etc. would be pretty welcome, because it’s a lot of work and it’s pretty difficult to justify the time/effort expenditure.
I can’t see how this is true. It isn’t obvious to me that one could conclude anything from a video like that without a substantial prior knowledge of mathematical physics. Seeing a red, vaguely circular object, move across a screen tells me nothing unless I already know an enormous amount.
This DeepMind paper describes their neural network learning from an emulated Atari 2600 display as its only input and eventually learning to directly use its output to the emulated Atari controls to do very well at several games. The neural network was not built with prior knowledge of Atari game systems or the games in question, except for the training using the internal game score as a direct measurement of success.
More than 3 frames from the display were used for training, but it arguably wasn’t a superintelligence looking at them.
I can’t see how this is true. It isn’t obvious to me that one could conclude anything from a video like that without a substantial prior knowledge of mathematical physics. Seeing a red, vaguely circular object, move across a screen tells me nothing unless I already know an enormous amount.
And it still doesn’t make any sense. Think about the motion of a helium balloon. Think about the motion of a charged particle in a magnetic field. There’s literally an infinite number of possible formal mathematical models that could include 12 frames of an apple falling. Thing about how enormous the leap of logic is to go from, “This one thing moved” to “All things must move the way that one thing does.” There’s quite simply not enough observations and not enough information in seeing something happen once to prove a theory. I feel like the reason people like this analogy is because an apple falling feels like something we understand, and so it’s easy to imagine something smarter than us understanding it too, but we only understand apples falling because we’ve seen so many things fall.
How much of physics can you generalize from this image? If you want to really get an idea of how hard this problem is, try and tell me how much physics you can learn from this sound clip. It’s the same information as the image, just presented in a format that doesn’t allow you to easily access all of the incredibly difficult learning you’ve already done that allows you to easily interpret images.
That comment you link to walks directly through a correct chain of reasoning, and it has AI-Einstein miraculously picking the correct needle out of an infinite haystack. But it’s a fiction story, and so horrendously improbably things are allowed to happen. The millions and billions of other possible theories that fit the data that tiny-boxed-Einstein could have also invented don’t warrant a mention. How many curves can you draw that correctly fit two data points? There are an infinite number of possible theories and no amount of intelligence is going to allow you count to infinity any faster than anyone else.
it’s a very obvious hypothesis to the right kind of Bayesian.
It’s not at all clear to me what this means given the existence of Aumann’s agreement theorem.
...it has AI-Einstein miraculously picking the correct needle out of an infinite haystack. But it’s a fiction story, and so horrendously improbably things are allowed to happen.
I’ve been making similar complaints for years. And the replies I get are along the following lines:
Skeptic01: X is a highly conjunctive hypothesis. There’s a lot of hand-waving, and arguments about things that seem true, but “seem true” is a pretty terrible argument.
LW-Member01: This is what we call the “unpacking fallacy” or “conjunction fallacy fallacy”. It is very easy to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions.
Skeptic01: But you are telling a detailed story about the future. You are predicting “the lottery will roll 12345134”, while I merely point out that the negation is more likely.
LW-Member02: Not everyone here is some kind of brainwashed cultist. I am a trained computer scientist, and I held lots of skepticism about MIRI’s claims, so I used my training and education to actually check them.
Skeptic01: Fine, could you share your research?
LW-Member02: No, that’s not what I meant!
LW-Member03: Ignore him, Skeptic01 is a troll!!!
...much later...
Skeptic01: I still think this is all highly speculative...
LW-Member03: We’ve already explained to Skeptic01 why he is wrong. He’s a troll!!!
Eliezer Yudkowsky has made the claim “A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.”
This is one of those grandiose and silly claims that gives this site a bad rap. There is no way to prove this statement of faith in Bayesian Superintelligence (BS for short), because: there is no BS (ehm...) around to test it with, and even if there were, the setup itself (BS+webcam) is so ridiculous, it would never be tried. Anyway, as far as I know, Eliezer’s absolute faith in Bayesianism is not shared by anyone else at MIRI/CFAR, at least not nearly as fervently.
I’m broadly interested in the question, what physical limits if any, will a superintelligence face? What problems will it have to solve and which ones will it struggle with?
Eliezer Yudkowsky has made the claim “A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.”
I can’t see how this is true. It isn’t obvious to me that one could conclude anything from a video like that without a substantial prior knowledge of mathematical physics. Seeing a red, vaguely circular object, move across a screen tells me nothing unless I already know an enormous amount.
We can put absolute physical limits on the energy cost of a computation, at least in classical physics. How many computations would we expect an AI to need in order to do X or Y. Can we effectively box an AI by only giving it a 50W power supply?
I think there are some interesting questions at the intersection of information theory/physics/computer science that seem like they would be relevant for the AI discussion that I haven’t seen addressed anywhere. There’s a lot of hand-waving, and arguments about things that seem true, but “seem true” is a pretty terrible argument. Unlike math, “seem true” pretty reliably yields whatever you wanted to believe in the first place.
I’m making slow progress on some of these questions, and I’ll eventually write it up, but encouragement, suggestions, etc. would be pretty welcome, because it’s a lot of work and it’s pretty difficult to justify the time/effort expenditure.
This DeepMind paper describes their neural network learning from an emulated Atari 2600 display as its only input and eventually learning to directly use its output to the emulated Atari controls to do very well at several games. The neural network was not built with prior knowledge of Atari game systems or the games in question, except for the training using the internal game score as a direct measurement of success.
More than 3 frames from the display were used for training, but it arguably wasn’t a superintelligence looking at them.
He elaborates on the claim a bit in this comment.
And it still doesn’t make any sense. Think about the motion of a helium balloon. Think about the motion of a charged particle in a magnetic field. There’s literally an infinite number of possible formal mathematical models that could include 12 frames of an apple falling. Thing about how enormous the leap of logic is to go from, “This one thing moved” to “All things must move the way that one thing does.” There’s quite simply not enough observations and not enough information in seeing something happen once to prove a theory. I feel like the reason people like this analogy is because an apple falling feels like something we understand, and so it’s easy to imagine something smarter than us understanding it too, but we only understand apples falling because we’ve seen so many things fall.
How much of physics can you generalize from this image? If you want to really get an idea of how hard this problem is, try and tell me how much physics you can learn from this sound clip. It’s the same information as the image, just presented in a format that doesn’t allow you to easily access all of the incredibly difficult learning you’ve already done that allows you to easily interpret images.
That comment you link to walks directly through a correct chain of reasoning, and it has AI-Einstein miraculously picking the correct needle out of an infinite haystack. But it’s a fiction story, and so horrendously improbably things are allowed to happen. The millions and billions of other possible theories that fit the data that tiny-boxed-Einstein could have also invented don’t warrant a mention. How many curves can you draw that correctly fit two data points? There are an infinite number of possible theories and no amount of intelligence is going to allow you count to infinity any faster than anyone else.
It’s not at all clear to me what this means given the existence of Aumann’s agreement theorem.
I’ve been making similar complaints for years. And the replies I get are along the following lines:
Skeptic01: X is a highly conjunctive hypothesis. There’s a lot of hand-waving, and arguments about things that seem true, but “seem true” is a pretty terrible argument.
LW-Member01: This is what we call the “unpacking fallacy” or “conjunction fallacy fallacy”. It is very easy to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions.
Skeptic01: But you are telling a detailed story about the future. You are predicting “the lottery will roll 12345134”, while I merely point out that the negation is more likely.
LW-Member02: Not everyone here is some kind of brainwashed cultist. I am a trained computer scientist, and I held lots of skepticism about MIRI’s claims, so I used my training and education to actually check them.
Skeptic01: Fine, could you share your research?
LW-Member02: No, that’s not what I meant!
LW-Member03: Ignore him, Skeptic01 is a troll!!!
...much later...
Skeptic01: I still think this is all highly speculative...
LW-Member03: We’ve already explained to Skeptic01 why he is wrong. He’s a troll!!!
Yes, I’m sure that’s what it looks like from inside your head.
I’m guessing that the super-intelligence would deduce more from the details of the webcam than from the details of a short film or single image.
It couldn’t know whether the image represented something real or a hypothetical construction.
This is one of those grandiose and silly claims that gives this site a bad rap. There is no way to prove this statement of faith in Bayesian Superintelligence (BS for short), because: there is no BS (ehm...) around to test it with, and even if there were, the setup itself (BS+webcam) is so ridiculous, it would never be tried. Anyway, as far as I know, Eliezer’s absolute faith in Bayesianism is not shared by anyone else at MIRI/CFAR, at least not nearly as fervently.