I don’t understand this. The post makes a reference to the Open Source Genie Project, whose description says:
The goal of the Open-Source Wish Project is to create perfectly-worded wishes, so that when the genie comes and grants us our wish we can get precisely what we want. The genie, of course, will attempt to interpret the wish in the most malicious way possible, using any loophole to turn our wish into a living hell. The Open-Source Wish Project hopes to use the collective wisdom of all humanity to create wishes with no loopholes whatsoever.
The post is about how to phrase wishes in the context of something that is actively interested in subverting them.
Even beyond that, I think “prior probability of a thing happening” is one kind of outcome pump, but the post does not specify that as the kind of outcome pump it’s talking about. “Minimal matter that needs to be modified”, “Minimal energy expenditure” or “complicated alien set of preferences that will be maximized along with your wish” are also reasonable priors for outcome pumps.
I agree that I wish the post was clearer that certain kinds of outcome pumps might be fine, but I don’t understand the basis for saying the post is false, especially given the explicit reference to the Open-Source Wish Project which directly specifies they are dealing with a malicious genie.
The outcome pump is defined in a way that excludes the possibility of active subversion: it literally just keeps rerunning until the outcome is satisfied, which is a way of sampling based on (some kind of) prior probability. Yudkowsky is arguing that this is equivalent to a malicious genie. But this is a claim that can be false.
In this specific case, I agree with Thomas that whether or not it’s actually false will depend on the details of the function: “The further she gets from the building’s center, the less the time machine’s reset probability.” But there’s probably some not-too-complicated way to define it which would render the pump safe-ish (since this was a user-defined function).
The Outcome Pump is not sentient. It contains a tiny time machine, which resets time unless a specified outcome occurs. For example, if you hooked up the Outcome Pump’s sensors to a coin, and specified that the time machine should keep resetting until it sees the coin come up heads, and then you actually flipped the coin, you would see the coin come up heads. (The physicists say that any future in which a “reset” occurs is inconsistent, and therefore never happens in the first place—so you aren’t actually killing any versions of yourself.)
Whatever proposition you can manage to input into the Outcome Pump, somehow happens, though not in a way that violates the laws of physics. If you try to input a proposition that’s too unlikely, the time machine will suffer a spontaneous mechanical failure before that outcome ever occurs.
I find this a bit confusing to think about. In a classical universe this machine is impossible. It seems like this basically relies on quantum uncertainty. The resulting probability distribution of events will definitely not reflect your prior probability distribution, so I think Thomas’ argument still doesn’t go through. The best guess I have is that it would reflect the shape of the quantum wave-function.
My guess is at a practical level this ends up kind of close to “particles being moved the minimum necessary distance to achieve the outcome”, which I think would generally favor outcomes like “the building explodes”. I definitely don’t think it would favor outcomes like “the fire department arrives 5 minutes earlier” since any macro-level events like that would likely require sampling from much lower amplitude parts of the wave-function (or something, this also doesn’t seem super-compatible with an Everett-interpretation of quantum mechanics, but I can kind of squint and make it work with a Copenhagen-interpretation model).
So I do think I was wrong about Eliezer not specifying how the outcome pump works, but I think his specification still suggests that the result would definitely not be anywhere close to sampling from your prior (which I think might result in reasonable outcome), but would involve some pretty intense maximization and unintended outcomes as you start to put constraints on that prior.
The resulting probability distribution of events will definitely not reflect your prior probability distribution, so I think Thomas’ argument still doesn’t go through. It will reflect the shape of the wave-function.
This is a good point. But I don’t think “particles being moved the minimum necessary distance to achieve the outcome” actually favors explosions. I think it probably favors the sensor hardware getting corrupted, or it might actually favor messing with the firemens’ brains to make them decide to come earlier (or messing with your mother’s brain to make her jump out of the building)—because both of these are highly sensitive systems where small changes can have large effects.
Does this undermine the parable? Kinda, I think. If you built a machine that samples from some bizarre inhuman distribution, and then you get bizarre outcomes, then the problem is not really about your wish any more, the problem is that you built a weirdly-sampling machine. (And then we can debate about the extent to which NNs are weirdly-sampling machines, I guess.)
Does this undermine the parable? Kinda, I think. If you built a machine that samples from some bizarre inhuman distribution, and then you get bizarre outcomes, then the problem is not really about your wish any more, the problem is that you built a weirdly-sampling machine. (And then we can debate about the extent to which NNs are weirdly-sampling machines, I guess.)
This is roughly how I would interpret the post. Physics itself is a bizarre inhuman distribution, and in-general many probability distributions from which you might want to sample from will be bizarre and inhuman.
Agree that it’s then arguable to what degree the optimization pressure of a mature AGI arising from NNs would also be bizarre. My guess is quite bizarre, since a lot of the constraints it will face will be constraints of physics.
Even beyond that, I think “prior probability of a thing happening” is one kind of outcome pump, but the post does not specify that as the kind of outcome pump it’s talking about.
Disagree. The Outcome Pump is explicitly described as conditioning the future trajectory of the universe according to the reset function:
The Outcome Pump is not sentient. It contains a tiny time machine, which resets time unless a specified outcome occurs. For example, if you hooked up the Outcome Pump’s sensors to a coin, and specified that the time machine should keep resetting until it sees the coin come up heads, and then you actually flipped the coin, you would see the coin come up heads. (The physicists say that any future in which a “reset” occurs is inconsistent, and therefore never happens in the first place—so you aren’t actually killing any versions of yourself.)
Also because the Outcome Pump is not sentient, it cannot be actively interested in subverting your wish. Eliezer claims “The Outcome Pump is a genie of the second class. No wish is safe.”, implying that the subversion effect will happen even with the non-sentient, quantilizer-like Outcome Pump. It may happen that future AIs are unsafe, but this will be because they apply too much optimization.
Yeah, see my response to Richard. I was wrong about the Outcome Pump not being specified, but think that your use of “probability” in the top-level comment is still wrong. Clearly the outcome pump would not sample from your prior over likely events.
It would sample from some universal prior over events (this is playing fast-and-loose with quantum mechanics, but a reasonable interpretation might be sampling from the quantum wave-function, if you take a more Copenhagen perspective). Almost any universal prior here would be very oddly shaped, so that indeed you would observe the kinds of things that Eliezer is talking about.
I thought it was sampling from the quantum wavefunction, and still I think my argument works, unless this was a building that was basically deterministically going to kill your mother if you run physics from that point forward, or already had hazardous materials with a significant chance of exploding. I agree that you can’t use your own prior probabilities.
Maybe I’m wrong about how much quantum randomness can influence events at a 5 minute timescale and the universe is actually very deterministic? If it’s very little such that you have to condition very hard to get anything to happen, then maybe the building does explode, but I’m not really sure what would happen.
I liked this discussion but I’ve reread the text a few times now, and I don’t think this fictional Outcome Pump can be sampling from the quantum wavefunction. The post gives examples that work with classical randomness, and not so much with quantum randomness. Most strikingly:
… maybe a powerful enough Outcome Pump has aliens coincidentally showing up in the neighborhood at exactly that moment.
The aliens coincidentally showing up in the neighborhood is a surprise to the user of the Outcome Pump, but not to the aliens who have been traveling for a thousand years to coincidentally arrive at this exact moment. They could be from the future, but the story allows time rewinding, not time travel. It’s not sampling from the user’s prior, because the user didn’t even consider the gas main blowing up.
I think the simplest answer consistent with the text is that the Outcome Pump is magic, and sampling from what the user’s prior “should be”, given their observations.
As I said, the best approximation I have is “move particles the smallest joint distance from my highest prior configuration”. Some particles are in people’s brains, but changing people’s beliefs or intentions seems like it’s very unlikely to happen via this operation, since my guess is the brain is highly redundant and works on ion channels that would require actually a quite substantial amount of matter to be displaced (comparatively). Very locally causing a chemical cain reaction somewhere seems easier, though that’s just a guess.
I am not really sure what happens here, since I think overall physics is highly deterministic even taking into account quantumness, and my guess is for a macro-level outcome here you would need to go very quickly into astronomically low probabilities if you sample from the wave-function, and I don’t trust my reasoning for what happens in 0.00000000000000000000001% scenarios.
My best guess is something pretty close to what Eliezer describes happens, but I couldn’t prove it to you.
my guess is the brain is highly redundant and works on ion channels that would require actually a quite substantial amount of matter to be displaced (comparatively)
Neurons are very small, though, compared with the size of a hole in a gas pipe that would be necessary to cause an explosive gas leak. (Especially because you then can’t control where the gas goes after leaking, so it could take a lot of intervention to give the person a bunch of away-from-building momentum.)
I would probably agree with you if the building happened to have a ton of TNT sitting around in the basement.
Oh, I was definitely not thinking of a hole in a gas pipe. I was expecting something much much subtler than that (more like very highly localized temperature-increases which then chain-react). You are dealing with omniscient levels of consequence-control here.
I don’t understand this. The post makes a reference to the Open Source Genie Project, whose description says:
The post is about how to phrase wishes in the context of something that is actively interested in subverting them.
Even beyond that, I think “prior probability of a thing happening” is one kind of outcome pump, but the post does not specify that as the kind of outcome pump it’s talking about. “Minimal matter that needs to be modified”, “Minimal energy expenditure” or “complicated alien set of preferences that will be maximized along with your wish” are also reasonable priors for outcome pumps.
I agree that I wish the post was clearer that certain kinds of outcome pumps might be fine, but I don’t understand the basis for saying the post is false, especially given the explicit reference to the Open-Source Wish Project which directly specifies they are dealing with a malicious genie.
The outcome pump is defined in a way that excludes the possibility of active subversion: it literally just keeps rerunning until the outcome is satisfied, which is a way of sampling based on (some kind of) prior probability. Yudkowsky is arguing that this is equivalent to a malicious genie. But this is a claim that can be false.
In this specific case, I agree with Thomas that whether or not it’s actually false will depend on the details of the function: “The further she gets from the building’s center, the less the time machine’s reset probability.” But there’s probably some not-too-complicated way to define it which would render the pump safe-ish (since this was a user-defined function).
Ah, rereading the post I think you are right:
I find this a bit confusing to think about. In a classical universe this machine is impossible. It seems like this basically relies on quantum uncertainty. The resulting probability distribution of events will definitely not reflect your prior probability distribution, so I think Thomas’ argument still doesn’t go through. The best guess I have is that it would reflect the shape of the quantum wave-function.
My guess is at a practical level this ends up kind of close to “particles being moved the minimum necessary distance to achieve the outcome”, which I think would generally favor outcomes like “the building explodes”. I definitely don’t think it would favor outcomes like “the fire department arrives 5 minutes earlier” since any macro-level events like that would likely require sampling from much lower amplitude parts of the wave-function (or something, this also doesn’t seem super-compatible with an Everett-interpretation of quantum mechanics, but I can kind of squint and make it work with a Copenhagen-interpretation model).
So I do think I was wrong about Eliezer not specifying how the outcome pump works, but I think his specification still suggests that the result would definitely not be anywhere close to sampling from your prior (which I think might result in reasonable outcome), but would involve some pretty intense maximization and unintended outcomes as you start to put constraints on that prior.
This is a good point. But I don’t think “particles being moved the minimum necessary distance to achieve the outcome” actually favors explosions. I think it probably favors the sensor hardware getting corrupted, or it might actually favor messing with the firemens’ brains to make them decide to come earlier (or messing with your mother’s brain to make her jump out of the building)—because both of these are highly sensitive systems where small changes can have large effects.
Does this undermine the parable? Kinda, I think. If you built a machine that samples from some bizarre inhuman distribution, and then you get bizarre outcomes, then the problem is not really about your wish any more, the problem is that you built a weirdly-sampling machine. (And then we can debate about the extent to which NNs are weirdly-sampling machines, I guess.)
This is roughly how I would interpret the post. Physics itself is a bizarre inhuman distribution, and in-general many probability distributions from which you might want to sample from will be bizarre and inhuman.
Agree that it’s then arguable to what degree the optimization pressure of a mature AGI arising from NNs would also be bizarre. My guess is quite bizarre, since a lot of the constraints it will face will be constraints of physics.
Disagree. The Outcome Pump is explicitly described as conditioning the future trajectory of the universe according to the reset function:
Also because the Outcome Pump is not sentient, it cannot be actively interested in subverting your wish. Eliezer claims “The Outcome Pump is a genie of the second class. No wish is safe.”, implying that the subversion effect will happen even with the non-sentient, quantilizer-like Outcome Pump. It may happen that future AIs are unsafe, but this will be because they apply too much optimization.
Yeah, see my response to Richard. I was wrong about the Outcome Pump not being specified, but think that your use of “probability” in the top-level comment is still wrong. Clearly the outcome pump would not sample from your prior over likely events.
It would sample from some universal prior over events (this is playing fast-and-loose with quantum mechanics, but a reasonable interpretation might be sampling from the quantum wave-function, if you take a more Copenhagen perspective). Almost any universal prior here would be very oddly shaped, so that indeed you would observe the kinds of things that Eliezer is talking about.
I thought it was sampling from the quantum wavefunction, and still I think my argument works, unless this was a building that was basically deterministically going to kill your mother if you run physics from that point forward, or already had hazardous materials with a significant chance of exploding. I agree that you can’t use your own prior probabilities.
Maybe I’m wrong about how much quantum randomness can influence events at a 5 minute timescale and the universe is actually very deterministic? If it’s very little such that you have to condition very hard to get anything to happen, then maybe the building does explode, but I’m not really sure what would happen.
I liked this discussion but I’ve reread the text a few times now, and I don’t think this fictional Outcome Pump can be sampling from the quantum wavefunction. The post gives examples that work with classical randomness, and not so much with quantum randomness. Most strikingly:
The aliens coincidentally showing up in the neighborhood is a surprise to the user of the Outcome Pump, but not to the aliens who have been traveling for a thousand years to coincidentally arrive at this exact moment. They could be from the future, but the story allows time rewinding, not time travel. It’s not sampling from the user’s prior, because the user didn’t even consider the gas main blowing up.
I think the simplest answer consistent with the text is that the Outcome Pump is magic, and sampling from what the user’s prior “should be”, given their observations.
As I said, the best approximation I have is “move particles the smallest joint distance from my highest prior configuration”. Some particles are in people’s brains, but changing people’s beliefs or intentions seems like it’s very unlikely to happen via this operation, since my guess is the brain is highly redundant and works on ion channels that would require actually a quite substantial amount of matter to be displaced (comparatively). Very locally causing a chemical cain reaction somewhere seems easier, though that’s just a guess.
I am not really sure what happens here, since I think overall physics is highly deterministic even taking into account quantumness, and my guess is for a macro-level outcome here you would need to go very quickly into astronomically low probabilities if you sample from the wave-function, and I don’t trust my reasoning for what happens in 0.00000000000000000000001% scenarios.
My best guess is something pretty close to what Eliezer describes happens, but I couldn’t prove it to you.
Neurons are very small, though, compared with the size of a hole in a gas pipe that would be necessary to cause an explosive gas leak. (Especially because you then can’t control where the gas goes after leaking, so it could take a lot of intervention to give the person a bunch of away-from-building momentum.)
I would probably agree with you if the building happened to have a ton of TNT sitting around in the basement.
Oh, I was definitely not thinking of a hole in a gas pipe. I was expecting something much much subtler than that (more like very highly localized temperature-increases which then chain-react). You are dealing with omniscient levels of consequence-control here.