Suppose there’s a paperclip maximizer that could either be running on a 1 kg computer or a 2 kg computer—say the humans flipped a coin when picking which computer to run it on.
Since the computations are the same, the paperclip maximizer doesn’t know whether it’s 1 kg or 2 kg until I tell it. But before I tell it, I offer the paperclip maximizer a choice between options A and B: A results in 5 paperclips if it’s 1 kg and 0 otherwise, B results in 4 paperclips if it’s 2 kg and 0 otherwise.
It seems like the paperclip-maximizing strategy is to give equal weight (ha) to being 1 kg and 2 kg, and pick A.
What if, instead of a paperclip-maximizer, we had a machine that was designed to maximize the amount of machine-pleasure in the world, where machine-pleasure is “the firing of a certain reward circuit in a system that is sufficiently similar to myself.”
Then it seems yttrium has a point: it is all going to come down to when the machine decides there are two systems in the world, and when it decides there is only one. And there is no “obvious” choice for the machine to make in this regard.
Edit: And so, if we want to make an AI that maximizes (among other things) certain subjective human experiences, we will have to make sure it doesn’t come to some sort of crazy conclusion about what that entails.
I’ve followed you, Manfred, in framing this question in terms of values and right actions. But the original question was framed in terms of expectations and future experiences. Do you think that the original question doesn’t make sense, or do you have something to say about the original formulation as well? I myself am on the fence.
If you make a robot that explicitly cares about things differently depending on how heavy it is, then sure, it can take actions as if it cared about things more when it’s heavier.
But that is done using the same probabilities as normal, merely a different utility function. Changing your utilities without changing your probabilities has no impact on the “probability of being a mind.”
We don’t have to program the machine to explicitly care about things differently depending on how heavy they are. Instead, we program the machine to care simply about how many systems exist—but wait! It turns out it we don’t know what we mean by that! According to yttrium.
That’s because the maximizer is now conditioning not on the probability that he is running on 1kg vs 2kg hardware, but on the probability that you/Omega selected the 1kg/2kg machine to talk to, which sounds intuitively more close to 50⁄50 based on your arguments.
But now suppose that I made the same deal to the maximizer.
There was a post about this point recently somewhere around here, that your solution to the Monty Hall problem should depend about what you know about the algorithm behind the moderators choice to open a door, and which.
If I understand you correctly, your scenario is different from the one I had in mind in that I’d have both computers instantiated at the same time (I’ve clarified that in the post), and then considering the relative probability of experiencing what the 1 kg computer experiences vs experiencing what the 2 kg computer experiences. It seems like one could adapt your scenario by creating a 1 kg and a 2 kg computer at the same time, offering both of them a choice between A and B, and then generating 5 paperclips if the 1 kg computer chooses A and (additionally) 4 paperclips if the 2 kg computer chooses B. Then, the right choice for both systems (who still can’t distinguish themselves from each other) would still be A, but I don’t see how this is related to the relative weight of both maximizer’s experiences—after all, how much value to give each of the computer’s votes is decided by the operators of the experiment, not the computers. To the contrary, if the maximizer cares about the experienced number of paperclips, and each of the maximizers only learns about the paperclips generated by it’s own choice regarding the given options, I’d still say that the maximizer should choose B.
To the contrary, if the maximizer cares about the experienced number of paperclips, and each of the maximizers only learns about the paperclips generated by it’s own choice regarding the given options
Right, that’s why I split them up into different worlds, so that they don’t get any utility from paperclips created by the other paperclip maximizer.
how much value to give each of the computer’s votes is decided by the operators of the experiment, not the computers
I still think that the scenario you describe is not obviously and according to all philosophical intuitions the same as one where both minds exist in parallel.
Also, the expected number of paperclips (what you describe) is not equal to the expected experienced number of paperclips (what would be the relevant weighting for my post). After all, if A involves killing the maximizer before generating any paperclip, the paperclip-maximizer would choose A, while the experienced-paperclip-maximizer would choose B. The probability of experiencing paperclips would be obviously different from the probability of paperclips existing, when choosing A.
Also, the expected number of paperclips (what you describe) is not equal to the expected experienced number of paperclips (what would be the relevant weighting for my post).
If you make robots that maximize your proposed “subjective experience” (proportional to mass) and I make robots that maximize some totally different “subjective experience” (how about proportional to mass squared!), all of those robots will act exactly like one would expect—the linear-experience maximizers would maximize linear-experience, the squared-experience maximizers would maximize squared-experience.
Because anything can be putt into a utility function, it’s very hard to talk about subjective experience by referencing utility functions. We want to reduce “subjective experience” to some kind of behavior that we don’t have to put into the utility function by hand.
In the Sleeping Beauty problem, we can start with an agent that selfishly values some payoff (say, candy bars), with no specific weighting on the number of copies, and no explicit terms for “subjective experience.” But then we put it in an unusual situation, and it turns out that the optimum betting strategy is the one where it gives more weight to world where there are more copies of it. That kind o behavior is what indicate to me that there’s something going on with subjective experience.
Suppose there’s a paperclip maximizer that could either be running on a 1 kg computer or a 2 kg computer—say the humans flipped a coin when picking which computer to run it on.
Since the computations are the same, the paperclip maximizer doesn’t know whether it’s 1 kg or 2 kg until I tell it. But before I tell it, I offer the paperclip maximizer a choice between options A and B: A results in 5 paperclips if it’s 1 kg and 0 otherwise, B results in 4 paperclips if it’s 2 kg and 0 otherwise.
It seems like the paperclip-maximizing strategy is to give equal weight (ha) to being 1 kg and 2 kg, and pick A.
What if, instead of a paperclip-maximizer, we had a machine that was designed to maximize the amount of machine-pleasure in the world, where machine-pleasure is “the firing of a certain reward circuit in a system that is sufficiently similar to myself.”
Then it seems yttrium has a point: it is all going to come down to when the machine decides there are two systems in the world, and when it decides there is only one. And there is no “obvious” choice for the machine to make in this regard.
Edit: And so, if we want to make an AI that maximizes (among other things) certain subjective human experiences, we will have to make sure it doesn’t come to some sort of crazy conclusion about what that entails.
I’ve followed you, Manfred, in framing this question in terms of values and right actions. But the original question was framed in terms of expectations and future experiences. Do you think that the original question doesn’t make sense, or do you have something to say about the original formulation as well? I myself am on the fence.
If you make a robot that explicitly cares about things differently depending on how heavy it is, then sure, it can take actions as if it cared about things more when it’s heavier.
But that is done using the same probabilities as normal, merely a different utility function. Changing your utilities without changing your probabilities has no impact on the “probability of being a mind.”
We don’t have to program the machine to explicitly care about things differently depending on how heavy they are. Instead, we program the machine to care simply about how many systems exist—but wait! It turns out it we don’t know what we mean by that! According to yttrium.
That’s because the maximizer is now conditioning not on the probability that he is running on 1kg vs 2kg hardware, but on the probability that you/Omega selected the 1kg/2kg machine to talk to, which sounds intuitively more close to 50⁄50 based on your arguments.
But now suppose that I made the same deal to the maximizer.
There was a post about this point recently somewhere around here, that your solution to the Monty Hall problem should depend about what you know about the algorithm behind the moderators choice to open a door, and which.
If I understand you correctly, your scenario is different from the one I had in mind in that I’d have both computers instantiated at the same time (I’ve clarified that in the post), and then considering the relative probability of experiencing what the 1 kg computer experiences vs experiencing what the 2 kg computer experiences. It seems like one could adapt your scenario by creating a 1 kg and a 2 kg computer at the same time, offering both of them a choice between A and B, and then generating 5 paperclips if the 1 kg computer chooses A and (additionally) 4 paperclips if the 2 kg computer chooses B. Then, the right choice for both systems (who still can’t distinguish themselves from each other) would still be A, but I don’t see how this is related to the relative weight of both maximizer’s experiences—after all, how much value to give each of the computer’s votes is decided by the operators of the experiment, not the computers. To the contrary, if the maximizer cares about the experienced number of paperclips, and each of the maximizers only learns about the paperclips generated by it’s own choice regarding the given options, I’d still say that the maximizer should choose B.
Right, that’s why I split them up into different worlds, so that they don’t get any utility from paperclips created by the other paperclip maximizer.
Not true—see the Sleeping Beauty problem.
I still think that the scenario you describe is not obviously and according to all philosophical intuitions the same as one where both minds exist in parallel.
Also, the expected number of paperclips (what you describe) is not equal to the expected experienced number of paperclips (what would be the relevant weighting for my post). After all, if A involves killing the maximizer before generating any paperclip, the paperclip-maximizer would choose A, while the experienced-paperclip-maximizer would choose B. The probability of experiencing paperclips would be obviously different from the probability of paperclips existing, when choosing A.
If you make robots that maximize your proposed “subjective experience” (proportional to mass) and I make robots that maximize some totally different “subjective experience” (how about proportional to mass squared!), all of those robots will act exactly like one would expect—the linear-experience maximizers would maximize linear-experience, the squared-experience maximizers would maximize squared-experience.
Because anything can be putt into a utility function, it’s very hard to talk about subjective experience by referencing utility functions. We want to reduce “subjective experience” to some kind of behavior that we don’t have to put into the utility function by hand.
In the Sleeping Beauty problem, we can start with an agent that selfishly values some payoff (say, candy bars), with no specific weighting on the number of copies, and no explicit terms for “subjective experience.” But then we put it in an unusual situation, and it turns out that the optimum betting strategy is the one where it gives more weight to world where there are more copies of it. That kind o behavior is what indicate to me that there’s something going on with subjective experience.