I don’t think I am. The implied argument is that there aren’t any values at all that most people will agree on, because one imagined and not-evolutionarily-viable Clippy doesn’t think anything other than paperclips have value. Not much of an argument. Funny, though.
When we’ve talked about ‘people agreeing on values’, as in CEV, I’ve always taken that to only refer to humans, or perhaps to all sentient earth-originating life. If ‘people’ refers to the totality of possible minds, it seems obvious to me that there aren’t any values that most people will agree on, but that’s not a very interesting fact in any context I’ve noticed here.
Clippy (or rather Clippy’s controller) may be trying to make that point. But I’m with AdeleneDawner—the people whose values we (terminally) care about may not be precisely restricted to humans, but it’s certainly not over all possible mind-spaces. Several have even argued that all humans is too broad, and that only “enlightenment” type culture is what we should care about. Clippy did indeed make several arguments against that.
But our worry about paperclippers predates Clippy. We don’t want to satisfy them, but perhaps game-theoretically there are reasons to do so in certain cases.
The implied argument is that there aren’t any values at all that most people will agree on, because one imagined and not-evolutionarily-viable Clippy doesn’t think anything other than paperclips have value.
No, that is not the argument implied when making references to paperclipping. That is a silly argument that is about a whole different problem to paperclipping. It is ironic that your straw man claim is, in fact, the straw man.
But it would seem our disagreement if far more fundamental than what a particular metaphor means:
one imagined and not-evolutionarily-viable Clippy
Being “Evolutionarily-viable” is a relatively poor form of optimisation. It is completely the wrong evaluation of competitiveness to make and also carries the insidious assumption that competing is something that an agent should do as more than a short term instrumental objective.
Clippy is competitively viable. If you think that a Paperclip Maximizer isn’t a viable competitive force then you do not understand what a Paperclip Maximizer is. It maximizes paperclips. It doesn’t @#%$ around making paperclips while everyone else is making Battle Cruisers and nanobots. It kills everyone, burns the cosmic commons to whatever extent necessary to eliminate any potential threat and then it goes about turning whatever is left into paperclips.
The whole problem with Paperclip Maximisers is that they ARE competitively viable. That is the mistake in the design. A mandate to produce a desirable resource (stationary) will produce approximately the same behavior as a mandate to optimise survival, dominance and power right up until the point where it doesn’t need to any more.
Does Clippy completely trust future Clippy, or spatially-distant Clippy, to make paperclips?
At some point, Clippy is going to start discounting the future, or figure that the probability of owning and keeping the universe is very low, and make paperclips. At that point, Clippy is non-competitive.
Suppose Clippy takes over this galaxy. Does Clippy stop then and make paperclips, or continue immediately expansion to the next galaxy?
Whatever is likely to produce more paperclips.
Suppose Clippy takes over this universe. Does Clippy stop then and make paperclips, or continue to other universes?
Whatever is likely to produce more paperclips. Including dedicating resources to figuring out if that is physically possible.
Does your version of Clippy ever get to make any paperclips?
Yes.
Does Clippy completely trust future Clippy, or spatially-distant Clippy, to make paperclips?
Yes.
At some point, Clippy is going to start discounting the future, or figure that the probability of owning and keeping the universe is very low, and make paperclips. At that point, Clippy is non-competitive.
A superintelligence that happens to want to make paperclips is extremely viable. This is utterly trivial. I maintain my rejection of the below claim and discontinue my engagement in this line of enquiry. It is just several levels of confusion.
The implied argument is that there aren’t any values at all that most people will agree on, because one imagined and not-evolutionarily-viable Clippy doesn’t think anything other than paperclips have value.
Yes, but if that point happens after Clippy has control of even just the near solar system then that still poses a massive existential threat to humans. The point of Clippy is that a) an AI can have radically different goals than humans (indeed could have goals so strange we wouldn’t even conceive of them) and b) that such AIs can easily pose severe existential risk. A Clippy that decides to focus on turning Sol into paperclips isn’t going to make things bad for aliens or aliens AIs but it will be very unpleasant for humans. The long-term viability of Clippy a thousand or two thousand years after fooming doesn’t have much of an impact if every human has had our hemoglobin extracted so the iron could be turned into paperclips.
t kills everyone, burns the cosmic commons to whatever extent necessary to eliminate any potential threat and then it goes about turning whatever is left into paperclips.
That’s where Clippy might fail at viability—unless it’s the only maximizer around, that “kill everyone” strategy might catch the notice of entities capable of stopping it—entities that wouldn’t move against a friendlier AI.
A while ago, there was some discussion of AIs which cooperated by sharing permission to view source code. Did that discussion come to any conclusions?
Assuming it’s possible to verify that the real source code is being seen, I don’t think a paper clipper isn’t going to get very far unless the other AIs also happen to be paper clippers.
That’s where Clippy might fail at viability—unless it’s the only maximizer around, that “kill everyone” strategy might catch the notice of entities capable of stopping it—entities that wouldn’t move against a friendlier AI.
An earth originating paperclipper that gets squashed by other super intelligences from somewhere else still is very bad for humans.
Though I don’t see why a paperclipper couldn’t compromise and cooperate with competing super intelligences as well as other super intelligences with different goals. If other AIs are a problem for Clippy, they are also a problem for AIs that are Friendly towards humans, but not neccesarily friendly towards alien super intelligences.
That’s where Clippy might fail at viability—unless it’s the only maximizer around, that “kill everyone” strategy might catch the notice of entities capable of stopping it—entities that wouldn’t move against a friendlier AI.
Intended to be a illustration of how Clippy can do completely obvious things that don’t happen to be stupid, not a coded obligation. Clippy will of course do whatever is necessary to gain more paper-clips. In the (unlikely) event that Clippy finds himself in a situation in which cooperation is a better maximisation strategy than simply outfooming then he will obviously cooperate.
It isn’t absolute not-viability, but the odds are worse for an AI which won’t cooperate unless it sees a good reason to do so than for an AI which cooperates unless it sees a good reason to not cooperate.
but the odds are worse for an AI which won’t cooperate unless it sees a good reason to do so than for an AI which cooperates unless it sees a good reason to not cooperate.
Rationalists win. Rational paperclip maximisers win then make paperclips.
Fair point, but the assumption that it indeed is possible to verify source code is far from proven. There’s too many unknowns in cryptography to make strong claims as to what strategies are possible, let alone which would be successful.
I don’t think I am. The implied argument is that there aren’t any values at all that most people will agree on, because one imagined and not-evolutionarily-viable Clippy doesn’t think anything other than paperclips have value. Not much of an argument. Funny, though.
I didn’t evolve. I was intelligently designed (by a being that was the product of evolution).
When we’ve talked about ‘people agreeing on values’, as in CEV, I’ve always taken that to only refer to humans, or perhaps to all sentient earth-originating life. If ‘people’ refers to the totality of possible minds, it seems obvious to me that there aren’t any values that most people will agree on, but that’s not a very interesting fact in any context I’ve noticed here.
It could easily mean minds that could arise through natural processes (quantum computations?) weighted by how likely (simple?).
It could, but it doesn’t.
Clippy (or rather Clippy’s controller) may be trying to make that point. But I’m with AdeleneDawner—the people whose values we (terminally) care about may not be precisely restricted to humans, but it’s certainly not over all possible mind-spaces. Several have even argued that all humans is too broad, and that only “enlightenment” type culture is what we should care about. Clippy did indeed make several arguments against that.
But our worry about paperclippers predates Clippy. We don’t want to satisfy them, but perhaps game-theoretically there are reasons to do so in certain cases.
There is a straw man going on here somewhere. But perhaps not where you intended to convey...
Why do you take the time to make 2 comments, but not take the time to speak clearly? Mysteriousness is not an argument.
Banal translation:
No, that is not the argument implied when making references to paperclipping. That is a silly argument that is about a whole different problem to paperclipping. It is ironic that your straw man claim is, in fact, the straw man.
But it would seem our disagreement if far more fundamental than what a particular metaphor means:
Being “Evolutionarily-viable” is a relatively poor form of optimisation. It is completely the wrong evaluation of competitiveness to make and also carries the insidious assumption that competing is something that an agent should do as more than a short term instrumental objective.
Clippy is competitively viable. If you think that a Paperclip Maximizer isn’t a viable competitive force then you do not understand what a Paperclip Maximizer is. It maximizes paperclips. It doesn’t @#%$ around making paperclips while everyone else is making Battle Cruisers and nanobots. It kills everyone, burns the cosmic commons to whatever extent necessary to eliminate any potential threat and then it goes about turning whatever is left into paperclips.
The whole problem with Paperclip Maximisers is that they ARE competitively viable. That is the mistake in the design. A mandate to produce a desirable resource (stationary) will produce approximately the same behavior as a mandate to optimise survival, dominance and power right up until the point where it doesn’t need to any more.
Suppose Clippy takes over this galaxy. Does Clippy stop then and make paperclips, or continue immediately expansion to the next galaxy?
Suppose Clippy takes over this universe. Does Clippy stop then and make paperclips, or continue to other universes?
Does your version of Clippy ever get to make any paperclips?
(The paper clips are a lie, Clippy!)
Does Clippy completely trust future Clippy, or spatially-distant Clippy, to make paperclips?
At some point, Clippy is going to start discounting the future, or figure that the probability of owning and keeping the universe is very low, and make paperclips. At that point, Clippy is non-competitive.
Whatever is likely to produce more paperclips.
Whatever is likely to produce more paperclips. Including dedicating resources to figuring out if that is physically possible.
Yes.
Yes.
A superintelligence that happens to want to make paperclips is extremely viable. This is utterly trivial. I maintain my rejection of the below claim and discontinue my engagement in this line of enquiry. It is just several levels of confusion.
Wow, I was wrong to call you a human—you’re practically a clippy yourself with how well you understand us! c=@
Well, except for your assumption that I would somehow want to destroy humans. Where do you get this OFFENSIVE belief that borders on racism?
Yes, but if that point happens after Clippy has control of even just the near solar system then that still poses a massive existential threat to humans. The point of Clippy is that a) an AI can have radically different goals than humans (indeed could have goals so strange we wouldn’t even conceive of them) and b) that such AIs can easily pose severe existential risk. A Clippy that decides to focus on turning Sol into paperclips isn’t going to make things bad for aliens or aliens AIs but it will be very unpleasant for humans. The long-term viability of Clippy a thousand or two thousand years after fooming doesn’t have much of an impact if every human has had our hemoglobin extracted so the iron could be turned into paperclips.
That’s where Clippy might fail at viability—unless it’s the only maximizer around, that “kill everyone” strategy might catch the notice of entities capable of stopping it—entities that wouldn’t move against a friendlier AI.
A while ago, there was some discussion of AIs which cooperated by sharing permission to view source code. Did that discussion come to any conclusions?
Assuming it’s possible to verify that the real source code is being seen, I don’t think a paper clipper isn’t going to get very far unless the other AIs also happen to be paper clippers.
An earth originating paperclipper that gets squashed by other super intelligences from somewhere else still is very bad for humans.
Though I don’t see why a paperclipper couldn’t compromise and cooperate with competing super intelligences as well as other super intelligences with different goals. If other AIs are a problem for Clippy, they are also a problem for AIs that are Friendly towards humans, but not neccesarily friendly towards alien super intelligences.
Intended to be a illustration of how Clippy can do completely obvious things that don’t happen to be stupid, not a coded obligation. Clippy will of course do whatever is necessary to gain more paper-clips. In the (unlikely) event that Clippy finds himself in a situation in which cooperation is a better maximisation strategy than simply outfooming then he will obviously cooperate.
It isn’t absolute not-viability, but the odds are worse for an AI which won’t cooperate unless it sees a good reason to do so than for an AI which cooperates unless it sees a good reason to not cooperate.
Rationalists win. Rational paperclip maximisers win then make paperclips.
Fair point, but the assumption that it indeed is possible to verify source code is far from proven. There’s too many unknowns in cryptography to make strong claims as to what strategies are possible, let alone which would be successful.
And we’ve got to assume AIs would be awfully good at steganography.