WARNING: This comment contains explicit discussion of an information hazard.
Imagine that you have two identical paperclip maximizers
I decline to do so. What imaginary creatures would choose whose choice has been written into their definition is of no significance. (This is also a reply to the comment of FeepingCreature you referenced.) I’m more interested in the practical question of how actual human beings, which this discussion began with, can avoid the pitfall of being taken over by a utility monster they’ve created in their own heads.
This is a basilisk problem. Unlike Roko’s, which depends on exotic decision theory, this one involves nothing more than plain utilitarianism. Unlike the standard Utility Monster scenario, this one involves no imaginary entities or hypothetical situations. You just have to look at the actual world around you through the eyes of utilitarianism. It’s a very short road from the innocent-sounding “the greatest good for the greatest number” to this: There are seven billion people on this planet. How can the good you could do them possibly be outweighed by any amount of your own happiness? Just by sitting there reading LessWrong you’re killing babies! Having a beer? You’re drinking dead babies. Own a car? You’re driving on a carpet of dead babies! Murderer! Murderer! Add a dash of transhumanism and you can up the stakes to an obligation to bringing about billions of billions of future humans throughout the universe living lives billions of times better than ours.
But even Peter Singer doesn’t go that far, continuing to be an academic professor and paying his utilitarian obligations by preaching utilitarianism and donating twenty percent of his salary to charity.
This is such an obvious failure mode for utilitarianism, a philosophy at least two centuries old, that surely philosophers must have addressed it. But I don’t know what their responses are.
Christianity has the same problem, and handles it in practice by testing the vocation of those who come to it seeking to devote their whole life to the service of God, to determine whether they are truly called by God. For it is written that many are called, yet few are chosen. In non-supernatural terms, that means determining whether the applicant is psychologically fitted for the life they feel called to, and if not, deflecting their mania into some more productive route.
And then you have people like H0, who notices H2 is crazy, decides that that means that they shouldn’t even try to be altruistic, and accuses H1 of hypocrisy because she’s not like H2. (Exhibit A)
That is my expectation also. However, persuading H2 of that (“but dead babies!”) is likely to be a work of counselling or spiritual guidance rather than reason.
Well… so, if we both expect H1 to do more good than H2, it seems that if we were to look at them through the eyes of utilitarianism, we would endorse being H1 over being H2. But you seem to be saying that H2, looking through the eyes of utilitarianism, endorses being H2 over being H1. I am therefore deeply confused by your model of what’s going on here.
Oh yes, H1 is more effective, heathier, saner, more rational, etc. than H2. H2 is experiencing existential panic and cannot relinquish his death-grip on the idea.
Do you think being a utilitarian makes someone less effective, healthy, sane, rational etc.? Or do you think H2 has these various traits independent of them being a utilitarian?
WARNING: More discussion of a basilisk, with a link to a real-world example.
It’s a possible failure mode of utilitarianism. Some people succumb to it (see George Price for an actual example of a similar failure) and some don’t.
I don’t understand your confusion and this pair of questions just seems misconceived.
(shrug) OK. I certainly agree with you that some utilitarians suffer from the existential panic and inability to relinquish their death-grips on unhealthy ideas, while others don’t. I’m tapping out here.
One could reason that one is better placed to do good effectively when focussing on oneself, ones family, one’s community, etc, simply because one understands them better.
What imaginary creatures would choose whose choice has been written into their definition is of no significance.
Are you saying that human choices are not “written into their definition” in some measure ?
Also, keep in mind that a goal like “make more paperclips” does leave a lot of room for other choices. The agent could spend its time studying metallurgy, or buying existing paperclip factories, or experimenting with alloys, or attempting to invent nanotechnology, or some combination of these and many more activities. It’s not constrained to just a single path.
Just by sitting there reading LessWrong you’re killing babies! … Add a dash of transhumanism and you can up the stakes to an obligation to bringing about billions of billions of future humans throughout the universe living lives billions of times better than ours.
On the one hand, I do agree with you, and I can’t wait to see your proposed solution. On the other hand, I’m not sure what this has to do with the topic. I wasn’t talking about billions of future humans or anything of the sort, merely about a single (semi-hypothetical) human and his goals; whether entertainment is a terminal or instrumental goal; and whether it is a good goal to have.
Let me put it in a different way: if you could take a magic pill which would remove (or, at the very least, greatly reduce) your desire for passive entertainment, would you do it ? People with extremely low preferences for passive entertainment do exist, after all, so this scenario isn’t entirely fantastic (other than for the magic pill part, of course).
Are you saying that human choices are not “written into their definition” in some measure ?
What is written in to humans by evolution is hardly relevant. The point is that you can’t prove anything about humansby drawing a comparison with imaginary creatures that have had something potentially quite different written into them by their creator.
Are you saying that human choices are not “written into their definition” in some measure ?
I have no idea what that even means.
On the one hand, I do agree with you, and I can’t wait to see your proposed solution.
My only solution is “don’t do that then”. It’s a broken thought process, and my interest in it ends with that recognition. Am I a soul doctor? I am not. I seem to be naturally resistant to that failure, but I don’t know how to fix anyone who isn’t.
Let me put it in a different way: if you could take a magic pill which would remove (or, at the very least, greatly reduce) your desire for passive entertainment, would you do it ?
What desire for passive entertainment? For that matter, what is this “passive entertainment”? I am not getting a clear idea of what we are talking about. At any rate, I can’t imagine “entertainment” in the ordinary meaning of that word being a terminal goal.
FWIW, I do not watch television, and have never attended spectator sports.
People with extremely low preferences for passive entertainment do exist, after all
Are you saying that human choices are not “written into their definition” in some measure ?
I have no idea what that even means.
To rephrase: do you believe that all choices made by humans are completely under the humans’ conscious control ? If not, what proportion of our choices is under our control, and what proportion is written into our genes and is thus difficult, if not impossible, to change (given our present level of technology) ?
You objected to my using Clippy as an analogy to human behaviour, on the grounds that Clippy’s choices are “written into its definition”. My point is that a). Clippy is free to make whatever choices it wants, as long as it believes (correctly or erroneously) such choices would lead to more paperclips, and b). we humans operate in a similar way, only we care about things other than paperclips, and therefore c). Clippy is a valid analogy.
My only solution is “don’t do that then”.
Don’t do what ? Do you have a moral theory which works better than utilitarianism/consequentialism ?
What desire for passive entertainment? For that matter, what is this “passive entertainment”?
You don’t watch TV or attend sports, but do you read any fiction books ? Listen to music ? Look at paintings or sculptures (on your own initiative, that is, and not as part of a job) ? Enjoy listening to some small subclass of jokes ? Watch any movies ? Play video games ? Stare at a fire at night ? I’m just trying to pinpoint your general level of interest in entertainment.
At any rate, I can’t imagine “entertainment” in the ordinary meaning of that word being a terminal goal.
Just because you personally can’t imagine something, doesn’t mean it’s not true. For example, art and music—both of which are forms of passive entertainment—has been a part of human history ever since the caveman days, and continue to flourish today. There may be something hardcoded in our genes (maybe not yours personally, but on average) that makes us enjoy art and music. On the other hand, there are lots of things hardcoded in our genes that we’d be better off without...
To rephrase: do you believe that all choices made by humans are completely under the humans’ conscious control ? If not, what proportion of our choices is under our control, and what proportion is written into our genes and is thus difficult, if not impossible, to change (given our present level of technology) ?
The whole language is wrong here.
What does it mean to talk about a choice being “completely under the humans’ conscious control”? Obviously, the causal connections wind through and through all manner of things that are outside consciousness as well as inside. When could you ever say that a decision is “completely under conscious control”?
Then you talk as if a decision not “completely under conscious control” must be “written into the genes”. Where does that come from?
do you read any fiction books?
Why do you specify fiction? Is fiction “passive entertainment” but non-fiction something else?
There may be something hardcoded in our genes (maybe not yours personally, but on average) that makes us enjoy art and music.
What is this “us” that is separate from and acted upon by our genes? Mentalistic dualism?
My only solution is “don’t do that then”.
Don’t do what ? Do you have a moral theory which works better than utilitarianism/consequentialism ?
Don’t crash and burn. I have no moral theory and am not impressed by anything on offer from the philosophers.
To sum up, there’s a large and complex set of assumptions behind everything you’re saying here that I don’t think I share, but I can only guess at from glimpsing the shadowy outlines. I doubt further discussion will get anywhere useful.
Are you saying that human choices are not “written into their definition” in some measure ?
I think Bugmaster is equating being “written in” in the sense of a stipulation in a thought experiment with being “written in” in the sense of being the outcome of an evolutionary process.
WARNING: This comment contains explicit discussion of an information hazard.
I decline to do so. What imaginary creatures would choose whose choice has been written into their definition is of no significance. (This is also a reply to the comment of FeepingCreature you referenced.) I’m more interested in the practical question of how actual human beings, which this discussion began with, can avoid the pitfall of being taken over by a utility monster they’ve created in their own heads.
This is a basilisk problem. Unlike Roko’s, which depends on exotic decision theory, this one involves nothing more than plain utilitarianism. Unlike the standard Utility Monster scenario, this one involves no imaginary entities or hypothetical situations. You just have to look at the actual world around you through the eyes of utilitarianism. It’s a very short road from the innocent-sounding “the greatest good for the greatest number” to this: There are seven billion people on this planet. How can the good you could do them possibly be outweighed by any amount of your own happiness? Just by sitting there reading LessWrong you’re killing babies! Having a beer? You’re drinking dead babies. Own a car? You’re driving on a carpet of dead babies! Murderer! Murderer! Add a dash of transhumanism and you can up the stakes to an obligation to bringing about billions of billions of future humans throughout the universe living lives billions of times better than ours.
But even Peter Singer doesn’t go that far, continuing to be an academic professor and paying his utilitarian obligations by preaching utilitarianism and donating twenty percent of his salary to charity.
This is such an obvious failure mode for utilitarianism, a philosophy at least two centuries old, that surely philosophers must have addressed it. But I don’t know what their responses are.
Christianity has the same problem, and handles it in practice by testing the vocation of those who come to it seeking to devote their whole life to the service of God, to determine whether they are truly called by God. For it is written that many are called, yet few are chosen. In non-supernatural terms, that means determining whether the applicant is psychologically fitted for the life they feel called to, and if not, deflecting their mania into some more productive route.
Consider two humans, H1 and H2, both utilitarians.
H1 looks at the world the way you describe Peter Singer here.
H2 looks at the world “through the eyes of utilitarianism” as you describe it here.
My expectation is that H1 will do more good in their lifetime than H2.
What’s your expectation?
And then you have people like H0, who notices H2 is crazy, decides that that means that they shouldn’t even try to be altruistic, and accuses H1 of hypocrisy because she’s not like H2. (Exhibit A)
That is my expectation also. However, persuading H2 of that (“but dead babies!”) is likely to be a work of counselling or spiritual guidance rather than reason.
Well… so, if we both expect H1 to do more good than H2, it seems that if we were to look at them through the eyes of utilitarianism, we would endorse being H1 over being H2.
But you seem to be saying that H2, looking through the eyes of utilitarianism, endorses being H2 over being H1.
I am therefore deeply confused by your model of what’s going on here.
Oh yes, H1 is more effective, heathier, saner, more rational, etc. than H2. H2 is experiencing existential panic and cannot relinquish his death-grip on the idea.
You confuse me further with every post.
Do you think being a utilitarian makes someone less effective, healthy, sane, rational etc.?
Or do you think H2 has these various traits independent of them being a utilitarian?
There’s a lot of different kinds of utilitarian.
WARNING: More discussion of a basilisk, with a link to a real-world example.
It’s a possible failure mode of utilitarianism. Some people succumb to it (see George Price for an actual example of a similar failure) and some don’t.
I don’t understand your confusion and this pair of questions just seems misconceived.
(shrug) OK.
I certainly agree with you that some utilitarians suffer from the existential panic and inability to relinquish their death-grips on unhealthy ideas, while others don’t.
I’m tapping out here.
One could reason that one is better placed to do good effectively when focussing on oneself, ones family, one’s community, etc, simply because one understands them better.
(Warning: replying to discussion of a potential information hazard.)
Gung’f na rknttrengvba (tvira gung ng gung cbvag lbh unqa’g nqqrq zragvbarq genafuhznavfz lrg) -- nf bs abj, vg’f rfgvzngrq gb gnxr zber guna gjb gubhfnaq qbyynef gb fnir bar puvyq’f yvsr jvgu Tvirjryy’f gbc-engrq punevgl. (Be vf ryrpgevpvgl naq orre zhpu zber rkcrafvir jurer lbh’er sebz?)
Infohazard reference with no warning sign. Edit and reply to this so I can restore.
Done. Sorry this took so long, I’ve been taken mostly offline by a biohazard for the last week.
Are you saying that human choices are not “written into their definition” in some measure ?
Also, keep in mind that a goal like “make more paperclips” does leave a lot of room for other choices. The agent could spend its time studying metallurgy, or buying existing paperclip factories, or experimenting with alloys, or attempting to invent nanotechnology, or some combination of these and many more activities. It’s not constrained to just a single path.
On the one hand, I do agree with you, and I can’t wait to see your proposed solution. On the other hand, I’m not sure what this has to do with the topic. I wasn’t talking about billions of future humans or anything of the sort, merely about a single (semi-hypothetical) human and his goals; whether entertainment is a terminal or instrumental goal; and whether it is a good goal to have.
Let me put it in a different way: if you could take a magic pill which would remove (or, at the very least, greatly reduce) your desire for passive entertainment, would you do it ? People with extremely low preferences for passive entertainment do exist, after all, so this scenario isn’t entirely fantastic (other than for the magic pill part, of course).
What is written in to humans by evolution is hardly relevant. The point is that you can’t prove anything about humansby drawing a comparison with imaginary creatures that have had something potentially quite different written into them by their creator.
I have no idea what that even means.
My only solution is “don’t do that then”. It’s a broken thought process, and my interest in it ends with that recognition. Am I a soul doctor? I am not. I seem to be naturally resistant to that failure, but I don’t know how to fix anyone who isn’t.
What desire for passive entertainment? For that matter, what is this “passive entertainment”? I am not getting a clear idea of what we are talking about. At any rate, I can’t imagine “entertainment” in the ordinary meaning of that word being a terminal goal.
FWIW, I do not watch television, and have never attended spectator sports.
Quite.
To rephrase: do you believe that all choices made by humans are completely under the humans’ conscious control ? If not, what proportion of our choices is under our control, and what proportion is written into our genes and is thus difficult, if not impossible, to change (given our present level of technology) ?
You objected to my using Clippy as an analogy to human behaviour, on the grounds that Clippy’s choices are “written into its definition”. My point is that a). Clippy is free to make whatever choices it wants, as long as it believes (correctly or erroneously) such choices would lead to more paperclips, and b). we humans operate in a similar way, only we care about things other than paperclips, and therefore c). Clippy is a valid analogy.
Don’t do what ? Do you have a moral theory which works better than utilitarianism/consequentialism ?
You don’t watch TV or attend sports, but do you read any fiction books ? Listen to music ? Look at paintings or sculptures (on your own initiative, that is, and not as part of a job) ? Enjoy listening to some small subclass of jokes ? Watch any movies ? Play video games ? Stare at a fire at night ? I’m just trying to pinpoint your general level of interest in entertainment.
Just because you personally can’t imagine something, doesn’t mean it’s not true. For example, art and music—both of which are forms of passive entertainment—has been a part of human history ever since the caveman days, and continue to flourish today. There may be something hardcoded in our genes (maybe not yours personally, but on average) that makes us enjoy art and music. On the other hand, there are lots of things hardcoded in our genes that we’d be better off without...
The whole language is wrong here.
What does it mean to talk about a choice being “completely under the humans’ conscious control”? Obviously, the causal connections wind through and through all manner of things that are outside consciousness as well as inside. When could you ever say that a decision is “completely under conscious control”?
Then you talk as if a decision not “completely under conscious control” must be “written into the genes”. Where does that come from?
Why do you specify fiction? Is fiction “passive entertainment” but non-fiction something else?
What is this “us” that is separate from and acted upon by our genes? Mentalistic dualism?
Don’t crash and burn. I have no moral theory and am not impressed by anything on offer from the philosophers.
To sum up, there’s a large and complex set of assumptions behind everything you’re saying here that I don’t think I share, but I can only guess at from glimpsing the shadowy outlines. I doubt further discussion will get anywhere useful.
I think Bugmaster is equating being “written in” in the sense of a stipulation in a thought experiment with being “written in” in the sense of being the outcome of an evolutionary process.
If he is, he shouldn’t. These are completely different concepts.