= “Remind me again what the rational response is in non-iterated PD when you’re sure your partner is a stupid Defect-Bot.”
Indeed that was my intended message.
How much can society last in such a way before the abandoned and despised “underclass” finally boils over? I don’t think that it’s more than a few generations. Would you want a somewhat nicer and safer life for your children without a future for their children?
I think it is unavoidable. As to saving “everything you love” or rather salvaging the things I value about our civilization. I have spent a lot of time on this problem. Let us say that my current best hope is FAI. Any effort towards which I consider nearly certain to fail horribly, yet still think should be attempted.
What does this tell you? But let me try and put it another way, especially since you bring up the stupid always-defect-bot. Imagine you have a unfriendly self-improving AI. Now imagine it is built out of people’s brains, brains that you care about. Those brains have dreams and values of their own, but they are not the dreams and values of the uFAI that is running on them. It will continue running and self-improving making their dreams and values matter less and less for how the universe is ordered and indeed it will edit and change those values arbitrarily and unpredictably. Imagine the only alternative to this is that it pretty much blows itself up together with all the brains it contains.
But in that world, defecting to ensure your own short-term gain is best replaced with wireheading; nothing fragile survives anyway, so why rob and oppress other brains for a fleeting illusion of contentment? Therefore, we should find happiness in simple things, avoid increasing strife and competition for resources and, ideally, just stop being troubled by the whole mess.
I don’t need gains to be “eternal” or “non-fragile” to count. Remember our debate about the “Hansonian hell word”, where I pointed out that a few centuries of human minds living in plenty followed by aeons of alien minds isn’t really a “hell world” to me?
My conclusions about the world simply mean that I have centuries or decades instead of billions of years of minds I care about arranging matter. I don’t find such a shortening a convincing reason to embrace counterfeit utility with wire-heading.
So you’ve no reinforcing reasons to give up your own fleeting well-being for anyone else’s sake, especially that of far people? Well, at least that’s honest. But I still want to aim for more than temporary gratification when deciding what to do—partly because I still feel very irresponsible and worried when trying not to care about what I see in society.
Sure, of course you care. I meant, well, I’ll think of how to explain it. But basically I’m talking about quasi-religious values again. I have this nagging feeling that we’re hugely missing out on both satisfaction and morality for not understanding their true use.
Ok I hope you can write it out because it sounds interesting. But let me pose a query of my own.
You seem to think that over a period a few billion years means that real utility optimization should occur rather than going for wire-heading. And think that if limited to a few centuries counterfeit utility is better. How would you feel about 10 000 years of the values you cherish followed by a alien or empty universe? What about 10 million years?
If it makes you feel better if everything I cared was destined to disappear tomorrow I think I would go for some wire-heading, so I guess we are just on different spots on the same curve. Is this so?
I’m thinking. But keep in mind that I’m basically a would-be deontologist; I’m just not sure what my deontic ethics should be. If only I could get a consistent (?) and satisfying system, I’d be fine with missing out on direct utility. I know, for humans that’s as impossible as becoming an utility maximizer, because we’re the antithesis of “consistency”.
Indeed that was my intended message.
I think it is unavoidable. As to saving “everything you love” or rather salvaging the things I value about our civilization. I have spent a lot of time on this problem. Let us say that my current best hope is FAI. Any effort towards which I consider nearly certain to fail horribly, yet still think should be attempted.
What does this tell you? But let me try and put it another way, especially since you bring up the stupid always-defect-bot. Imagine you have a unfriendly self-improving AI. Now imagine it is built out of people’s brains, brains that you care about. Those brains have dreams and values of their own, but they are not the dreams and values of the uFAI that is running on them. It will continue running and self-improving making their dreams and values matter less and less for how the universe is ordered and indeed it will edit and change those values arbitrarily and unpredictably. Imagine the only alternative to this is that it pretty much blows itself up together with all the brains it contains.
This is the world I think I am living in.
But in that world, defecting to ensure your own short-term gain is best replaced with wireheading; nothing fragile survives anyway, so why rob and oppress other brains for a fleeting illusion of contentment? Therefore, we should find happiness in simple things, avoid increasing strife and competition for resources and, ideally, just stop being troubled by the whole mess.
Oops, looks like I accidentally Buddhism.
I don’t need gains to be “eternal” or “non-fragile” to count. Remember our debate about the “Hansonian hell word”, where I pointed out that a few centuries of human minds living in plenty followed by aeons of alien minds isn’t really a “hell world” to me?
My conclusions about the world simply mean that I have centuries or decades instead of billions of years of minds I care about arranging matter. I don’t find such a shortening a convincing reason to embrace counterfeit utility with wire-heading.
So you’ve no reinforcing reasons to give up your own fleeting well-being for anyone else’s sake, especially that of far people? Well, at least that’s honest. But I still want to aim for more than temporary gratification when deciding what to do—partly because I still feel very irresponsible and worried when trying not to care about what I see in society.
The set of minds I care about obviously includes my own but isn’t limited to it!
Sure, of course you care. I meant, well, I’ll think of how to explain it. But basically I’m talking about quasi-religious values again. I have this nagging feeling that we’re hugely missing out on both satisfaction and morality for not understanding their true use.
Ok I hope you can write it out because it sounds interesting. But let me pose a query of my own.
You seem to think that over a period a few billion years means that real utility optimization should occur rather than going for wire-heading. And think that if limited to a few centuries counterfeit utility is better. How would you feel about 10 000 years of the values you cherish followed by a alien or empty universe? What about 10 million years?
If it makes you feel better if everything I cared was destined to disappear tomorrow I think I would go for some wire-heading, so I guess we are just on different spots on the same curve. Is this so?
I’m thinking. But keep in mind that I’m basically a would-be deontologist; I’m just not sure what my deontic ethics should be. If only I could get a consistent (?) and satisfying system, I’d be fine with missing out on direct utility. I know, for humans that’s as impossible as becoming an utility maximizer, because we’re the antithesis of “consistency”.