(We’ve won when the universe has been torn apart and rebuilt for our benefit, or a powerful and reliable process that doesn’t need human help is doing so. At that point we can focus on living valuable lives as people. For now we have to focus on being agents; steering the future towards that win.)
Whenever I read people talking about post-singularity utopias it makes me really glad I’ll probably die before they can be brought to fruition.
Can you actually imagine life where innate talent and learned skill had no value outside of social posturing or self-gratification? Where a Godlike machine secretly planned out your life according to some formula of ideal living? Where no-one suffered or died at all unless they chose to and real power is essentially non-existent? Where you’ll be, in all likelihood, locked into a computer simulation while the real world is being torn apart for materials to increase your jailer’s intelligence and extend its lifespan?
Even Hanson’s cockroach-topia sounds better than that.
Can you actually imagine life where innate talent and learned skill had no value outside of social posturing or self-gratification? Where a Godlike machine secretly planned out your life according to some formula of ideal living? Where no-one suffered or died at all unless they chose to and real power is essentially non-existent? Where you’ll be, in all likelihood, locked into a computer simulation while the real world is being torn apart for materials to increase your jailer’s intelligence and extend its lifespan?
In case you are simply making a mistake: If we win, it is hardly likely that we would build something that sucked that much. The scenario you describe is not winning, and not what anyone actually wants.
How about a world where random things that you could do nothing about did not just kill you. Where hard work and learning actually could push you as far as you wanted to go. etc.
I did that several years before I joined, actually. Although obviously we’ve taken away different things from it.
How about a world where random things that you could do nothing about did not just kill you. Where hard work and learning actually could push you as far as you wanted to go. etc.
That’s actually the problem though; the ‘unfairness’ of chance destruction and the hard limits which no individual can transcend in a lifetime are exactly the sorts of things which I find valuable. In my view, “[h]ard work pushing you” is not a choice any more than there’s a choice between eating and starvation; those who strive will extend their will into the future while those who don’t will be annihilated and forgotten. If there are safeties to turn off, a door to get out of or a 100% completion “Golden Ending” available then it’s not life but a game. I’d like to see humanity grow up and stop playing around with games, on a metaphorical level anyway.
I know that this is not a mainstream view and that few would choose to live in my personal utopia, but on the flip side why should I change my aesthetics just because they’d lose in a headcount?
those who strive will extend their will into the future while those who don’t will be annihilated and forgotten.
Those who strive can also be annihilated and forgotten, and those who don’t can extend their will into the future given sufficiently fortuitous starting conditions.
If that’s also part of a utopia you’d desire, so be it, but there are no points for pretending the chance destruction of our reality is fairer than it is.
Because your aesthetics are evil, possibly self-contradictory as a result, and sound ill-informed. (Picture from here, if the first link doesn’t work).
The part that puzzles me most is the way you admit the existence of “chance destruction” before claiming “those who strive will extend their will into the future”. In the real world, looks like 5.5 million children die annually between the ages of 4 weeks and 5 years. I picked that range because I’d expect many of them to try forming models of the world and optimizing it as best they can, before the world unceremoniously deletes them.
Like I said, I don’t think you’re evil. I think you espoused an evil position or preference, which may well contradict other and more real preferences. I think you’re looking at a ridiculously small sliver of the real world and using that to conclude that the status quo is fine. (Or you’re pattern-matching wildly, which has now grown in probability.)
If I haven’t already said enough, then it looks like one of the following holds:
You have effectively no empathy and I don’t want you to model me accurately.
You share my basic goals, but believe we can’t do much better at achieving them. Your earlier strawman really indicated that you see us as foolish dreamers.
Either way, further talk of morality seems counter-productive.
The difference between “evil” and “immoral within ” is that evil must be fought with and eradicated, if necessary without taking prisoners and leaving scorched earth behind. Now, “immoral”, that we can talk about, is it cool and sexy?
That’s one way to look at it; another is to question the unspoken assumption that everyone’s needs deserve equal treatment. Human beings are variable on every measurable dimension, most at least partially heritable, so why not the moral dimension as well?
The fact is, as Yvain elegantly puts it, we really don’t care about the vast majority people except in the abstract and quite a bit of that professed empathy is just a result of trying to sound like a ‘good person’ in a society obsessed with the wellbeing of its underclasses. So if our self-actualization (and that of our in-groups) is already more important to us than the hunger of distant people… why not stop following a moral code we don’t even really believe in and create new values which better match our natures?
(Also, is it just me or is the hyperbolic use of ‘verboten’ incredibly annoying when you know the ‘vee’ is actually a ‘fow’ and thus the German and English words are virtually indistinguishable when spoken?)
I totally agree that utilitarianism is overhyped nonsense, and true values are much more selfish and unsympathetic. That said, given massive power, I think a non-significant fraction of my resources would go to public works projects and not just my throne.
I totally agree that utilitarianism is overhyped nonsense, and true values are much more selfish and unsympathetic. That said, given massive power, I think a non-significant fraction of my resources would go to public works projects and not just my throne.
Exactly; just because you create your own values and live by them doesn’t mean you have to stop helping your neighbor cross the street, or mandate that you start turning cherry blossom trees into biodiesel to power your strip-mining robots. It just means that you have defined your own preferences and aesthetics and act on that knowledge. “Don’t make a future too ugly for you to enjoy it” is as good a maxim as any.
I really like the “maximize awesomeness” idea BTW. It’s very catchy and cuts away a lot of the fat of the issue.
(Also, is it just me or is the hyperbolic use of ‘verboten’ incredibly annoying when you know the ‘vee’ is actually a ‘fow’ and thus the German and English words are virtually indistinguishable when spoken?)
Unless I’m way out of whack with German pronunciation, the stressed vowel in verboten doesn’t sound anything like that in forbidden.
Personally I hear fer-boat-in and fer-bid-in as similar enough that if someone on the street said the former I’d assume they meant the latter and never think twice about it. But even if you’d hear the difference more acutely, I don’t think either of us would think ‘Gestapo!’ the way we would seeing it spelled out.
Edit: Oops, I used lmgtfy instead of tinyurl. Sorry, that was a bookmarks issue not condescension.
Can you actually imagine life where innate talent and learned skill had no value outside of social posturing or self-gratification?
… as opposed to … helping other people?
Where a Godlike machine secretly planned out your life according to some formula of ideal living?
I’m having trouble seeing the downsides to that. Oh, is “some formula” supposed to imply it’s arbitrary and false?
Where no-one suffered or died at all unless they chose to and real power is essentially non-existent?
I don’t even know what you mean by this one.
Where you’ll be, in all likelihood, locked into a computer simulation while the real world is being torn apart for materials to increase your jailer’s intelligence and extend its lifespan?
“Locked”?
But, to be fair, plenty of people don’t like the idea of living in a simulation. Probably didn’t play enough videogames as kids : P
… but seriously, folks, if this turns out to be an issue, you don’t have to keep posthumanity in simulations. Many people assume we wont. It’s just easier to simulate awesome things than build them, that’s all.
Even Hanson’s cockroach-topia sounds better than that.
OK, that’s a really excellent name for it. Upvoting just for that.
Can you actually imagine life where innate talent and learned skill had no value outside of social posturing or self-gratification?
Pretty sure I’m there already.
Where a Godlike machine secretly planned out your life according to some formula of ideal living?
Sounds comforting.
Where no-one suffered or died at all unless they chose to
Yes, a thousand times yes.
and real power is essentially non-existent?
Hmm. If I could only gain happiness from having power over others, I would be forced to consider myself as evil. That aside, in these post-singularity utopias I could have an arrangement with other people where we took turns, or gambled. Or I could lord it over a nation of entities that looked and acted like humans but weren’t really.
Is that a better deal for me personally? Depends on whether I’m powerful at the moment. But it’s certainly an improvement for people on average.
Where you’ll be, in all likelihood, locked into a computer simulation while the real world is being torn apart for materials to increase your jailer’s intelligence and extend its lifespan?
Agreed that this is bad. Post-singularity utopian AI should not do this.
Can you state clearly what’s wrong with the world you describe… that is, what you would prefer instead? If not, do you think someone much smarter than you might be able to state it clearly? Either way… why not assume a world like that, instead?
For my own part, the things you describe don’t sound particularly bad, weighted language notwithstanding. They’re certainly better than what we currently live with.
Whenever I read people talking about post-singularity utopias it makes me really glad I’ll probably die before they can be brought to fruition.
Can you actually imagine life where innate talent and learned skill had no value outside of social posturing or self-gratification? Where a Godlike machine secretly planned out your life according to some formula of ideal living? Where no-one suffered or died at all unless they chose to and real power is essentially non-existent? Where you’ll be, in all likelihood, locked into a computer simulation while the real world is being torn apart for materials to increase your jailer’s intelligence and extend its lifespan?
Even Hanson’s cockroach-topia sounds better than that.
In case you are simply making a mistake: If we win, it is hardly likely that we would build something that sucked that much. The scenario you describe is not winning, and not what anyone actually wants.
How about a world where random things that you could do nothing about did not just kill you. Where hard work and learning actually could push you as far as you wanted to go. etc.
Read the Fun Theory Sequence
I did that several years before I joined, actually. Although obviously we’ve taken away different things from it.
That’s actually the problem though; the ‘unfairness’ of chance destruction and the hard limits which no individual can transcend in a lifetime are exactly the sorts of things which I find valuable. In my view, “[h]ard work pushing you” is not a choice any more than there’s a choice between eating and starvation; those who strive will extend their will into the future while those who don’t will be annihilated and forgotten. If there are safeties to turn off, a door to get out of or a 100% completion “Golden Ending” available then it’s not life but a game. I’d like to see humanity grow up and stop playing around with games, on a metaphorical level anyway.
I know that this is not a mainstream view and that few would choose to live in my personal utopia, but on the flip side why should I change my aesthetics just because they’d lose in a headcount?
Those who strive can also be annihilated and forgotten, and those who don’t can extend their will into the future given sufficiently fortuitous starting conditions.
If that’s also part of a utopia you’d desire, so be it, but there are no points for pretending the chance destruction of our reality is fairer than it is.
Because your aesthetics are evil, possibly self-contradictory as a result, and sound ill-informed. (Picture from here, if the first link doesn’t work).
Trigger warning: The parent comment links to a photograph of a screaming child covered in blood.
For the down-voters:
Thanks for the pic, but I’m curious as to why you thought an ‘evil’ person would be bothered by it.
Edit: Actually, while I’ve got you here, can you let me know where I’ve been contradictory? I’d like to fix that sort of thing going forward.
The part that puzzles me most is the way you admit the existence of “chance destruction” before claiming “those who strive will extend their will into the future”. In the real world, looks like 5.5 million children die annually between the ages of 4 weeks and 5 years. I picked that range because I’d expect many of them to try forming models of the world and optimizing it as best they can, before the world unceremoniously deletes them.
Like I said, I don’t think you’re evil. I think you espoused an evil position or preference, which may well contradict other and more real preferences. I think you’re looking at a ridiculously small sliver of the real world and using that to conclude that the status quo is fine. (Or you’re pattern-matching wildly, which has now grown in probability.)
Now that you’ve brought this word into the conversation, would you care to define “evil”?
No I would not.
I think you meant to link to this page, but I’m still not sure how either relates to the question.
Maybe we can taboo ‘evil’ and replace it with something like “immoral within ′ to get a firmer sense of your specific objection.
If I haven’t already said enough, then it looks like one of the following holds:
You have effectively no empathy and I don’t want you to model me accurately.
You share my basic goals, but believe we can’t do much better at achieving them. Your earlier strawman really indicated that you see us as foolish dreamers.
Either way, further talk of morality seems counter-productive.
The difference between “evil” and “immoral within ” is that evil must be fought with and eradicated, if necessary without taking prisoners and leaving scorched earth behind. Now, “immoral”, that we can talk about, is it cool and sexy?
See Yvain on how certain ‘dystopias’ only look like ones from a First World person perspective.
That’s one way to look at it; another is to question the unspoken assumption that everyone’s needs deserve equal treatment. Human beings are variable on every measurable dimension, most at least partially heritable, so why not the moral dimension as well?
The fact is, as Yvain elegantly puts it, we really don’t care about the vast majority people except in the abstract and quite a bit of that professed empathy is just a result of trying to sound like a ‘good person’ in a society obsessed with the wellbeing of its underclasses. So if our self-actualization (and that of our in-groups) is already more important to us than the hunger of distant people… why not stop following a moral code we don’t even really believe in and create new values which better match our natures?
(Also, is it just me or is the hyperbolic use of ‘verboten’ incredibly annoying when you know the ‘vee’ is actually a ‘fow’ and thus the German and English words are virtually indistinguishable when spoken?)
Well said. This is true.
You mean like this?
I totally agree that utilitarianism is overhyped nonsense, and true values are much more selfish and unsympathetic. That said, given massive power, I think a non-significant fraction of my resources would go to public works projects and not just my throne.
Exactly; just because you create your own values and live by them doesn’t mean you have to stop helping your neighbor cross the street, or mandate that you start turning cherry blossom trees into biodiesel to power your strip-mining robots. It just means that you have defined your own preferences and aesthetics and act on that knowledge. “Don’t make a future too ugly for you to enjoy it” is as good a maxim as any.
I really like the “maximize awesomeness” idea BTW. It’s very catchy and cuts away a lot of the fat of the issue.
Unless I’m way out of whack with German pronunciation, the stressed vowel in verboten doesn’t sound anything like that in forbidden.
Verboten v Forbidden
Personally I hear fer-boat-in and fer-bid-in as similar enough that if someone on the street said the former I’d assume they meant the latter and never think twice about it. But even if you’d hear the difference more acutely, I don’t think either of us would think ‘Gestapo!’ the way we would seeing it spelled out.
Edit: Oops, I used lmgtfy instead of tinyurl. Sorry, that was a bookmarks issue not condescension.
Consider incorporating Nisan’s trigger warning into your post.
… as opposed to … helping other people?
I’m having trouble seeing the downsides to that. Oh, is “some formula” supposed to imply it’s arbitrary and false?
I don’t even know what you mean by this one.
“Locked”?
But, to be fair, plenty of people don’t like the idea of living in a simulation. Probably didn’t play enough videogames as kids : P
… but seriously, folks, if this turns out to be an issue, you don’t have to keep posthumanity in simulations. Many people assume we wont. It’s just easier to simulate awesome things than build them, that’s all.
OK, that’s a really excellent name for it. Upvoting just for that.
Pretty sure I’m there already.
Sounds comforting.
Yes, a thousand times yes.
Hmm. If I could only gain happiness from having power over others, I would be forced to consider myself as evil. That aside, in these post-singularity utopias I could have an arrangement with other people where we took turns, or gambled. Or I could lord it over a nation of entities that looked and acted like humans but weren’t really.
Is that a better deal for me personally? Depends on whether I’m powerful at the moment. But it’s certainly an improvement for people on average.
Agreed that this is bad. Post-singularity utopian AI should not do this.
Hire a different world-builder, then. :)
Can you state clearly what’s wrong with the world you describe… that is, what you would prefer instead?
If not, do you think someone much smarter than you might be able to state it clearly?
Either way… why not assume a world like that, instead?
For my own part, the things you describe don’t sound particularly bad, weighted language notwithstanding.
They’re certainly better than what we currently live with.