This isn’t really ‘upgrading’ moral theories by taking account of moral intuitions, but rather ‘trying to patch them up to accord with a first order intuition of mine, regardless of distortions elsewhere’.
You want to avoid wireheading like scenarios by saying (in effect) ‘fulfilled humane values have lexical priority over non-humane values’, so smaller populations of happy populations of humans can trump very large populations of happy mice, even if that would be a better utilitarian deal (I’m sceptical that’s how the empirical calculus plays out, but that’s another issue).
This just seems wrong on its face: it seems a wrong to refuse to trade one happy human for e.g. a trillion (or quadrillion, 3^^3) happy mice (or octopi, or dolphins—just how ‘humane’ are we aiming for?). It seems even worse to trade one happy human (although you don’t specify, this could be ‘barely positive welfare’ human life) for torturing unbounded numbers of mice or dolphins.
You seem to find the latter problem unacceptable (although I think you should find the first really bad too), so you offer a different patch: although positive human welfare lexically dominates positive animal welfare, it doesn’t dominate negative animal welfare, which can be traded off, so happy humans at the expense of vast mice-torture is out. Yet this has other problems due to weird separability violations. Adjusting the example you give for clarity:
1) 100 A
2) 1H + 100A − 50A
3) 0
4) 1H − 50A
The account seems to greenlight the move from 1 -->2, as the welfare of the animal population remains positive, and although it goes down, we don’t care because human welfare trumps it. However, we cannot move from 3-4, as the total negative animal welfare trumps the human population. This seems embarrassing, particularly as our 100 happy animal welfare could be on an adjecent supercluster, and so virtually any trade of human versus animal interests is rendered right or wrong by total welfare across the universe.
As far as I can see, the next patch you offer (never accept a population with overall negative welfare is better than one with overall positive welfare) won’t help you avoid these separability concerns: 4 is still worse than 3, as it is net negative, but 2 is net positive, so is still better than 1.
You want to assert nigh-deontological ‘do not do this if X’ to avoid RC-like edge cases. The problem with these sorts of rules is you need to set them to strict lexical priority so they really rule out the edge case (as you rightly point out, if you, like I, take the ‘human happiness has a high weight’ out, you still are forced to replace humans with stoned mice so long as enough stoned mice are on the other end of the scales). Yet lexical priority seems to lead to another absurd edge case where you have to refuse even humongous trade-offs: is one happy human really worth more than a gazillion happy mice? Or indeed torturing trillions of dolphins?
So then you have to introduce other nigh-deontological trumping rules to rule out these nasty edge cases (e.g. ‘you can trade any amount of positive animal welfare for any increase in human welfare, but negative welfare has a 1-1 trade with human welfare’). But then these too have nasty edge cases, and worse, you can (as deontologists find) find these rules clash to give you loss of transitivity, loss of separability, dependence on irrelevant alternatives, etc.
FWIW, I think the best fix is empirical: just take humans to be capable of much greater positive welfare states than animals, such that you have to make really big trades of humans to animals to be worth it. Although these leaves open the possibility we’d end up with little complex value, it does not seem a likely possibliity (I trade humans to mice at ~ 1000000 : 1, and I don’t think mice are 1000000x cheaper). It also avoids getting edge cased, and thanks to classical util cardinally ordering states of affairs, you don’t violate transitivity or separability.
As a second recommendation (with pre-emptive apologies for offense: I can’t find a better way of saying this), I’d recommend going back to the population ethics literature (and philosophy generally) rather than trying to reconstitute ethical theory yourself. I don’t think you know enough or have developed enough philosophical skill to make good inroads into these problems, and the posts you’ve made so far I think are unlikely to produce anything interesting or important in the field, and are considerably inferior to academic work ongoing.
By the way, 3^^3 = 3^27 is “only” 7625597484987, which is less than a quadrillion. If you want a really big number, you should add a third arrow (or use a higher number than three).
I wouldn’t actually condone the move from 1 to 2. I would not condone inflicting huge harms on animals to create a moderately well-off human. But I would condone never creating some happy animals in the first place. Not creating is not the same as harming. The fact that TU treats not creating an animal with 50 utility to be equivalent to inflicting 50 points of disutility on an animal that has 50 utility is a strike against it. If we create an animal we have a responsibility to make it happy, if we don’t we’re free to make satisfied humans instead (to make things simpler I’m leaving the option of painlessly killing the animals off the table).
My argument is existing animals and existing humans have equal moral significance (or maybe humans have somewhat more if you’re actually right about humans being capable of greaters level of welfare), but when deciding which to create, human creation is superior to animal (or paperclipper or wirehead) creation.
FWIW, I think the best fix is empirical: just take humans to be capable of much greater positive welfare states than animals, such that you have to make really big trades of humans to animals to be worth it. Although these leaves open the possibility we’d end up with little complex value, it does not seem a likely possibliity
I lack your faith in this fix. I consider it almost certain that if we were to create a utilitarian AI it would kill the entire human race and replace it with creatures whose preferences are easier to satisfy. And by “easier to satisfy” I mean “simpler and less ambitious,” not that the creatures are more mentally and physically capable of satisfying humane desires.
In addition to this, total utilitarianism has several other problems that need fixing, the most obvious being that it considers it a morally good act to kill someone destined to live a good long life and replace them with a new person whose overall utility is slightly higher than the utility of the remaining years of that person’s life would have been (I know in practice doing this is impossible without side-effects that create large dis-utilities, but that shouldn’t matter, it’s bad because it’s bad, not because it has bad side-effects).
Quite frankly I consider the Repugnant Conclusion to be far less horrible than the “Kill and Replace” conclusion. I’d probably accept total utilitarianism as a good moral system if the RC was all it implied. The fact that TU implies something strange like the RC in an extremely unrealistic toy scenario isn’t that big a deal.But that’s not all TU implies. The “Kill and Replace” conclusion isn’t a toy scenario, there are tons of handicapped people that could be killed and replaced with able people right now. But we don’t do that, because it’s wrong.
I don’t spend my days cursing the fact that various disutilitous side effects prevent me from killing handicapped people and creating able people to replace them. Individual people, once they are created, matter. Once someone has been brought into existence we have a greater duty to make sure they stay alive and happy then we do to create new people. There may be some vastly huge amount of happy people that it’s okay to kill one slightly-less-happy-person in order to create, but that number should be way, way, way, way, bigger than 1.
As a second recommendation (with pre-emptive apologies for offense: I can’t find a better way of saying this), I’d recommend going back to the population ethics literature (and philosophy generally) rather than trying to reconstitute ethical theory yourself.
I’ve looked at some of the literature, and I’ve noticed there does not seem to be much interest in the main question I am interested in, which is, “Why make humans and not something else?” Peter Singer mentions it a few times in one essay I read, but didn’t offer any answers, he just seemed to accept it as obvious. I thought it was a fallow field that I might be able to do some actual work in.
Peter Singer also has the decency to argue that human beings are not replaceable, that killing one person to replace them with a slightly happier one is bad. But I have trouble seeing how his arguments work against total utilitarianism.
I consider it almost certain that if we were to create a utilitarian AI it would kill the entire human race and replace it with creatures whose preferences are easier to satisfy. And by “easier to satisfy” I mean “simpler and less ambitious,” not that the creatures are more mentally and physically capable of satisfying humane desires.
It would not necessarily kill off humanity to replace it by something else, though. Looking at the world right now, many countries run smoothly, and others horribly, even though they are all inhabited and governed by humans. Even if you made the AI “prefer” human beings, it could still evaluate that “fixing” humanity would be too slow and costly and that “rebooting” it is a much better option. That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
Once someone has been brought into existence we have a greater duty to make sure they stay alive and happy then we do to create new people. There may be some vastly huge amount of happy people that it’s okay to kill one slightly-less-happy-person in order to create, but that number should be way, way, way, way, bigger than 1.
Sure. Just add the number of deaths to the utility function with an appropriate multiplier, so that world states obtained through killing get penalized. Of course, an AI who wishes to get rid of humanity in order to set up a better world unobstructed could attempt to circumvent the limitation: create an infertility epidemic to extinguish humanity within a few generations, fudge genetics to tame it (even if it is only temporary), and so forth.
Ultimately, though, it seems that you just want the AI to do whatever you want it to do and nothing you don’t want it to do. I very much doubt there is any formalization of what you, me, or any other human really wants. The society we have now is the result of social progress that elders have fought tooth and nail against. Given that in general humans can’t get their own offspring to respect their taboos, what if your grandchildren come to embrace some options that you find repugnant or disagree with your idea of utopia? What if the AI tells itself “I can’t kill humanity now, but if I do this and that, eventually, it will give me the mandate”? Society is an iceberg drifting along the current, only sensing the direction it’s going at the moment, but with poor foresight as to what the direction is going to be after that.
I’ve noticed there does not seem to be much interest in the main question I am interested in, which is, “Why make humans and not something else?”
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans. Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
That would be bad, but it would still be way better than replacing us with paperclippers or orgasmium.
The society we have now is the result of social progress that elders have fought tooth and nail against.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should. So I’m fine with future creatures that have something like human nature deciding some new society I’m kind of uncomfortable with is the best way to express their natures. What I’m not fine with is throwing human nature out and starting from scratch with something new, which is what I think a utilitarian AI would do.
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.” For instance, if given a choice between creating an AI with human-like values, and creating a human sociopath, I would pick the AI. And it wouldn’t just be because there was a chance the sociopath would harm others. I simply consider the values of the AI more worthy of creation than the sociopath’s.
Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
I don’t necessarily disagree. If having a large population of creatures with humane values and high welfare was assured then it might be better to have a variety of creatures. But I still think maybe there should be some limits on the sort of creatures we should create, i.e. lawful creativity. Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
It’s quite hard to tell. “Progress” is always relative to the environment you grew up in and on which your ideas and aspirations are based. At the scale of a human life, our trajectory looks a lot like a straight line, but for all we know, it could be circular. At every point on the circle, we would aim to follow the tangent, and it would look like that’s what we are doing. However, as we move along, the tangent would shift ever so subtly and over the course of millennia we would end up doing a roundabout.
I am not saying that’s precisely what we are doing, but there is some truth to it: human goals and values shift. Our environment and upbringing mold us very deeply, in a way that we cannot really abstract away. A big part of what we consider “ideal” is therefore a function of that imprint. However, we rarely ponder the fact that people born and raised in our “ideal world” would be molded differently and thus may have a fundamentally different outlook on life, including wishing for something else. That’s a bit contrived, of course, but it would probably be possible to make a society which wants X when raised on Y, and Y when raised on X, so that it would constantly oscillate between X and Y. We would have enough foresight to figure out a simple oscillator, but if ethics were a kind of semi-random walk, I don’t think it would be obvious. The idea that we are converging towards something might be a bit of an illusion due to underestimating how different future people will be from ourselves.
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should.
I suspect the negative aspects of our natures occur primarily when access to resources is strained. If every human is sheltered, well fed, has access to plentiful energy, and so on, there aren’t really be any problems to blame on anyone, so everything should work fine (for the most part, anyway). In a sense, this simplifies the task of the AI: you ask it to optimize supply to existing demand and the rest is smooth sailing.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.”
Still, the criterion is explicitly based on human values. Even if not human specifically, you want “human-like” creatures.
Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
Still fairly anthropomorphic (not necessarily a bad thing, just an observation). In principle, extremely interesting entities could have no conception of self. Sympathy is only relevant to social entities—but why not create solitary ones as well? As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former? I mean, when you think about it, in order to bond with each other, social entities must share a lot of preferences, the encoding of which is redundant. Solitary entities with fringe preferences could thus be a cheap and easy way to increase variety.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
Sympathy is only relevant to social entities—but why not create solitary ones as well?
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former?
It’s not about having a specialized interest and exploring it. A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.” They wouldn’t be like an autistic savant who knows one subject inside and out. They’d be little better than a wirehead. Someone with narrow interests still explores every single aspect of that interest in great detail. A creature with no boredom would find one tiny aspect of that interest and do it forever.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
Fair enough, though at the level of omnipotence we’re supposing, there would be no chance meetups. You might as well just isolate the creature and be done with it.
A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.”
Or it would do it once, and then die happy. Human-like entities might have a lifespan of centuries, and then you would have ephemeral beings living their own limited fantasy for thirty seconds. I mean, why not? We are all bound to repeat ourselves once our interests are exhausted—perhaps entities could be made to embrace death when that happens.
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
I agree, though an entity with the power to choose the kind of creatures that come to exist probably wouldn’t have much difficulty doing it safely.
This isn’t really ‘upgrading’ moral theories by taking account of moral intuitions, but rather ‘trying to patch them up to accord with a first order intuition of mine, regardless of distortions elsewhere’.
You want to avoid wireheading like scenarios by saying (in effect) ‘fulfilled humane values have lexical priority over non-humane values’, so smaller populations of happy populations of humans can trump very large populations of happy mice, even if that would be a better utilitarian deal (I’m sceptical that’s how the empirical calculus plays out, but that’s another issue).
This just seems wrong on its face: it seems a wrong to refuse to trade one happy human for e.g. a trillion (or quadrillion, 3^^3) happy mice (or octopi, or dolphins—just how ‘humane’ are we aiming for?). It seems even worse to trade one happy human (although you don’t specify, this could be ‘barely positive welfare’ human life) for torturing unbounded numbers of mice or dolphins.
You seem to find the latter problem unacceptable (although I think you should find the first really bad too), so you offer a different patch: although positive human welfare lexically dominates positive animal welfare, it doesn’t dominate negative animal welfare, which can be traded off, so happy humans at the expense of vast mice-torture is out. Yet this has other problems due to weird separability violations. Adjusting the example you give for clarity:
1) 100 A 2) 1H + 100A − 50A 3) 0 4) 1H − 50A
The account seems to greenlight the move from 1 -->2, as the welfare of the animal population remains positive, and although it goes down, we don’t care because human welfare trumps it. However, we cannot move from 3-4, as the total negative animal welfare trumps the human population. This seems embarrassing, particularly as our 100 happy animal welfare could be on an adjecent supercluster, and so virtually any trade of human versus animal interests is rendered right or wrong by total welfare across the universe.
As far as I can see, the next patch you offer (never accept a population with overall negative welfare is better than one with overall positive welfare) won’t help you avoid these separability concerns: 4 is still worse than 3, as it is net negative, but 2 is net positive, so is still better than 1.
You want to assert nigh-deontological ‘do not do this if X’ to avoid RC-like edge cases. The problem with these sorts of rules is you need to set them to strict lexical priority so they really rule out the edge case (as you rightly point out, if you, like I, take the ‘human happiness has a high weight’ out, you still are forced to replace humans with stoned mice so long as enough stoned mice are on the other end of the scales). Yet lexical priority seems to lead to another absurd edge case where you have to refuse even humongous trade-offs: is one happy human really worth more than a gazillion happy mice? Or indeed torturing trillions of dolphins?
So then you have to introduce other nigh-deontological trumping rules to rule out these nasty edge cases (e.g. ‘you can trade any amount of positive animal welfare for any increase in human welfare, but negative welfare has a 1-1 trade with human welfare’). But then these too have nasty edge cases, and worse, you can (as deontologists find) find these rules clash to give you loss of transitivity, loss of separability, dependence on irrelevant alternatives, etc.
FWIW, I think the best fix is empirical: just take humans to be capable of much greater positive welfare states than animals, such that you have to make really big trades of humans to animals to be worth it. Although these leaves open the possibility we’d end up with little complex value, it does not seem a likely possibliity (I trade humans to mice at ~ 1000000 : 1, and I don’t think mice are 1000000x cheaper). It also avoids getting edge cased, and thanks to classical util cardinally ordering states of affairs, you don’t violate transitivity or separability.
As a second recommendation (with pre-emptive apologies for offense: I can’t find a better way of saying this), I’d recommend going back to the population ethics literature (and philosophy generally) rather than trying to reconstitute ethical theory yourself. I don’t think you know enough or have developed enough philosophical skill to make good inroads into these problems, and the posts you’ve made so far I think are unlikely to produce anything interesting or important in the field, and are considerably inferior to academic work ongoing.
By the way, 3^^3 = 3^27 is “only” 7625597484987, which is less than a quadrillion. If you want a really big number, you should add a third arrow (or use a higher number than three).
I wouldn’t actually condone the move from 1 to 2. I would not condone inflicting huge harms on animals to create a moderately well-off human. But I would condone never creating some happy animals in the first place. Not creating is not the same as harming. The fact that TU treats not creating an animal with 50 utility to be equivalent to inflicting 50 points of disutility on an animal that has 50 utility is a strike against it. If we create an animal we have a responsibility to make it happy, if we don’t we’re free to make satisfied humans instead (to make things simpler I’m leaving the option of painlessly killing the animals off the table).
My argument is existing animals and existing humans have equal moral significance (or maybe humans have somewhat more if you’re actually right about humans being capable of greaters level of welfare), but when deciding which to create, human creation is superior to animal (or paperclipper or wirehead) creation.
I lack your faith in this fix. I consider it almost certain that if we were to create a utilitarian AI it would kill the entire human race and replace it with creatures whose preferences are easier to satisfy. And by “easier to satisfy” I mean “simpler and less ambitious,” not that the creatures are more mentally and physically capable of satisfying humane desires.
In addition to this, total utilitarianism has several other problems that need fixing, the most obvious being that it considers it a morally good act to kill someone destined to live a good long life and replace them with a new person whose overall utility is slightly higher than the utility of the remaining years of that person’s life would have been (I know in practice doing this is impossible without side-effects that create large dis-utilities, but that shouldn’t matter, it’s bad because it’s bad, not because it has bad side-effects).
Quite frankly I consider the Repugnant Conclusion to be far less horrible than the “Kill and Replace” conclusion. I’d probably accept total utilitarianism as a good moral system if the RC was all it implied. The fact that TU implies something strange like the RC in an extremely unrealistic toy scenario isn’t that big a deal. But that’s not all TU implies. The “Kill and Replace” conclusion isn’t a toy scenario, there are tons of handicapped people that could be killed and replaced with able people right now. But we don’t do that, because it’s wrong.
I don’t spend my days cursing the fact that various disutilitous side effects prevent me from killing handicapped people and creating able people to replace them. Individual people, once they are created, matter. Once someone has been brought into existence we have a greater duty to make sure they stay alive and happy then we do to create new people. There may be some vastly huge amount of happy people that it’s okay to kill one slightly-less-happy-person in order to create, but that number should be way, way, way, way, bigger than 1.
I’ve looked at some of the literature, and I’ve noticed there does not seem to be much interest in the main question I am interested in, which is, “Why make humans and not something else?” Peter Singer mentions it a few times in one essay I read, but didn’t offer any answers, he just seemed to accept it as obvious. I thought it was a fallow field that I might be able to do some actual work in.
Peter Singer also has the decency to argue that human beings are not replaceable, that killing one person to replace them with a slightly happier one is bad. But I have trouble seeing how his arguments work against total utilitarianism.
It would not necessarily kill off humanity to replace it by something else, though. Looking at the world right now, many countries run smoothly, and others horribly, even though they are all inhabited and governed by humans. Even if you made the AI “prefer” human beings, it could still evaluate that “fixing” humanity would be too slow and costly and that “rebooting” it is a much better option. That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
Sure. Just add the number of deaths to the utility function with an appropriate multiplier, so that world states obtained through killing get penalized. Of course, an AI who wishes to get rid of humanity in order to set up a better world unobstructed could attempt to circumvent the limitation: create an infertility epidemic to extinguish humanity within a few generations, fudge genetics to tame it (even if it is only temporary), and so forth.
Ultimately, though, it seems that you just want the AI to do whatever you want it to do and nothing you don’t want it to do. I very much doubt there is any formalization of what you, me, or any other human really wants. The society we have now is the result of social progress that elders have fought tooth and nail against. Given that in general humans can’t get their own offspring to respect their taboos, what if your grandchildren come to embrace some options that you find repugnant or disagree with your idea of utopia? What if the AI tells itself “I can’t kill humanity now, but if I do this and that, eventually, it will give me the mandate”? Society is an iceberg drifting along the current, only sensing the direction it’s going at the moment, but with poor foresight as to what the direction is going to be after that.
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans. Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
That would be bad, but it would still be way better than replacing us with paperclippers or orgasmium.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should. So I’m fine with future creatures that have something like human nature deciding some new society I’m kind of uncomfortable with is the best way to express their natures. What I’m not fine with is throwing human nature out and starting from scratch with something new, which is what I think a utilitarian AI would do.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.” For instance, if given a choice between creating an AI with human-like values, and creating a human sociopath, I would pick the AI. And it wouldn’t just be because there was a chance the sociopath would harm others. I simply consider the values of the AI more worthy of creation than the sociopath’s.
I don’t necessarily disagree. If having a large population of creatures with humane values and high welfare was assured then it might be better to have a variety of creatures. But I still think maybe there should be some limits on the sort of creatures we should create, i.e. lawful creativity. Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
It’s quite hard to tell. “Progress” is always relative to the environment you grew up in and on which your ideas and aspirations are based. At the scale of a human life, our trajectory looks a lot like a straight line, but for all we know, it could be circular. At every point on the circle, we would aim to follow the tangent, and it would look like that’s what we are doing. However, as we move along, the tangent would shift ever so subtly and over the course of millennia we would end up doing a roundabout.
I am not saying that’s precisely what we are doing, but there is some truth to it: human goals and values shift. Our environment and upbringing mold us very deeply, in a way that we cannot really abstract away. A big part of what we consider “ideal” is therefore a function of that imprint. However, we rarely ponder the fact that people born and raised in our “ideal world” would be molded differently and thus may have a fundamentally different outlook on life, including wishing for something else. That’s a bit contrived, of course, but it would probably be possible to make a society which wants X when raised on Y, and Y when raised on X, so that it would constantly oscillate between X and Y. We would have enough foresight to figure out a simple oscillator, but if ethics were a kind of semi-random walk, I don’t think it would be obvious. The idea that we are converging towards something might be a bit of an illusion due to underestimating how different future people will be from ourselves.
I suspect the negative aspects of our natures occur primarily when access to resources is strained. If every human is sheltered, well fed, has access to plentiful energy, and so on, there aren’t really be any problems to blame on anyone, so everything should work fine (for the most part, anyway). In a sense, this simplifies the task of the AI: you ask it to optimize supply to existing demand and the rest is smooth sailing.
Still, the criterion is explicitly based on human values. Even if not human specifically, you want “human-like” creatures.
Still fairly anthropomorphic (not necessarily a bad thing, just an observation). In principle, extremely interesting entities could have no conception of self. Sympathy is only relevant to social entities—but why not create solitary ones as well? As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former? I mean, when you think about it, in order to bond with each other, social entities must share a lot of preferences, the encoding of which is redundant. Solitary entities with fringe preferences could thus be a cheap and easy way to increase variety.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
It’s not about having a specialized interest and exploring it. A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.” They wouldn’t be like an autistic savant who knows one subject inside and out. They’d be little better than a wirehead. Someone with narrow interests still explores every single aspect of that interest in great detail. A creature with no boredom would find one tiny aspect of that interest and do it forever.
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
Fair enough, though at the level of omnipotence we’re supposing, there would be no chance meetups. You might as well just isolate the creature and be done with it.
Or it would do it once, and then die happy. Human-like entities might have a lifespan of centuries, and then you would have ephemeral beings living their own limited fantasy for thirty seconds. I mean, why not? We are all bound to repeat ourselves once our interests are exhausted—perhaps entities could be made to embrace death when that happens.
I agree, though an entity with the power to choose the kind of creatures that come to exist probably wouldn’t have much difficulty doing it safely.