I consider it almost certain that if we were to create a utilitarian AI it would kill the entire human race and replace it with creatures whose preferences are easier to satisfy. And by “easier to satisfy” I mean “simpler and less ambitious,” not that the creatures are more mentally and physically capable of satisfying humane desires.
It would not necessarily kill off humanity to replace it by something else, though. Looking at the world right now, many countries run smoothly, and others horribly, even though they are all inhabited and governed by humans. Even if you made the AI “prefer” human beings, it could still evaluate that “fixing” humanity would be too slow and costly and that “rebooting” it is a much better option. That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
Once someone has been brought into existence we have a greater duty to make sure they stay alive and happy then we do to create new people. There may be some vastly huge amount of happy people that it’s okay to kill one slightly-less-happy-person in order to create, but that number should be way, way, way, way, bigger than 1.
Sure. Just add the number of deaths to the utility function with an appropriate multiplier, so that world states obtained through killing get penalized. Of course, an AI who wishes to get rid of humanity in order to set up a better world unobstructed could attempt to circumvent the limitation: create an infertility epidemic to extinguish humanity within a few generations, fudge genetics to tame it (even if it is only temporary), and so forth.
Ultimately, though, it seems that you just want the AI to do whatever you want it to do and nothing you don’t want it to do. I very much doubt there is any formalization of what you, me, or any other human really wants. The society we have now is the result of social progress that elders have fought tooth and nail against. Given that in general humans can’t get their own offspring to respect their taboos, what if your grandchildren come to embrace some options that you find repugnant or disagree with your idea of utopia? What if the AI tells itself “I can’t kill humanity now, but if I do this and that, eventually, it will give me the mandate”? Society is an iceberg drifting along the current, only sensing the direction it’s going at the moment, but with poor foresight as to what the direction is going to be after that.
I’ve noticed there does not seem to be much interest in the main question I am interested in, which is, “Why make humans and not something else?”
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans. Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
That would be bad, but it would still be way better than replacing us with paperclippers or orgasmium.
The society we have now is the result of social progress that elders have fought tooth and nail against.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should. So I’m fine with future creatures that have something like human nature deciding some new society I’m kind of uncomfortable with is the best way to express their natures. What I’m not fine with is throwing human nature out and starting from scratch with something new, which is what I think a utilitarian AI would do.
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.” For instance, if given a choice between creating an AI with human-like values, and creating a human sociopath, I would pick the AI. And it wouldn’t just be because there was a chance the sociopath would harm others. I simply consider the values of the AI more worthy of creation than the sociopath’s.
Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
I don’t necessarily disagree. If having a large population of creatures with humane values and high welfare was assured then it might be better to have a variety of creatures. But I still think maybe there should be some limits on the sort of creatures we should create, i.e. lawful creativity. Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
It’s quite hard to tell. “Progress” is always relative to the environment you grew up in and on which your ideas and aspirations are based. At the scale of a human life, our trajectory looks a lot like a straight line, but for all we know, it could be circular. At every point on the circle, we would aim to follow the tangent, and it would look like that’s what we are doing. However, as we move along, the tangent would shift ever so subtly and over the course of millennia we would end up doing a roundabout.
I am not saying that’s precisely what we are doing, but there is some truth to it: human goals and values shift. Our environment and upbringing mold us very deeply, in a way that we cannot really abstract away. A big part of what we consider “ideal” is therefore a function of that imprint. However, we rarely ponder the fact that people born and raised in our “ideal world” would be molded differently and thus may have a fundamentally different outlook on life, including wishing for something else. That’s a bit contrived, of course, but it would probably be possible to make a society which wants X when raised on Y, and Y when raised on X, so that it would constantly oscillate between X and Y. We would have enough foresight to figure out a simple oscillator, but if ethics were a kind of semi-random walk, I don’t think it would be obvious. The idea that we are converging towards something might be a bit of an illusion due to underestimating how different future people will be from ourselves.
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should.
I suspect the negative aspects of our natures occur primarily when access to resources is strained. If every human is sheltered, well fed, has access to plentiful energy, and so on, there aren’t really be any problems to blame on anyone, so everything should work fine (for the most part, anyway). In a sense, this simplifies the task of the AI: you ask it to optimize supply to existing demand and the rest is smooth sailing.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.”
Still, the criterion is explicitly based on human values. Even if not human specifically, you want “human-like” creatures.
Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
Still fairly anthropomorphic (not necessarily a bad thing, just an observation). In principle, extremely interesting entities could have no conception of self. Sympathy is only relevant to social entities—but why not create solitary ones as well? As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former? I mean, when you think about it, in order to bond with each other, social entities must share a lot of preferences, the encoding of which is redundant. Solitary entities with fringe preferences could thus be a cheap and easy way to increase variety.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
Sympathy is only relevant to social entities—but why not create solitary ones as well?
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former?
It’s not about having a specialized interest and exploring it. A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.” They wouldn’t be like an autistic savant who knows one subject inside and out. They’d be little better than a wirehead. Someone with narrow interests still explores every single aspect of that interest in great detail. A creature with no boredom would find one tiny aspect of that interest and do it forever.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
Fair enough, though at the level of omnipotence we’re supposing, there would be no chance meetups. You might as well just isolate the creature and be done with it.
A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.”
Or it would do it once, and then die happy. Human-like entities might have a lifespan of centuries, and then you would have ephemeral beings living their own limited fantasy for thirty seconds. I mean, why not? We are all bound to repeat ourselves once our interests are exhausted—perhaps entities could be made to embrace death when that happens.
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
I agree, though an entity with the power to choose the kind of creatures that come to exist probably wouldn’t have much difficulty doing it safely.
It would not necessarily kill off humanity to replace it by something else, though. Looking at the world right now, many countries run smoothly, and others horribly, even though they are all inhabited and governed by humans. Even if you made the AI “prefer” human beings, it could still evaluate that “fixing” humanity would be too slow and costly and that “rebooting” it is a much better option. That is to say, it would kill all humans, restructure the whole planet, and then repopulate the planet with human beings devoid of cultural biases, ensuring plentiful resources throughout. But the genetic makeup would stay the exact same.
Sure. Just add the number of deaths to the utility function with an appropriate multiplier, so that world states obtained through killing get penalized. Of course, an AI who wishes to get rid of humanity in order to set up a better world unobstructed could attempt to circumvent the limitation: create an infertility epidemic to extinguish humanity within a few generations, fudge genetics to tame it (even if it is only temporary), and so forth.
Ultimately, though, it seems that you just want the AI to do whatever you want it to do and nothing you don’t want it to do. I very much doubt there is any formalization of what you, me, or any other human really wants. The society we have now is the result of social progress that elders have fought tooth and nail against. Given that in general humans can’t get their own offspring to respect their taboos, what if your grandchildren come to embrace some options that you find repugnant or disagree with your idea of utopia? What if the AI tells itself “I can’t kill humanity now, but if I do this and that, eventually, it will give me the mandate”? Society is an iceberg drifting along the current, only sensing the direction it’s going at the moment, but with poor foresight as to what the direction is going to be after that.
Because we are humans and we want more of ourselves, so of course we will work towards that particular goal. You won’t find any magical objective reason to do it. Sure, we are sentient, intelligent, complex, but if those were the criteria, then we would want to make more AI, not more humans. Personally, I can’t see the utility of plastering the whole universe with humans who will never see more than their own little sector, so I would taper off utility with the number of humans, so that eventually you just have to create other stuff. Basically, I would give high utility to variety. It’s more interesting that way.
That would be bad, but it would still be way better than replacing us with paperclippers or orgasmium.
That’s true, but if it’s “progress” then it must be progress towards something. Will we eventually arrive at our destination, decide society is pretty much perfect, and then stop? Is progress somehow asymptotic so we’ll keep progressing and never quite reach our destination?
The thing is, it seems to me that what we’ve been progressing towards is greater expression of our human natures. Greater ability to do what the most positive parts of our natures think we should. So I’m fine with future creatures that have something like human nature deciding some new society I’m kind of uncomfortable with is the best way to express their natures. What I’m not fine with is throwing human nature out and starting from scratch with something new, which is what I think a utilitarian AI would do.
I didn’t literally mean humans, I meant “Creatures with the sorts of goals, values, and personalities that humans have.” For instance, if given a choice between creating an AI with human-like values, and creating a human sociopath, I would pick the AI. And it wouldn’t just be because there was a chance the sociopath would harm others. I simply consider the values of the AI more worthy of creation than the sociopath’s.
I don’t necessarily disagree. If having a large population of creatures with humane values and high welfare was assured then it might be better to have a variety of creatures. But I still think maybe there should be some limits on the sort of creatures we should create, i.e. lawful creativity. Eliezer has suggested that consciousness, sympathy, and boredom are the essential characteristics any intelligent creature should have. I’d love for there to be a wide variety of creatures, but maybe it would be best if they all had those characteristics.
It’s quite hard to tell. “Progress” is always relative to the environment you grew up in and on which your ideas and aspirations are based. At the scale of a human life, our trajectory looks a lot like a straight line, but for all we know, it could be circular. At every point on the circle, we would aim to follow the tangent, and it would look like that’s what we are doing. However, as we move along, the tangent would shift ever so subtly and over the course of millennia we would end up doing a roundabout.
I am not saying that’s precisely what we are doing, but there is some truth to it: human goals and values shift. Our environment and upbringing mold us very deeply, in a way that we cannot really abstract away. A big part of what we consider “ideal” is therefore a function of that imprint. However, we rarely ponder the fact that people born and raised in our “ideal world” would be molded differently and thus may have a fundamentally different outlook on life, including wishing for something else. That’s a bit contrived, of course, but it would probably be possible to make a society which wants X when raised on Y, and Y when raised on X, so that it would constantly oscillate between X and Y. We would have enough foresight to figure out a simple oscillator, but if ethics were a kind of semi-random walk, I don’t think it would be obvious. The idea that we are converging towards something might be a bit of an illusion due to underestimating how different future people will be from ourselves.
I suspect the negative aspects of our natures occur primarily when access to resources is strained. If every human is sheltered, well fed, has access to plentiful energy, and so on, there aren’t really be any problems to blame on anyone, so everything should work fine (for the most part, anyway). In a sense, this simplifies the task of the AI: you ask it to optimize supply to existing demand and the rest is smooth sailing.
Still, the criterion is explicitly based on human values. Even if not human specifically, you want “human-like” creatures.
Still fairly anthropomorphic (not necessarily a bad thing, just an observation). In principle, extremely interesting entities could have no conception of self. Sympathy is only relevant to social entities—but why not create solitary ones as well? As for boredom, what makes a population of entities that seek variety in their lives better than one of entities who each have highly specialized interests (all different from each other)? As a whole, wouldn’t the latter display more variation than the former? I mean, when you think about it, in order to bond with each other, social entities must share a lot of preferences, the encoding of which is redundant. Solitary entities with fringe preferences could thus be a cheap and easy way to increase variety.
Or how about creating psychopaths and putting them in controlled environments that they can destroy at will, or creating highly violent entities to throw in fighting pits? Isn’t there a point where this is preferable to creating yet another conscious creature capable of sympathy and boredom?
A creature that loves solitude might not necessarily be bad to create. But it would still be good to give it capacity for sympathy for pragmatic reasons, to ensure that if it ever did meet another creature it would want to treat it kindly and avoid harming it.
It’s not about having a specialized interest and exploring it. A creature with no concept of boredom would would, (to paraphrase Eliezer), “play the same screen of the same level of the same fun videogame over and over again.” They wouldn’t be like an autistic savant who knows one subject inside and out. They’d be little better than a wirehead. Someone with narrow interests still explores every single aspect of that interest in great detail. A creature with no boredom would find one tiny aspect of that interest and do it forever.
Yes, I concede that if there is a sufficient quantity of creatures with humane values, it might be good to create other types of creatures for variety’s sake. However, such creatures could be potentially dangerous, we’d have to be very careful.
Fair enough, though at the level of omnipotence we’re supposing, there would be no chance meetups. You might as well just isolate the creature and be done with it.
Or it would do it once, and then die happy. Human-like entities might have a lifespan of centuries, and then you would have ephemeral beings living their own limited fantasy for thirty seconds. I mean, why not? We are all bound to repeat ourselves once our interests are exhausted—perhaps entities could be made to embrace death when that happens.
I agree, though an entity with the power to choose the kind of creatures that come to exist probably wouldn’t have much difficulty doing it safely.