Anyone who predicts that some decision may result in the world being optimized according to something other than their own values, and is okay with that, is probably not thinking about terminal values. More likely, they’re thinking that humanity (or its successor) will clarify its terminal values and/or get better at reasoning from them to instrumental values to concrete decisions, and that their understanding of their own values will follow that. Of course, when people are considering whether it’s a good idea to create a certain kind of mind, that kind of thinking probably means they’re presuming that Friendliness comes mostly automatically. It’s hard for the idea of an agent with different terminal values to really sink in; I’ve had a little bit of experience with trying to explain to people the idea of minds with really fundamentally different values, and they still often try to understand it in terms of justifications that are compelling (or at least comprehensible) to them personally. Like, imagining that a paperclip maximizer is just like a quirky highly-intelligent human who happens to love paperclips, or is under the mistaken impression that maximizing paperclips is the right thing could do and could be talked out of it by the right arguments. I think Ben and Robin (“Human value … will continue to [evolve]”, “I think it is ok (if not ideal) if our descendants’ values deviate from ours”) are thinking as though AI-aided value loss would be similar to the gradual refinement of instrumental values that takes place within societies consisting of largely-similar human brains (the kind of refinement that we can anticipate in advance and expect we’ll be okay with), rather than something that could result in powerful minds that actually don’t care about morality.
And I feel like anyone who really has internalized the idea that minds are allowed to fundamentally care about completely different things that we do, and still thinks they’re okay with that actually happening, probably just haven’t taken five minutes to think creatively about what kinds of terrible worlds or non-worlds could be created as a result of powerfully optimizing for a value system based on our present muddled values plus just a little bit of drift.
I suppose what remains are the people who don’t buy the generalized idea of optimization processes as a superset of person-like minds in the first place, with really powerful optimization processes being another subset. Would Ben be in that group? Some of his statements (e.g. “It’s possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I’m not terribly certain of this, any more than I’m terribly certain of its opposite.” (still implying a privileged 50-50 credence in this unsupported idea)) do suggest that he is expecting AGIs to automatically be people in some sense.
(I do think that “Value Deathism” gives the wrong impression of what this post is about. Something like “Value Loss Escapism” might be better; the analogy to deathism seems too much like a surface analogy of minor relevance. I’m not convinced that the tendency to believe that value loss is illusory or desirable is caused by the same thought processes that cause those beliefs about death. More likely, most people who try to think about AI ethics are going to be genuinely really confused about it for a while or forever, whereas “is death okay/good?” is not a confusing question.)
Ok. Well done. You have managed to frighten me. Frightened me enough to make me ask the question: “Just why do we want to build a powerful optimizer, anyways?”
More likely, most people who try to think about AI ethics are going to be genuinely really confused about it for a while or forever, whereas “is death okay/good?” is not a confusing question.
Oh, yeah. Now I remember. The reason we want to build a powerful optimizer is because some people think that “Is death okay/good?” is not a confusing question but that the question “Is it okay/good to risk the future of the Earth by building an amoral agent much more powerful than ourselves?” is confusing.
Ok. Well done. You have managed to frighten me. Frightened me enough to make me ask the question: “Just why do we want to build a powerful optimizer, anyways?”
I feel like I remember trying to answer the same question (asked by you) before, but essentially, the answer is that (1) eventually (assuming humanity survives long enough) someone is probably going to build one anyway, probably without being extremely careful about understanding what kind of optimizer it’s goint to be, and getting FAI before then will probably be the only way to prevent it; (2) there are many reasons why humanity might not survive long enough for that to happen — it’s likely that humanity’s technological progress over the next century will continuously lower the amount of skill, intelligence, and resources needed to accidentally or intentionally do terrible things — and getting FAI before then may be the best long-term solution to that; (3) given that pursuing FAI is likely necessary to avert other huge risks, and is therefore less risky than doing nothing, it’s an especially good cause considering that it subsumes all other humanitarian causes (if executed successfully).
I feel like I remember trying to answer the same question (asked by you) before …
Perhaps you did. This time, my question was mostly rhetorical, but since you gave a thoughtful response, it seems a shame to waste it.
(1) eventually … someone is probably going to build one anyway, probably without being extremely careful …, and getting FAI before then will probably be the only way to prevent it;
Uh. Prevent it how. I’m curious how that particular sausage will be made.
(2) … it’s likely that humanity’s technological progress over the next century will continuously lower the amount of skill, intelligence, and resources needed to accidentally or intentionally do terrible things — and getting FAI before then may be the best long-term solution to that;
More sausage. How does the FAI solve that problem? It seemed that you said the root cause of the problem was technological progress, but perhaps I misunderstood.
(3) … it subsumes all other humanitarian causes …
Hmmm. Amnesty International, Doctors without Borders, and the Humane Society are three humanitarian causes that come to mind. FAI subsumes these … how, exactly?
Again, my questions are somewhat rhetorical. If I really wanted to engage in this particular dialog, I should probably do so in a top-level posting. So please do not feel obligated to respond.
It is just that if Ben Goertzel is so confused as to hope that any sufficiently intelligent entity will automatically empathize with humans, then how much confusion exists here regarding just how much humans will automatically accept the idea of sharing a planet with an FAI? Smart people can have amazing blind spots.
If I knew how that sausage will be made, I’d make it myself. The point of FAI is to do a massive amount of good that we’re not smart enough to figure out how to do on our own.
Hmmm. Amnesty International, Doctors without Borders, and the Humane Society are three humanitarian causes that come to mind. FAI subsumes these … how, exactly?
If humanity’s extrapolated volition largely agrees that those causes are working on important problems, problems urgent enough that we’re okay with giving up the chance to solve them ourselves if they can be solved faster and better by superintelligence, then it’ll do so. Doctors Without Borders? We shouldn’t be needing doctors (or borders) anymore. Saying how that happens is explicitly not our job — as I said, that’s the whole point of making something massively smarter than we are. Don’t underestimate something potentially hundreds or thousands or billions of times smarter than every human put together.
I actually think we know how to do the major ‘trauma care for civilization’ without FAI at this point. FAI looks much cheaper and possibly faster though, so in the process of doing the “trauma care” we should obviously fund it as a top priority. I basically see it as the largest “victory point” option in a strategy game.
When answering questions like this, it’s important to make the following disclaimer: I do not know what the best solution is. If a genuine FAI considers these questions, ve will probably come up with something much better. I’m proposing ideas solely to show that some options exist which are strictly preferable to human extinction, dystopias, and the status quo.
It’s pretty clear that (1) we don’t want to be exterminated by a rogue AI, or nanotech, or plague, or nukes, (2) we want to have aging and disease fixed for us (at least for long enough to sit back and clearly think about what we want of the future), and (3) we don’t want an FAI to strip us of all autonomy and growth in order to protect us. There are plenty of ways to avoid both these possibilities. For one, the FAI could basically act as a good Deist god should have: fix the most important aspects of aging, disease and dysfunction, make murder (and construction of superweapons/unsafe AIs) impossible via occasional miraculous interventions, but otherwise hang back and let us do our growing up. (If at some point humanity decides we’ve outgrown its help, it should fade out at our request.) None of this is technically that difficult, given nanotech.
Personally, I think a FAI could do much better than this scenario, but if I talked about that we’d get lost arguing the weird points. I just want to ask, is there a sense in which this lower bound would really seem like a dystopia to you? (If so, please think for a few minutes about possible fixes first.)
I just want to ask, is there a sense in which this lower bound would really seem like a dystopia to you?
No, not at all. It sounds pretty good. However, my opinion of what you describe is not the issue. The issue is what ordinary, average, stupid, paranoid, and conservative people think about the prospect of a powerful AI totally changing their lives when they have only your self-admittedly ill informed assurances regarding how good it is going to be.
Please don’t move the goalposts. I’d much rather know whether I’m convincing you than whether I’m convincing a hypothetical average Joe. Figuring out a political case for FAI is important, but secondary to figuring out whether it’s actually possible and desirable.
Ok, I don’t mean to be unfairly moving goalposts around. But I will point out that gaining my assent to a hypothetical is not the same as gaining my agreement regarding the course that ought to be followed into an uncertain future.
That’s fair enough. The choice of course depends on whether FAI is even possible, and whether any group could be trusted to build it. But conditional on those factors, we can at least agree that such a thing is desirable.
And I feel like anyone who really has internalized the idea that minds are allowed to fundamentally care about completely different things...
What is questionable is not the possibility of fundamentally different values but that they could accidentally be implemented. What you are suggesting is that some intelligence is able to evolve a vast repertoire of heuristics, acquire vast amounts of knowledge about the universe, dramatically improve its cognitive flexibility and yet never evolve its values but keep its volition at the level of a washing machine. I think this idea is flawed, or at least not sufficiently backed up to take it serious right now. I believe that such an incentive, or any incentive, will have to be deliberately and carefully hardcoded or evolved. Otherwise we are merely talking about grey goo scenarios.
Is Death absolutely bad or not is a somewhat confusing question. If you can’t phrase questions, at an emotional level, only choose between them, that can become “Is death okay/good” by pattern match.
Anyone who predicts that some decision may result in the world being optimized according to something other than their own values, and is okay with that, is probably not thinking about terminal values. More likely, they’re thinking that humanity (or its successor) will clarify its terminal values and/or get better at reasoning from them to instrumental values to concrete decisions, and that their understanding of their own values will follow that. Of course, when people are considering whether it’s a good idea to create a certain kind of mind, that kind of thinking probably means they’re presuming that Friendliness comes mostly automatically. It’s hard for the idea of an agent with different terminal values to really sink in; I’ve had a little bit of experience with trying to explain to people the idea of minds with really fundamentally different values, and they still often try to understand it in terms of justifications that are compelling (or at least comprehensible) to them personally. Like, imagining that a paperclip maximizer is just like a quirky highly-intelligent human who happens to love paperclips, or is under the mistaken impression that maximizing paperclips is the right thing could do and could be talked out of it by the right arguments. I think Ben and Robin (“Human value … will continue to [evolve]”, “I think it is ok (if not ideal) if our descendants’ values deviate from ours”) are thinking as though AI-aided value loss would be similar to the gradual refinement of instrumental values that takes place within societies consisting of largely-similar human brains (the kind of refinement that we can anticipate in advance and expect we’ll be okay with), rather than something that could result in powerful minds that actually don’t care about morality.
And I feel like anyone who really has internalized the idea that minds are allowed to fundamentally care about completely different things that we do, and still thinks they’re okay with that actually happening, probably just haven’t taken five minutes to think creatively about what kinds of terrible worlds or non-worlds could be created as a result of powerfully optimizing for a value system based on our present muddled values plus just a little bit of drift.
I suppose what remains are the people who don’t buy the generalized idea of optimization processes as a superset of person-like minds in the first place, with really powerful optimization processes being another subset. Would Ben be in that group? Some of his statements (e.g. “It’s possible that with sufficient real-world intelligence tends to come a sense of connectedness with the universe that militates against squashing other sentiences. But I’m not terribly certain of this, any more than I’m terribly certain of its opposite.” (still implying a privileged 50-50 credence in this unsupported idea)) do suggest that he is expecting AGIs to automatically be people in some sense.
(I do think that “Value Deathism” gives the wrong impression of what this post is about. Something like “Value Loss Escapism” might be better; the analogy to deathism seems too much like a surface analogy of minor relevance. I’m not convinced that the tendency to believe that value loss is illusory or desirable is caused by the same thought processes that cause those beliefs about death. More likely, most people who try to think about AI ethics are going to be genuinely really confused about it for a while or forever, whereas “is death okay/good?” is not a confusing question.)
Ok. Well done. You have managed to frighten me. Frightened me enough to make me ask the question: “Just why do we want to build a powerful optimizer, anyways?”
Oh, yeah. Now I remember. The reason we want to build a powerful optimizer is because some people think that “Is death okay/good?” is not a confusing question but that the question “Is it okay/good to risk the future of the Earth by building an amoral agent much more powerful than ourselves?” is confusing.
I feel like I remember trying to answer the same question (asked by you) before, but essentially, the answer is that (1) eventually (assuming humanity survives long enough) someone is probably going to build one anyway, probably without being extremely careful about understanding what kind of optimizer it’s goint to be, and getting FAI before then will probably be the only way to prevent it; (2) there are many reasons why humanity might not survive long enough for that to happen — it’s likely that humanity’s technological progress over the next century will continuously lower the amount of skill, intelligence, and resources needed to accidentally or intentionally do terrible things — and getting FAI before then may be the best long-term solution to that; (3) given that pursuing FAI is likely necessary to avert other huge risks, and is therefore less risky than doing nothing, it’s an especially good cause considering that it subsumes all other humanitarian causes (if executed successfully).
Perhaps you did. This time, my question was mostly rhetorical, but since you gave a thoughtful response, it seems a shame to waste it.
Uh. Prevent it how. I’m curious how that particular sausage will be made.
More sausage. How does the FAI solve that problem? It seemed that you said the root cause of the problem was technological progress, but perhaps I misunderstood.
Hmmm. Amnesty International, Doctors without Borders, and the Humane Society are three humanitarian causes that come to mind. FAI subsumes these … how, exactly?
Again, my questions are somewhat rhetorical. If I really wanted to engage in this particular dialog, I should probably do so in a top-level posting. So please do not feel obligated to respond.
It is just that if Ben Goertzel is so confused as to hope that any sufficiently intelligent entity will automatically empathize with humans, then how much confusion exists here regarding just how much humans will automatically accept the idea of sharing a planet with an FAI? Smart people can have amazing blind spots.
If I knew how that sausage will be made, I’d make it myself. The point of FAI is to do a massive amount of good that we’re not smart enough to figure out how to do on our own.
If humanity’s extrapolated volition largely agrees that those causes are working on important problems, problems urgent enough that we’re okay with giving up the chance to solve them ourselves if they can be solved faster and better by superintelligence, then it’ll do so. Doctors Without Borders? We shouldn’t be needing doctors (or borders) anymore. Saying how that happens is explicitly not our job — as I said, that’s the whole point of making something massively smarter than we are. Don’t underestimate something potentially hundreds or thousands or billions of times smarter than every human put together.
I actually think we know how to do the major ‘trauma care for civilization’ without FAI at this point. FAI looks much cheaper and possibly faster though, so in the process of doing the “trauma care” we should obviously fund it as a top priority. I basically see it as the largest “victory point” option in a strategy game.
When answering questions like this, it’s important to make the following disclaimer: I do not know what the best solution is. If a genuine FAI considers these questions, ve will probably come up with something much better. I’m proposing ideas solely to show that some options exist which are strictly preferable to human extinction, dystopias, and the status quo.
It’s pretty clear that (1) we don’t want to be exterminated by a rogue AI, or nanotech, or plague, or nukes, (2) we want to have aging and disease fixed for us (at least for long enough to sit back and clearly think about what we want of the future), and (3) we don’t want an FAI to strip us of all autonomy and growth in order to protect us. There are plenty of ways to avoid both these possibilities. For one, the FAI could basically act as a good Deist god should have: fix the most important aspects of aging, disease and dysfunction, make murder (and construction of superweapons/unsafe AIs) impossible via occasional miraculous interventions, but otherwise hang back and let us do our growing up. (If at some point humanity decides we’ve outgrown its help, it should fade out at our request.) None of this is technically that difficult, given nanotech.
Personally, I think a FAI could do much better than this scenario, but if I talked about that we’d get lost arguing the weird points. I just want to ask, is there a sense in which this lower bound would really seem like a dystopia to you? (If so, please think for a few minutes about possible fixes first.)
No, not at all. It sounds pretty good. However, my opinion of what you describe is not the issue. The issue is what ordinary, average, stupid, paranoid, and conservative people think about the prospect of a powerful AI totally changing their lives when they have only your self-admittedly ill informed assurances regarding how good it is going to be.
Please don’t move the goalposts. I’d much rather know whether I’m convincing you than whether I’m convincing a hypothetical average Joe. Figuring out a political case for FAI is important, but secondary to figuring out whether it’s actually possible and desirable.
Ok, I don’t mean to be unfairly moving goalposts around. But I will point out that gaining my assent to a hypothetical is not the same as gaining my agreement regarding the course that ought to be followed into an uncertain future.
That’s fair enough. The choice of course depends on whether FAI is even possible, and whether any group could be trusted to build it. But conditional on those factors, we can at least agree that such a thing is desirable.
I’d really appreciate your attempting to write up some SIAI literature to communicate these points to the audiences you are talking about. It is hard.
What is questionable is not the possibility of fundamentally different values but that they could accidentally be implemented. What you are suggesting is that some intelligence is able to evolve a vast repertoire of heuristics, acquire vast amounts of knowledge about the universe, dramatically improve its cognitive flexibility and yet never evolve its values but keep its volition at the level of a washing machine. I think this idea is flawed, or at least not sufficiently backed up to take it serious right now. I believe that such an incentive, or any incentive, will have to be deliberately and carefully hardcoded or evolved. Otherwise we are merely talking about grey goo scenarios.
Is Death absolutely bad or not is a somewhat confusing question. If you can’t phrase questions, at an emotional level, only choose between them, that can become “Is death okay/good” by pattern match.