that would have to be a really sophisticated bug to misinterpret “always answer questions thruthfully as far as possible while admitting uncertainty” as “kill all humans”. I’d imagine that something as drastic as that could be found and corrected long before that. Consider that you have its goal set to this. It knows no other motivation but to respond thruthfully. It doesn’t care about the survival of humanity, or itself or about how reality really is. All it cares for is to answer the questions to the best of its abilities.
I don’t think that this goal would be all too hard to define either, as “the truth” is a pretty simple concept. As long it deals with uncertainty in the right way (by admitting it), how could this be misinterpreted?
Friendliness is far harder to define because we don’t even know a definition for it ourselves. There are far too many things to consider when defining “friendliness”.
Trivial Failure Case: The AI turns the universe into hardware to support really big computations, so it can be really sure it’s got the right answer, and also callibrate itself really well on the uncertainty.
I don’t think that this goal would be all too hard to define either, as “the truth” is a pretty simple concept.
Legions of philosophers would disagree with you
that would have to be a really sophisticated bug to misinterpret “always answer questions thruthfully as far as possible while admitting uncertainty” as “kill all humans”.
Maybe “Humans should die” is the truth. Maybe humans are bad for the planet. One of the problems with FAI is that you don’t want to give it objective morality because of that risk. You want it to side with humans. Hence “friendly” AI rather than “righteous AI”.
They just bicker endlessly about uncertainty. “can you really know that 1+1=2?”. No, but it can be used as valid until proven otherwise (which will never happen). As I said, the AI would need to understand the idea of uncertainty.
Maybe “Humans should die” is the truth. Maybe humans are bad for the planet. One of the problems >with FAI is that you don’t want to give it objective morality because of that risk. You want it to side with >humans. Hence “friendly” AI rather than “righteous AI”.
there is no such thing as objective morality. Good and evil are subjective ideas, nothing more. Firstly, unless someone explicitly tells the AI that it is a fundamental truth that nature is important to preserve, this can not happen. Secondly, the AI would also have to be incredibly gullible to just swallow such a claim. Thirdly, even if the AI does believe that, it will plainly say so to the people it is conversing with, in accordance with its goal to always tell the truth, thus warning us of this bug.
They just bicker endlessly about uncertainty. “can you really know that 1+1=2?”.
I agree with you that I don’t think a AGI would have the same problems humans have with the concept of truth. However, what you described is neither the issues philosophers raise nor the sorts of big-universe issues the AI might get stuck on.
But wouldn’t that actually support my approach? Assuming that there really is something important that all of humanity misses but the AI understands:
-If you hardcode the AI’s optimal goal based on human deliberations you are guaranteed to miss this important thing.
-If you use the method I suggested, the AI will, driven by the desire to speak the truth, try to explain the problem to the humans who will in turn tell the AI what they think of that.
[Philosophers] just bicker endlessly about uncertainty. “can you really know that 1+1=2?”.
I don’t think that is a good characterisation of the debate. It isn’t just about uncertainty.
there is no such thing as objective morality. Good and evil are subjective ideas, nothing more.
That’s what you think. Some smart humans disagree with you. A supermsart AI might disagree with you and might
be right. How can you second guess it? You cannot predict the behaviour of a supersmart AI on the basis that i t will
agree with you, who are less smart.
Firstly, unless someone explicitly tells the AI that it is a fundamental truth that nature is important to preserve, this can not happen.
Unless it figures it out.
Secondly, the AI would also have to be incredibly gullible to just swallow such a claim.
Why would that require more gullibility than “species X is more important than all the others”? That doesn’t
even looklike a moral claim.
Thirdly, even if the AI does believe that, it will plainly say so to the people it is conversing with, in accordance with its goal to always tell the truth, thus warning us of this bug.
If it has “swallowedthat* claim. You are assuming that the AI has a free choice about some goals and is just programmed with others.
If it has “swallowed* that claim. You are assuming that the AI has a free choice about some goals >and is just programmed with others.
This is the important part.
the “optimal goal” is not actually controlling the AI.
the “optimal goal” is merely the subject of a discussion.
what is controlling the AI is the desire the tell the truth to the humans it is talking to, nothing more.
Why would that require more gullibility than “species X is more important than all the others”? >That doesn’t even look like a moral claim.
The entire discussion is not supposed to unearth some kind of pure, inherently good, perfect optimal goal that transcends all reason and is true by virtue of existing or something.
The AI is supposed to take the human POV and think “if I were these humans, what would I want the AI’s goal to be”.
I didn’t mention this explicitly because I didn’t think it was necessary but the “optimal goal” is purely subjective from the POV of humanity and the AI is aware of this.
some kind of pure, inherently good, perfect optimal goal that transcends all reason and is true by virtue of existing or something.
But if that is true, the AI will say so. What’s more, you kind of need the AI to refrain from acting on it, if it is a human-unfriendly objective moral truth. There are ethical puzzles where it is apparently right to lie or keep schtum, because of the consequences of telling the truth.
that would have to be a really sophisticated bug to misinterpret “always answer questions thruthfully as far as possible while admitting uncertainty” as “kill all humans”. I’d imagine that something as drastic as that could be found and corrected long before that. Consider that you have its goal set to this. It knows no other motivation but to respond thruthfully. It doesn’t care about the survival of humanity, or itself or about how reality really is. All it cares for is to answer the questions to the best of its abilities.
I don’t think that this goal would be all too hard to define either, as “the truth” is a pretty simple concept. As long it deals with uncertainty in the right way (by admitting it), how could this be misinterpreted? Friendliness is far harder to define because we don’t even know a definition for it ourselves. There are far too many things to consider when defining “friendliness”.
Trivial Failure Case: The AI turns the universe into hardware to support really big computations, so it can be really sure it’s got the right answer, and also callibrate itself really well on the uncertainty.
Legions of philosophers would disagree with you
Maybe “Humans should die” is the truth. Maybe humans are bad for the planet. One of the problems with FAI is that you don’t want to give it objective morality because of that risk. You want it to side with humans. Hence “friendly” AI rather than “righteous AI”.
They just bicker endlessly about uncertainty. “can you really know that 1+1=2?”. No, but it can be used as valid until proven otherwise (which will never happen). As I said, the AI would need to understand the idea of uncertainty.
there is no such thing as objective morality. Good and evil are subjective ideas, nothing more. Firstly, unless someone explicitly tells the AI that it is a fundamental truth that nature is important to preserve, this can not happen. Secondly, the AI would also have to be incredibly gullible to just swallow such a claim. Thirdly, even if the AI does believe that, it will plainly say so to the people it is conversing with, in accordance with its goal to always tell the truth, thus warning us of this bug.
I agree with you that I don’t think a AGI would have the same problems humans have with the concept of truth. However, what you described is neither the issues philosophers raise nor the sorts of big-universe issues the AI might get stuck on.
But wouldn’t that actually support my approach? Assuming that there really is something important that all of humanity misses but the AI understands:
-If you hardcode the AI’s optimal goal based on human deliberations you are guaranteed to miss this important thing.
-If you use the method I suggested, the AI will, driven by the desire to speak the truth, try to explain the problem to the humans who will in turn tell the AI what they think of that.
I don’t see how that’s relivant to philosophical questions about truth. Did you mean to reply to my other comment?
I don’t think that is a good characterisation of the debate. It isn’t just about uncertainty.
That’s what you think. Some smart humans disagree with you. A supermsart AI might disagree with you and might be right. How can you second guess it? You cannot predict the behaviour of a supersmart AI on the basis that i t will agree with you, who are less smart.
Unless it figures it out.
Why would that require more gullibility than “species X is more important than all the others”? That doesn’t even look like a moral claim.
If it has “swallowed that* claim. You are assuming that the AI has a free choice about some goals and is just programmed with others.
This is the important part.
the “optimal goal” is not actually controlling the AI.
the “optimal goal” is merely the subject of a discussion.
what is controlling the AI is the desire the tell the truth to the humans it is talking to, nothing more.
The entire discussion is not supposed to unearth some kind of pure, inherently good, perfect optimal goal that transcends all reason and is true by virtue of existing or something.
The AI is supposed to take the human POV and think “if I were these humans, what would I want the AI’s goal to be”.
I didn’t mention this explicitly because I didn’t think it was necessary but the “optimal goal” is purely subjective from the POV of humanity and the AI is aware of this.
But if that is true, the AI will say so. What’s more, you kind of need the AI to refrain from acting on it, if it is a human-unfriendly objective moral truth. There are ethical puzzles where it is apparently right to lie or keep schtum, because of the consequences of telling the truth.