(nods) Whether it’s possible or not is generally an open question. There’s a lot of skepticism about it (I’m fairly skeptical myself), but as with most technical questions, I’m generally content to have smart people research the question in more detail than I’m going to.
As to whether it’s desirable, though… well, sure, of course it depends on our goals. If all I want is (as you say) to create a new race to replace humanity, and I’m indifferent as to the values of that race, then of course there’s no reason for me to care about whether a self-improving AI I create will avoid value drift.
Personally, I’m more or less OK with something replacing humanity, but I’d prefer whatever that is to value certain things. For example, a commonly used trivial example around here of a hypothetical failure mode is a “paperclip maximizer”—an AI that only valued the existence of paperclips, and consequently reassembled all matter it can get its effectors on as paperclips. A paperclip maximizer with powerful enough effectors reassembles everything into paperclips.
I would prefer that not happen, from which I conclude that I’m not in fact indifferent as to the values of a sufficiently powerful AI… I desire that such a system preserve at least certain values. (It is difficult to state precisely what values those are, of course. Human values are complex.) I therefore prefer that it avoid value drift with respect to those values.
Well first, I was all for creating an AI to become the next stage. I was a very singularity-happy type of guy. I saw it as a way out of this world’s status quo—corruption, state of politics, etc… but the singularity would ultimately mean I and everybody else would cease to exist, at least in their true sense. You know, I have these romantic dreams, similar to Yudkowsky’s idea of dancing in an orbital night club around Saturn, and such. I don’t want to be fused in one, even though possibly amazing, matrix of intelligence, which I think is how the things will play out, eventually. Even though, I can’t imagine what it will be like and how it will pan out, as of now I just don’t cherish the idea much.
But yea, I could say that I am torn between moving on, advancing, and between more or less stagnating and in our human form.
But in answer to your question: if we were to creating an AI to replace us, I’d hate it to become paperclip maximizer. I don’t think it’s likely.
(nods) Whether it’s possible or not is generally an open question. There’s a lot of skepticism about it (I’m fairly skeptical myself), but as with most technical questions, I’m generally content to have smart people research the question in more detail than I’m going to.
As to whether it’s desirable, though… well, sure, of course it depends on our goals. If all I want is (as you say) to create a new race to replace humanity, and I’m indifferent as to the values of that race, then of course there’s no reason for me to care about whether a self-improving AI I create will avoid value drift.
Personally, I’m more or less OK with something replacing humanity, but I’d prefer whatever that is to value certain things. For example, a commonly used trivial example around here of a hypothetical failure mode is a “paperclip maximizer”—an AI that only valued the existence of paperclips, and consequently reassembled all matter it can get its effectors on as paperclips. A paperclip maximizer with powerful enough effectors reassembles everything into paperclips.
I would prefer that not happen, from which I conclude that I’m not in fact indifferent as to the values of a sufficiently powerful AI… I desire that such a system preserve at least certain values. (It is difficult to state precisely what values those are, of course. Human values are complex.) I therefore prefer that it avoid value drift with respect to those values.
How about you?
Well first, I was all for creating an AI to become the next stage. I was a very singularity-happy type of guy. I saw it as a way out of this world’s status quo—corruption, state of politics, etc… but the singularity would ultimately mean I and everybody else would cease to exist, at least in their true sense. You know, I have these romantic dreams, similar to Yudkowsky’s idea of dancing in an orbital night club around Saturn, and such. I don’t want to be fused in one, even though possibly amazing, matrix of intelligence, which I think is how the things will play out, eventually. Even though, I can’t imagine what it will be like and how it will pan out, as of now I just don’t cherish the idea much.
But yea, I could say that I am torn between moving on, advancing, and between more or less stagnating and in our human form.
But in answer to your question: if we were to creating an AI to replace us, I’d hate it to become paperclip maximizer. I don’t think it’s likely.