Yeah, I agree that the concepts of “goals”, “values”, “wanting”, etc. are useful concepts to have, and point to something real. For those concepts, it is true that the constant-twitching agent does not “want” to constantly twitch, nor does it have it as a “goal”. On the other hand, you can say that humans “want” to not suffer.
I’m not arguing that we should drop these concepts altogether. Separately from this post, I want to make the claim that we can try to build an AI system that does not have “goals”. A common counterargument is that due to coherence theorems any sufficiently advanced AI system will have “goals”. I’m rebutting that counterargument with this post.
(The next couple of posts in the sequence should address this a bit more, I think the sequence is going to resume very soon.)
Yeah, I agree that the concepts of “goals”, “values”, “wanting”, etc. are useful concepts to have, and point to something real. For those concepts, it is true that the constant-twitching agent does not “want” to constantly twitch, nor does it have it as a “goal”. On the other hand, you can say that humans “want” to not suffer.
I’m not arguing that we should drop these concepts altogether. Separately from this post, I want to make the claim that we can try to build an AI system that does not have “goals”. A common counterargument is that due to coherence theorems any sufficiently advanced AI system will have “goals”. I’m rebutting that counterargument with this post.
(The next couple of posts in the sequence should address this a bit more, I think the sequence is going to resume very soon.)