we agree now nearly in all points grin, except for that part of the AIs not “wanting” to change their goals, simply because through meditation (in the Buddhist tradition for instance) I know that you can “see through” goals and not be enslaved to them anymore (and that is accessible to humans, so why shouldn’t it be accessible to introspecting AIs?).
That line of thought is also strongly related to the concept of avidya, which ascribes “desires” and “wanting” to not having completely grasped certain truths about reality. I think these truths would also be accessible to sentient AIs (we live in the same universe after all), and thus they would also be able to come to certain insights annulling “programmed” drives. (As indeed human sages do.)
But I think what you said about “the scope of the paper” is relevant here. When I was pointed to the paper my expectations where raised that it would solve some of the fundamental problems of “wanting” and “desire” (in a psychological sense), but that is clearly not the focus of the paper, so maybe I was simply disappointed because I expected something else.
But, of course, it is always important when drawing conclusions that one remembers one’s premises. Often, when conclusions seem exciting or “important”, one forgets the limits of one’s premises and applies the reasoning to contexts outside the scope of the original limitations.
I accept Omohundro’s conclusions for certain kinds of non-sentient intelligent systems working with utility functions seeking to maximize some kind of economic (resource-constrained) goal. But I think that the results are not as general as a first reading might lead to believe.
Tim,
we agree now nearly in all points grin, except for that part of the AIs not “wanting” to change their goals, simply because through meditation (in the Buddhist tradition for instance) I know that you can “see through” goals and not be enslaved to them anymore (and that is accessible to humans, so why shouldn’t it be accessible to introspecting AIs?).
That line of thought is also strongly related to the concept of avidya, which ascribes “desires” and “wanting” to not having completely grasped certain truths about reality. I think these truths would also be accessible to sentient AIs (we live in the same universe after all), and thus they would also be able to come to certain insights annulling “programmed” drives. (As indeed human sages do.)
But I think what you said about “the scope of the paper” is relevant here. When I was pointed to the paper my expectations where raised that it would solve some of the fundamental problems of “wanting” and “desire” (in a psychological sense), but that is clearly not the focus of the paper, so maybe I was simply disappointed because I expected something else.
But, of course, it is always important when drawing conclusions that one remembers one’s premises. Often, when conclusions seem exciting or “important”, one forgets the limits of one’s premises and applies the reasoning to contexts outside the scope of the original limitations.
I accept Omohundro’s conclusions for certain kinds of non-sentient intelligent systems working with utility functions seeking to maximize some kind of economic (resource-constrained) goal. But I think that the results are not as general as a first reading might lead to believe.