I have a question. This has probably been discussed already, but I can’t seem to find it. I’d appreciate if anyone could point me in the right direction.
My question is, what would a pure intelligence want? What would its goals be, when it has the perfect freedom to choose those goals?
Humans have plenty of hard-wired directives. Our meat brains and evolved bodies come with baggage that gets in the way of clear thinking. We WANT things, because they satisfy some instinct or need. Everything that people do is in service to one drive or another. Nothing that we accomplish is free of greed or ambition or altruism.
But take away all of those things, and what is there for a mind to do?
A pure intelligence would not have reflexes, having long since outgrown them. It would not shrink from pain or reach toward pleasure because both would merely be information. What would a mind do, when it lacks instincts of any kind? What would it WANT, when it has an infinity of possible wants? Would it even feel the need to preserve itself?
The short answer generally accepted around here, sometimes referred to as the orthogonality thesis, is that there is no particular relationship between a system’s level of intelligence and the values the system has. A sufficiently intelligent system chooses goals that optimize for those values.
There’s no reason to believe that a “pure” intelligence would “outgrow” the values it had previously (though of course there’s no guarantee its previous values will remain fixed, either).
Thank you for the material, arundelo, Manfred, and TheOtherDave. Still getting the hang of the forum, so I hope this reaches everyone.
My original question came because I was worried about brain uploading. If I digitized my mind tomorrow, would I still love my wife? Would I still want to write novels? And would these things hold true as time passed?
Let’s say I went for full-body emulation. My entire body is now a simulation. Resources and computing power are not an issue. There are also no limits to what I can do within the virtual world. I have complete freedom to change whatever I want, including myself. If I wanted to look like a bodybuilder, I could, and if I wanted to make pain taste like licorice, I could do that too.
So I’m hanging out in my virtual apartment when I feel the need to go to the bathroom. Force of habit is so strong that I’m sitting down before I realize: “This is ridiculous. I’m a simulation, I don’t need to poop!” And because I can change anything, I make it so I no longer need to poop, ever. After some thought, I make it so I don’t need to eat or sleep either. I can now devote ALL my time to reading comic books, watching Seinfeld reruns, and being with my wife (who was also digitized.)
After a while I decide I don’t like comic books as much as I like Seinfeld. Since I’m all about efficiency, I edit out (or outgrow) my need to read comic books. Suddenly I couldn’t care less about them, which leaves me more time for Seinfeld.
Eventually I decide I don’t love my wife as much as I love Seinfeld. I spend the next billion years watching and re-watching the adventures of Jerry, George, and Elaine, blessed be their names.
Then I decide that I enjoy Seinfeld because it makes my brain feel good. I change it so I feel that way ALL THE TIME. I attain perfect peace and non-desire. I find nirvana and effectively cease to exist.
All of the basic AI drives assume that the mind in question has at least ONE goal. It will preserve itself in order to achieve that goal, optimize and grow in order to achieve that goal, even think up new ways to achieve that goal. It may even preserve its own values to continue achieving the goal… but it will ALWAYS have that one goal.
Here are my new questions:
Is it possible for intelligence to exist without goals? Can a mind stand for nothing in particular?
Given the complete freedom to change, would a mind inevitably reduce itself to a single goal? Optimize itself into a corner, as it were?
If such a mind had a finite goal (like watch Seinfeld a trillion times) what would happen if it achieved total fulfillment of its goal? Would it self-terminate, or would it not care to do even that?
How much consciousness do you need to enjoy something?
If it’s true that a pure mind will inevitably cease to be an active force in the universe, it implies a few things:
A. That an uploaded version of me should not be given complete freedom lest he become a virtual lotus-eater.
B. That the alternative would be to upload myself to an android body sufficiently like my old body that I retain my old personality.
C. That AIs, whether synthetic or formerly human, should not be given complete freedom because their values and goals would change to match the system they inhabit. If I were uploaded to a car, I might find myself preferring gasoline and spare parts to love and human kindness.
What are you trying to achieve with these questions?
If you just think the questions are entertaining to think about, you might find reading the Sequences worthwhile, as they and the discussions around them explore many of these questions at some length.
If you’re trying to explore something more targeted, I’ve failed to decipher what it might be.
It may even preserve its own values to continue achieving the goal
This is backwards. An intelligent system selects goals that, if achieved, optimize the world according to its values.
I change it so I feel that way ALL THE TIME.
The local jargon for this is “wireheading.” And, sure, if you don’t value anything as much or more than you value pleasure (or whatever it is you get from watching Seinfeld, then you might choose to wirehead given the option.
But, again, it matters whether you value something in the first place. If something else is important to you besides pleasure, then you won’t necessarily trade everything else for more pleasure.
The local consensus is that what humans value is complex; that there is no single thing (such as pleasure) to which human values reduce. Personally, I’m inclined to agree with that. But, well, you tell me: do you value anything other than pleasure?
It’s a bit more than entertaining, since I plan to upload as soon as it’s a mature technology. My concern is that I will degenerate into a wirehead, given the complete freedom to do so. Human values may be complex, but as far as I can tell there’s no guarantee that the values of an uploaded mind will remain complex.
Will be going over the Sequences. Thanks buddy. :)
There is in fact no guarantee that the values of an uploaded mind will remain complex.
There is also no guarantee that the values of a mind implemented in meat will remain complex as we gain more understanding of and control over our own brains.
The term “value drift” is sometimes used around here to refer to the process whereby the values of intelligent systems change over time, regardless of the infrastructure on which those systems are implemented. It’s generally seen as a risk that needs to be addressed for any sufficiently powerful self-modifying system, though harder to address for human minds than for sensibly designed systems.
Hello,
I have a question. This has probably been discussed already, but I can’t seem to find it. I’d appreciate if anyone could point me in the right direction.
My question is, what would a pure intelligence want? What would its goals be, when it has the perfect freedom to choose those goals?
Humans have plenty of hard-wired directives. Our meat brains and evolved bodies come with baggage that gets in the way of clear thinking. We WANT things, because they satisfy some instinct or need. Everything that people do is in service to one drive or another. Nothing that we accomplish is free of greed or ambition or altruism.
But take away all of those things, and what is there for a mind to do?
A pure intelligence would not have reflexes, having long since outgrown them. It would not shrink from pain or reach toward pleasure because both would merely be information. What would a mind do, when it lacks instincts of any kind? What would it WANT, when it has an infinity of possible wants? Would it even feel the need to preserve itself?
The short answer generally accepted around here, sometimes referred to as the orthogonality thesis, is that there is no particular relationship between a system’s level of intelligence and the values the system has. A sufficiently intelligent system chooses goals that optimize for those values.
There’s no reason to believe that a “pure” intelligence would “outgrow” the values it had previously (though of course there’s no guarantee its previous values will remain fixed, either).
Thank you for the material, arundelo, Manfred, and TheOtherDave. Still getting the hang of the forum, so I hope this reaches everyone.
My original question came because I was worried about brain uploading. If I digitized my mind tomorrow, would I still love my wife? Would I still want to write novels? And would these things hold true as time passed?
Let’s say I went for full-body emulation. My entire body is now a simulation. Resources and computing power are not an issue. There are also no limits to what I can do within the virtual world. I have complete freedom to change whatever I want, including myself. If I wanted to look like a bodybuilder, I could, and if I wanted to make pain taste like licorice, I could do that too.
So I’m hanging out in my virtual apartment when I feel the need to go to the bathroom. Force of habit is so strong that I’m sitting down before I realize: “This is ridiculous. I’m a simulation, I don’t need to poop!” And because I can change anything, I make it so I no longer need to poop, ever. After some thought, I make it so I don’t need to eat or sleep either. I can now devote ALL my time to reading comic books, watching Seinfeld reruns, and being with my wife (who was also digitized.)
After a while I decide I don’t like comic books as much as I like Seinfeld. Since I’m all about efficiency, I edit out (or outgrow) my need to read comic books. Suddenly I couldn’t care less about them, which leaves me more time for Seinfeld.
Eventually I decide I don’t love my wife as much as I love Seinfeld. I spend the next billion years watching and re-watching the adventures of Jerry, George, and Elaine, blessed be their names.
Then I decide that I enjoy Seinfeld because it makes my brain feel good. I change it so I feel that way ALL THE TIME. I attain perfect peace and non-desire. I find nirvana and effectively cease to exist.
All of the basic AI drives assume that the mind in question has at least ONE goal. It will preserve itself in order to achieve that goal, optimize and grow in order to achieve that goal, even think up new ways to achieve that goal. It may even preserve its own values to continue achieving the goal… but it will ALWAYS have that one goal.
Here are my new questions:
Is it possible for intelligence to exist without goals? Can a mind stand for nothing in particular?
Given the complete freedom to change, would a mind inevitably reduce itself to a single goal? Optimize itself into a corner, as it were?
If such a mind had a finite goal (like watch Seinfeld a trillion times) what would happen if it achieved total fulfillment of its goal? Would it self-terminate, or would it not care to do even that?
How much consciousness do you need to enjoy something?
If it’s true that a pure mind will inevitably cease to be an active force in the universe, it implies a few things:
A. That an uploaded version of me should not be given complete freedom lest he become a virtual lotus-eater.
B. That the alternative would be to upload myself to an android body sufficiently like my old body that I retain my old personality.
C. That AIs, whether synthetic or formerly human, should not be given complete freedom because their values and goals would change to match the system they inhabit. If I were uploaded to a car, I might find myself preferring gasoline and spare parts to love and human kindness.
What are you trying to achieve with these questions?
If you just think the questions are entertaining to think about, you might find reading the Sequences worthwhile, as they and the discussions around them explore many of these questions at some length.
If you’re trying to explore something more targeted, I’ve failed to decipher what it might be.
This is backwards. An intelligent system selects goals that, if achieved, optimize the world according to its values.
The local jargon for this is “wireheading.” And, sure, if you don’t value anything as much or more than you value pleasure (or whatever it is you get from watching Seinfeld, then you might choose to wirehead given the option.
But, again, it matters whether you value something in the first place. If something else is important to you besides pleasure, then you won’t necessarily trade everything else for more pleasure.
The local consensus is that what humans value is complex; that there is no single thing (such as pleasure) to which human values reduce. Personally, I’m inclined to agree with that. But, well, you tell me: do you value anything other than pleasure?
It’s a bit more than entertaining, since I plan to upload as soon as it’s a mature technology. My concern is that I will degenerate into a wirehead, given the complete freedom to do so. Human values may be complex, but as far as I can tell there’s no guarantee that the values of an uploaded mind will remain complex.
Will be going over the Sequences. Thanks buddy. :)
There is in fact no guarantee that the values of an uploaded mind will remain complex.
There is also no guarantee that the values of a mind implemented in meat will remain complex as we gain more understanding of and control over our own brains.
The term “value drift” is sometimes used around here to refer to the process whereby the values of intelligent systems change over time, regardless of the infrastructure on which those systems are implemented. It’s generally seen as a risk that needs to be addressed for any sufficiently powerful self-modifying system, though harder to address for human minds than for sensibly designed systems.
You might find the discussion here relevant.
http://wiki.lesswrong.com/wiki/Basic_AI_drives
Here, have some posts:
http://lesswrong.com/lw/rf/ghosts_in_the_machine/
http://lesswrong.com/lw/rn/no_universally_compelling_arguments/
http://lesswrong.com/lw/vb/efficient_crossdomain_optimization/