You can have goals that presuppose false beliefs. If I want to get to Heaven, and in fact there is no such place, my goal of getting to Heaven at least closely resembles an “incorrect goal”.
This raises an interesting question—if a Friendly AI or altruistic human wants to help me, and I want to go to Heaven, and the helper does not believe in Heaven, what should it do? So far as I can tell, it should help me get what I would want if I had what the helper considers to be true beliefs.
In a more mundane context, if I want to go north to get groceries, and the only grocery store is to the south, you aren’t helping me by driving me north. If getting groceries is a concern that overrides others, and you can’t communicate with me, you should drive me south to the grocery store even if I claim to want to go north. (If we can exchange evidence about the location of the grocery store, or if I value having true knowledge of what you find if you drive north, things are more complicated, but let’s assume for the purposes of argument that neither of those hold.)
This leads to the practical experiment of asking religious people what they would do differently if their God spoke to them and said “I quit. From now on, the materialists are right, your mind is in your brain, there is no soul, no afterlife, no reincarnation, no heaven, and no hell. If your brain is destroyed before you can copy the information out, you’re gone.” If a religious person says they’d do something ridiculous if God quit, we have a problem when implementing an FAI, since the FAI would either believe in Heaven or be inclined to help religious people do something ridiculous.
So far, I’ve had one Jehovah’s Witness say he couldn’t imagine imagine God quitting. Everyone else said they wouldn’t do much different if God quit.
If you do this experiment, please report back.
It would be a problem if there are many religious people who would apparently want to commit suicide if their God quit, the FAI convinces itself that there is no God, so it helpfully goes and kills them.
Erm, that’s supposing the religious person would actually want to suicide or do the ridiculous thing, rather than this itself being an expression of belief, affirmation, and argument of the religion. (I.e., as appeal to consequences, or saying negative things about the negation.)
Erm, that’s supposing the religious person would actually want to suicide or do the ridiculous thing, rather than this itself being an expression of belief, affirmation, and argument of the religion. (I.e., as appeal to consequences, or saying negative things about the negation.)
The most reasonable interpretation I can find for your statement is that you’re responding to this:
If a religious person says they’d do something ridiculous if God quit, we have a problem when implementing an FAI, since the FAI would either believe in Heaven or be inclined to help religious people do something ridiculous.
I agree, the goal would be to figure out what they would want if their beliefs were revised, and revising their circumstances so that God puts Himself out of the picture isn’t quite the same as that.
The experiment also has other weaknesses:
Ebay bidding shows that many people can’t correctly answer hypothetical questions. Perhaps people will accidentally give false information when I ask.
The question is obviously connected with a project related to athiesm. Perhaps some religious people will give false answers deliberately because they don’t want projects related to athiesm to succeed.
The relevant question is what the FAI thinks they would want if there were no God, not what they think they would want. A decent FAI would be able to do evolutionary psychology and many people can’t, especially religious people who don’t think evolution happened.
It’s not a real experiment. I’m not systematically finding these people, I’m just occasionally asking religious people what they think. There could easily be a selection effect since I’m not asking this question of random religious people.
You can have goals that presuppose false beliefs. If I want to get to Heaven, and in fact there is no such place, my goal of getting to Heaven at least closely resembles an “incorrect goal”.
This raises an interesting question—if a Friendly AI or altruistic human wants to help me, and I want to go to Heaven, and the helper does not believe in Heaven, what should it do? So far as I can tell, it should help me get what I would want if I had what the helper considers to be true beliefs.
In a more mundane context, if I want to go north to get groceries, and the only grocery store is to the south, you aren’t helping me by driving me north. If getting groceries is a concern that overrides others, and you can’t communicate with me, you should drive me south to the grocery store even if I claim to want to go north. (If we can exchange evidence about the location of the grocery store, or if I value having true knowledge of what you find if you drive north, things are more complicated, but let’s assume for the purposes of argument that neither of those hold.)
This leads to the practical experiment of asking religious people what they would do differently if their God spoke to them and said “I quit. From now on, the materialists are right, your mind is in your brain, there is no soul, no afterlife, no reincarnation, no heaven, and no hell. If your brain is destroyed before you can copy the information out, you’re gone.” If a religious person says they’d do something ridiculous if God quit, we have a problem when implementing an FAI, since the FAI would either believe in Heaven or be inclined to help religious people do something ridiculous.
So far, I’ve had one Jehovah’s Witness say he couldn’t imagine imagine God quitting. Everyone else said they wouldn’t do much different if God quit.
If you do this experiment, please report back.
It would be a problem if there are many religious people who would apparently want to commit suicide if their God quit, the FAI convinces itself that there is no God, so it helpfully goes and kills them.
Erm, that’s supposing the religious person would actually want to suicide or do the ridiculous thing, rather than this itself being an expression of belief, affirmation, and argument of the religion. (I.e., as appeal to consequences, or saying negative things about the negation.)
The most reasonable interpretation I can find for your statement is that you’re responding to this:
I agree, the goal would be to figure out what they would want if their beliefs were revised, and revising their circumstances so that God puts Himself out of the picture isn’t quite the same as that.
The experiment also has other weaknesses:
Ebay bidding shows that many people can’t correctly answer hypothetical questions. Perhaps people will accidentally give false information when I ask.
The question is obviously connected with a project related to athiesm. Perhaps some religious people will give false answers deliberately because they don’t want projects related to athiesm to succeed.
The relevant question is what the FAI thinks they would want if there were no God, not what they think they would want. A decent FAI would be able to do evolutionary psychology and many people can’t, especially religious people who don’t think evolution happened.
It’s not a real experiment. I’m not systematically finding these people, I’m just occasionally asking religious people what they think. There could easily be a selection effect since I’m not asking this question of random religious people.