So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?
Somehow, trying to figure out the policy an FAI or paternalist government should have by examining our addictive reactions strikes me as like doing transportation planning by looking at people’s reactions to red, yellow, and green lights. Not that people’s reactions to these things are irrelevant to traffic planning, but rather that figuring out where people are trying to go is even more important than figuring out their reactions to traffic signals.
The post talks about signals in the brain that motivate us powerfully. Almost certainly, evolution favored these signalling mechanisms because of where it tended to take us rather than how it tended to direct us there.
Maybe an FAI rather than figuring how to make us feel like we are satisfied or happy would instead figure out how to evoke these responses when we were doing something good for us and how to get negative responses when we were doing something bad for us. Maybe an FAI would rather re-wire us rather than wirehead us.
How do we stop our CEV from bringing us all beyond human, and should we even want to?
Somehow, trying to figure out the policy an FAI or paternalist government should have by examining our addictive reactions strikes me as like doing transportation planning by looking at people’s reactions to red, yellow, and green lights. Not that people’s reactions to these things are irrelevant to traffic planning, but rather that figuring out where people are trying to go is even more important than figuring out their reactions to traffic signals.
The post talks about signals in the brain that motivate us powerfully. Almost certainly, evolution favored these signalling mechanisms because of where it tended to take us rather than how it tended to direct us there.
Maybe an FAI rather than figuring how to make us feel like we are satisfied or happy would instead figure out how to evoke these responses when we were doing something good for us and how to get negative responses when we were doing something bad for us. Maybe an FAI would rather re-wire us rather than wirehead us.
How do we stop our CEV from bringing us all beyond human, and should we even want to?