Asking people to state what their revealed preferences are is a fool’s game. Brains are built to deceive themselves about their preferences; even if someone was trying to be totally honest with you, they would still mislead you. If I wanted to figure out the preferences of an alien race, I wouldn’t try to initiate political or philosophical conversations. I would try to trade with them.
If I could only observe the aliens, I would try to figure out how they decide where to spend their energy. Whether the aliens hunt prey, grow crops, run solar farms, or maintain a Dyson swarm, they must gather energy in some fashion. Energy is and always will be a scarce resource, so building predictive models of alien energy allocation policy will reveal information about their preferences.
Looking at humans from an alien perspective:
Notice that humans allocate energy by swapping currency for a proportional amount of energy
Notice that there is a system by which humans swap currency for a wide variety of other things
Using a model like that (“revealed preferences” about energy) seems like it would fall into the trap mentioned here, in that it needs a very good model of human irrationality, or it would start concluding absurdities (eg that we don’t want to be rich since we don’t exploit the many arbitrage and profit opportunities that the alien sees all around us). Do you have a way around that issue?
If you point an arbitrage opportunity out to someone, and they take it, then ‘they didn’t notice it’ seems reasonable.
However, figuring out what someone has and has not noticed does sound difficult—perhaps in an intractable fashion. Even if someone saw an opportunity long ago, perhaps they took a better one and forgot it, or lacked the necessary knowledge to figure out the opportunity.
I think that ants like sugar. However, if I spill some sugar on the countertop, I’m not going to be shocked when every ant in a ten mile radius doesn’t immediately start walking towards the sugar. It’s reasonable to expect a model of an agent’s behavior to include a model of that agent’s model of its environment.
And so the assumptions pile up :-) We have to distinguish not knowing, from not caring, from not being able to plan the whole way through, from arious biases in the model… I agree that it’s necessary, but that doesn’t make it feasible.
I think the way around that issue is to bite the bullet—those things belong in a proper theory of mind. Most people want to be conformist (or at least to maintain a pleasant-to-them self-image) more than they want to be rich. That seems like a truth (lowercase t—it’s culture-sensitive, not necessarily universal) that should be modeled more than a trap to be avoided.
But people are still leaving a lot of efficient, low effort, conformity on the table—a superintelligent conformist human could be so much better at being (or appearing) conformist, than we can ever manage.
Asking people to state what their revealed preferences are is a fool’s game. Brains are built to deceive themselves about their preferences; even if someone was trying to be totally honest with you, they would still mislead you. If I wanted to figure out the preferences of an alien race, I wouldn’t try to initiate political or philosophical conversations. I would try to trade with them.
If I could only observe the aliens, I would try to figure out how they decide where to spend their energy. Whether the aliens hunt prey, grow crops, run solar farms, or maintain a Dyson swarm, they must gather energy in some fashion. Energy is and always will be a scarce resource, so building predictive models of alien energy allocation policy will reveal information about their preferences.
Looking at humans from an alien perspective:
Notice that humans allocate energy by swapping currency for a proportional amount of energy
Notice that there is a system by which humans swap currency for a wide variety of other things
Build a causal model of this system
In doing so, model the valuation structures of the human cortex
Using a model like that (“revealed preferences” about energy) seems like it would fall into the trap mentioned here, in that it needs a very good model of human irrationality, or it would start concluding absurdities (eg that we don’t want to be rich since we don’t exploit the many arbitrage and profit opportunities that the alien sees all around us). Do you have a way around that issue?
If you point an arbitrage opportunity out to someone, and they take it, then ‘they didn’t notice it’ seems reasonable.
However, figuring out what someone has and has not noticed does sound difficult—perhaps in an intractable fashion. Even if someone saw an opportunity long ago, perhaps they took a better one and forgot it, or lacked the necessary knowledge to figure out the opportunity.
Or maybe the opportunity was only available for someone who could do (super)intelligent follow up to the initial opportunity.
I think that ants like sugar. However, if I spill some sugar on the countertop, I’m not going to be shocked when every ant in a ten mile radius doesn’t immediately start walking towards the sugar. It’s reasonable to expect a model of an agent’s behavior to include a model of that agent’s model of its environment.
And so the assumptions pile up :-) We have to distinguish not knowing, from not caring, from not being able to plan the whole way through, from arious biases in the model… I agree that it’s necessary, but that doesn’t make it feasible.
I think the way around that issue is to bite the bullet—those things belong in a proper theory of mind. Most people want to be conformist (or at least to maintain a pleasant-to-them self-image) more than they want to be rich. That seems like a truth (lowercase t—it’s culture-sensitive, not necessarily universal) that should be modeled more than a trap to be avoided.
But people are still leaving a lot of efficient, low effort, conformity on the table—a superintelligent conformist human could be so much better at being (or appearing) conformist, than we can ever manage.
So a model that says people are ‘super intelligent’ would be badly wrong.