Using a model like that (“revealed preferences” about energy) seems like it would fall into the trap mentioned here, in that it needs a very good model of human irrationality, or it would start concluding absurdities (eg that we don’t want to be rich since we don’t exploit the many arbitrage and profit opportunities that the alien sees all around us). Do you have a way around that issue?
If you point an arbitrage opportunity out to someone, and they take it, then ‘they didn’t notice it’ seems reasonable.
However, figuring out what someone has and has not noticed does sound difficult—perhaps in an intractable fashion. Even if someone saw an opportunity long ago, perhaps they took a better one and forgot it, or lacked the necessary knowledge to figure out the opportunity.
I think that ants like sugar. However, if I spill some sugar on the countertop, I’m not going to be shocked when every ant in a ten mile radius doesn’t immediately start walking towards the sugar. It’s reasonable to expect a model of an agent’s behavior to include a model of that agent’s model of its environment.
And so the assumptions pile up :-) We have to distinguish not knowing, from not caring, from not being able to plan the whole way through, from arious biases in the model… I agree that it’s necessary, but that doesn’t make it feasible.
I think the way around that issue is to bite the bullet—those things belong in a proper theory of mind. Most people want to be conformist (or at least to maintain a pleasant-to-them self-image) more than they want to be rich. That seems like a truth (lowercase t—it’s culture-sensitive, not necessarily universal) that should be modeled more than a trap to be avoided.
But people are still leaving a lot of efficient, low effort, conformity on the table—a superintelligent conformist human could be so much better at being (or appearing) conformist, than we can ever manage.
Using a model like that (“revealed preferences” about energy) seems like it would fall into the trap mentioned here, in that it needs a very good model of human irrationality, or it would start concluding absurdities (eg that we don’t want to be rich since we don’t exploit the many arbitrage and profit opportunities that the alien sees all around us). Do you have a way around that issue?
If you point an arbitrage opportunity out to someone, and they take it, then ‘they didn’t notice it’ seems reasonable.
However, figuring out what someone has and has not noticed does sound difficult—perhaps in an intractable fashion. Even if someone saw an opportunity long ago, perhaps they took a better one and forgot it, or lacked the necessary knowledge to figure out the opportunity.
Or maybe the opportunity was only available for someone who could do (super)intelligent follow up to the initial opportunity.
I think that ants like sugar. However, if I spill some sugar on the countertop, I’m not going to be shocked when every ant in a ten mile radius doesn’t immediately start walking towards the sugar. It’s reasonable to expect a model of an agent’s behavior to include a model of that agent’s model of its environment.
And so the assumptions pile up :-) We have to distinguish not knowing, from not caring, from not being able to plan the whole way through, from arious biases in the model… I agree that it’s necessary, but that doesn’t make it feasible.
I think the way around that issue is to bite the bullet—those things belong in a proper theory of mind. Most people want to be conformist (or at least to maintain a pleasant-to-them self-image) more than they want to be rich. That seems like a truth (lowercase t—it’s culture-sensitive, not necessarily universal) that should be modeled more than a trap to be avoided.
But people are still leaving a lot of efficient, low effort, conformity on the table—a superintelligent conformist human could be so much better at being (or appearing) conformist, than we can ever manage.
So a model that says people are ‘super intelligent’ would be badly wrong.