even without being confident about each others’ exact belief/value split.
This seems consistent with the claim:
A person’s beliefs and behavior are intimately related the way a quantum wave packet’s position is related to its momentum. You cannot observe both of them independently and at the same time.
You did word things rather clearly.
Though I imagine some might object to ‘relatively simple utility functions(/rich environment)’ - i.e., people don’t have simple utility functions.
In principle, I was imagining talking about two AIs.
In practice, there are quite a few preferences I feel confident a random person would have, even if the details differ between people and even though there’s no canonical way to rectify our preferences into a utility function. I believe that the argument carries through practically with a decent amount of noise; I certainly treat it as some evidence for X when a thinker I respect believes X.
This seems consistent with the claim:
You did word things rather clearly.
Though I imagine some might object to ‘relatively simple utility functions(/rich environment)’ - i.e., people don’t have simple utility functions.
In principle, I was imagining talking about two AIs.
In practice, there are quite a few preferences I feel confident a random person would have, even if the details differ between people and even though there’s no canonical way to rectify our preferences into a utility function. I believe that the argument carries through practically with a decent amount of noise; I certainly treat it as some evidence for X when a thinker I respect believes X.