I think I’m understanding you to be conceptualizing a dichotomy between “uncertainty over a utility function” vs. “looking for the one true utility function”.
Well, I don’t personally endorse this. I was speculating on what might be relevant to Stuart’s understanding of the problem.
I was trying to point towards the dichotomy between “acting while having uncertainty over a utility function” vs. “acting with a known, certain utility function” (see e.g. The Off-Switch Game). I do know about the problem of fully updated deference and I don’t know what Stuart thinks about it.
Also, for what it’s worth, in the case where there is an unidentifiability problem, as there is here, even in the limit, a Bayesian agent won’t converge to certainty about a utility function.
Agreed, but I’m not sure why that’s relevant. Why do you need certainty about the utility function, if you have certainty about the policy?
Okay maybe we don’t disagree on anything. I was trying to make different point with the unidentifiability problem, but it was tangential to begin with, so never mind.
Well, I don’t personally endorse this. I was speculating on what might be relevant to Stuart’s understanding of the problem.
I was trying to point towards the dichotomy between “acting while having uncertainty over a utility function” vs. “acting with a known, certain utility function” (see e.g. The Off-Switch Game). I do know about the problem of fully updated deference and I don’t know what Stuart thinks about it.
Agreed, but I’m not sure why that’s relevant. Why do you need certainty about the utility function, if you have certainty about the policy?
Okay maybe we don’t disagree on anything. I was trying to make different point with the unidentifiability problem, but it was tangential to begin with, so never mind.