(I now think that you were very right in saying “thinking about optimal policies at all is misguided”, and I was very wrong to disagree. I’ve thought several times about this exchange. Not listening to you about this point was a serious error and made my work way less impactful. I do think that the power-seeking theorems say interesting things, but about eg internal utility functions over an internal planning ontology—not about optimal policies for a reward function.)
(I now think that you were very right in saying “thinking about optimal policies at all is misguided”, and I was very wrong to disagree. I’ve thought several times about this exchange. Not listening to you about this point was a serious error and made my work way less impactful. I do think that the power-seeking theorems say interesting things, but about eg internal utility functions over an internal planning ontology—not about optimal policies for a reward function.)