Another way of looking at it that (I think!) makes it obvious: it’s advice for AI, in the AI-in-the-box experiment.
I imagine it’s what Eliezer means when he says he won the experiment “the hard way”. He just kept poking for a psychological mechanism to exploit until he found one that happened to work with the given person. And the intermediate, failed attempts, probably helped him model the person and narrow down on whatever actually might work.
Another way of looking at it that (I think!) makes it obvious: it’s advice for AI, in the AI-in-the-box experiment.
I imagine it’s what Eliezer means when he says he won the experiment “the hard way”. He just kept poking for a psychological mechanism to exploit until he found one that happened to work with the given person. And the intermediate, failed attempts, probably helped him model the person and narrow down on whatever actually might work.