Interesting—intuitively it seems more likely to me that a well constructed mind just doesn’t develop sophisticated demons. I think plenty of powerful optimization algorithms are not best understood as creating mesa-optimizers. The low-level systems of the human brain like the visual stream don’t seem to ever attempt takeover. I suppose one could make the claim that some psychological disorders arise from optimization “daemons” but this seems mostly seems like pure speculation and possibly an abuse of the terminology. For instance it seems silly to describe optical illusions as optimization daemons.
Interesting—intuitively it seems more likely to me that a well constructed mind just doesn’t develop sophisticated demons. I think plenty of powerful optimization algorithms are not best understood as creating mesa-optimizers. The low-level systems of the human brain like the visual stream don’t seem to ever attempt takeover. I suppose one could make the claim that some psychological disorders arise from optimization “daemons” but this seems mostly seems like pure speculation and possibly an abuse of the terminology. For instance it seems silly to describe optical illusions as optimization daemons.