The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don’t have a good mental model of the differences between a world where optimization daemons can arise and a world where they can’t (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I’ve seen is the evolution of humans, but I find it concerning that I can’t make any other predictions about the world based on the idea of ODs.
What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?
[removed discussion of religion to make the question more clear/straightforward]
[Question] Evidence other than evolution for optimization daemons?
The idea of consequentialist agents arising in sufficiently strong optimizing systems intuitively makes sense to me. However, I don’t have a good mental model of the differences between a world where optimization daemons can arise and a world where they can’t (i.e. what facts about the world provide Bayesian evidence for the concept of ODs). The only example I’ve seen is the evolution of humans, but I find it concerning that I can’t make any other predictions about the world based on the idea of ODs.
What other Bayesian evidence/potential intuition pumps exist for the possibility of optimization daemons arising?
[removed discussion of religion to make the question more clear/straightforward]