Hmm. Thanks for clearly separating those two points, and I agree that I was mixing them together. I suspect that they _are_ mixed together, because reality will eventually win out (if the AI isn’t optimizing the universe as well as possible, it’ll be replaced by one that does), but I don’t think I can make that argument clearly (because I get tangled up in corrigibility and control—who is able to make the decision to alter the utility function or replace the AI? I hope it’s not Moloch, but fear that it is.)
Hmm. Thanks for clearly separating those two points, and I agree that I was mixing them together. I suspect that they _are_ mixed together, because reality will eventually win out (if the AI isn’t optimizing the universe as well as possible, it’ll be replaced by one that does), but I don’t think I can make that argument clearly (because I get tangled up in corrigibility and control—who is able to make the decision to alter the utility function or replace the AI? I hope it’s not Moloch, but fear that it is.)