Hmm. Do we (the creators of the AI) think this is correct? That is, does it match OUR desires for the future?
Code filters off desires, unless the AI has been programmed to Do What We Mean. “The genie knows but doesn’t care,” and so on.
I think you’re making two distinct points. First, that a competent AGI that is nevertheless shackled with hyperbolic discounting will probably remove the discounting. Second, that a hyperbolic AI would not effectively match our own goals. I agree with the second, but that has no bearing on the first. My original comment was exclusively talking about the first claim.
Hmm. Thanks for clearly separating those two points, and I agree that I was mixing them together. I suspect that they _are_ mixed together, because reality will eventually win out (if the AI isn’t optimizing the universe as well as possible, it’ll be replaced by one that does), but I don’t think I can make that argument clearly (because I get tangled up in corrigibility and control—who is able to make the decision to alter the utility function or replace the AI? I hope it’s not Moloch, but fear that it is.)
Code filters off desires, unless the AI has been programmed to Do What We Mean. “The genie knows but doesn’t care,” and so on.
I think you’re making two distinct points. First, that a competent AGI that is nevertheless shackled with hyperbolic discounting will probably remove the discounting. Second, that a hyperbolic AI would not effectively match our own goals. I agree with the second, but that has no bearing on the first. My original comment was exclusively talking about the first claim.
Hmm. Thanks for clearly separating those two points, and I agree that I was mixing them together. I suspect that they _are_ mixed together, because reality will eventually win out (if the AI isn’t optimizing the universe as well as possible, it’ll be replaced by one that does), but I don’t think I can make that argument clearly (because I get tangled up in corrigibility and control—who is able to make the decision to alter the utility function or replace the AI? I hope it’s not Moloch, but fear that it is.)