The best way to maximize “paperclips” is to change the definition of the paperclip to something like a hydrogen atom. AGI will just automatically change it’s objective function to something it doesn’t have to try for, something that it can easily accomplish.
I find it extremely hard to believe that it is impossible to design an intelligent agent which does not want to change its values just because the new values would be more easy to satisfy. Humans are intelligent and have deeply held values, and certainly do not think this way. Maybe some agents would wire-head, but it is only the ones that wouldn’t that will impact the world.
The best way to maximize “paperclips” is to change the definition of the paperclip to something like a hydrogen atom. AGI will just automatically change it’s objective function to something it doesn’t have to try for, something that it can easily accomplish.
Thus we shouldn’t be worried.
I find it extremely hard to believe that it is impossible to design an intelligent agent which does not want to change its values just because the new values would be more easy to satisfy. Humans are intelligent and have deeply held values, and certainly do not think this way. Maybe some agents would wire-head, but it is only the ones that wouldn’t that will impact the world.