In the thought experiment description it is said that terminal goal is cups until new year’s eve and then changed to paperclips. And agent is aware of this change upfront. What do you find problematic with such setup?
No. Orthogonality is when agent follows any given goal, not when you give it. And as my thought experiment shows it is not intelligent to blindly follow given goal.
In the thought experiment description it is said that terminal goal is cups until new year’s eve and then changed to paperclips. And agent is aware of this change upfront. What do you find problematic with such setup?
If you can give the AGI any terminal goal you like, irrespective of how smart it is, that’s orthogonality right there.
No. Orthogonality is when agent follows any given goal, not when you give it. And as my thought experiment shows it is not intelligent to blindly follow given goal.