A climate change defector also doesn’t get to “align” the entire future with the defector’s chosen value system.
I do understand that the problem with AGI is exactly that you don’t know how to align anything with anything at all, and if you know you can’t, then obviously you shouldn’t try. That would be stupid.
The problem is that there’ll be an arms race to become able to do so… and a huge amount of pressure to deploy any solution you think you have as soon as you possibly can. That kind of pressure leads to motivated cognition and institutional failure, so you become “sure” that something will work when it won’t. It also leads to building up all the prerequisite capabilities for a “pivotal act”, so that you can put it into practice immediately when (you think) you have an alignment solution.
Replying to myself to clarify this:
I do understand that the problem with AGI is exactly that you don’t know how to align anything with anything at all, and if you know you can’t, then obviously you shouldn’t try. That would be stupid.
The problem is that there’ll be an arms race to become able to do so… and a huge amount of pressure to deploy any solution you think you have as soon as you possibly can. That kind of pressure leads to motivated cognition and institutional failure, so you become “sure” that something will work when it won’t. It also leads to building up all the prerequisite capabilities for a “pivotal act”, so that you can put it into practice immediately when (you think) you have an alignment solution.
… which basically sets up a bunch of time bombs.
I agree with that.