I think the concern is that, if plans need some verification, it may be impossible to align smarter-than-human AGI. In order to verify those plans, we’d have to build one. If the plan doesn’t work (isn’t verified), that may be the end of us—no retries possible.
There are complex arguments on both sides, so I’m not arguing this is strictly true. I just wanted to clarify that that’s the concern and the point of asking people to solve it on the first try. I think ultimately this is partly but not 100% true of ASI alignment, and clarifying exactly how and to what degree we can verify plans empirically is critical to the project. Having a plan verified on weak, non-autonomous/self-aware/agentic systems may or may not generalize to that plan working on smarter systems with those properties. Some of the ways verification will or won’t generalize can probably be identified with careful analysis of how such systems will be functionally different.
I think the concern is that, if plans need some verification, it may be impossible to align smarter-than-human AGI. In order to verify those plans, we’d have to build one. If the plan doesn’t work (isn’t verified), that may be the end of us—no retries possible.
There are complex arguments on both sides, so I’m not arguing this is strictly true. I just wanted to clarify that that’s the concern and the point of asking people to solve it on the first try. I think ultimately this is partly but not 100% true of ASI alignment, and clarifying exactly how and to what degree we can verify plans empirically is critical to the project. Having a plan verified on weak, non-autonomous/self-aware/agentic systems may or may not generalize to that plan working on smarter systems with those properties. Some of the ways verification will or won’t generalize can probably be identified with careful analysis of how such systems will be functionally different.