I like your arguments on AGI timelines, but the last section of your post feels like you are reflecting on something I would call “civilization improvement” rather than on a 20+ years plan for AGI alignment.
I am a bit confused by the way you are conflating “civilization improvement” with a strategy for alignment (when you discuss enhanced humans solving alignment, or discuss empathy in communicating a message “If you and people you know succeed at what you’re trying to do, everyone will die”). Yes, given longer timelines, civilization improvement can play a big role in reducing existential risk including AGI x-risk, but I would prefer to sell the broad merits of interventions on their own, rather than squeeze them into a strategy for alignment from today’s limited viewpoint. When making a multi-decade plan for civilization improvement, I think it is also important to consider the possibility of AGI-driven “civilization improvement”, i.e. interventions will not only influence AGI development, but they may also be critically influenced by it.
I like your arguments on AGI timelines, but the last section of your post feels like you are reflecting on something I would call “civilization improvement” rather than on a 20+ years plan for AGI alignment.
I am a bit confused by the way you are conflating “civilization improvement” with a strategy for alignment (when you discuss enhanced humans solving alignment, or discuss empathy in communicating a message “If you and people you know succeed at what you’re trying to do, everyone will die”). Yes, given longer timelines, civilization improvement can play a big role in reducing existential risk including AGI x-risk, but I would prefer to sell the broad merits of interventions on their own, rather than squeeze them into a strategy for alignment from today’s limited viewpoint. When making a multi-decade plan for civilization improvement, I think it is also important to consider the possibility of AGI-driven “civilization improvement”, i.e. interventions will not only influence AGI development, but they may also be critically influenced by it.
Finally, when considering strategy for alignment under longer timelines, people can have useful non-standard insights, see for example this discussion on AGI paradigms and this post on agent foundations research.