(Chiming in late here, sorry!) I think this is a totally valid concern, but I think it’s generally helpful to discuss technical and political challenges separately. I think pessimistic folks often say things like “We have no idea how to align an AI,” and I see this post as a partial counterpoint to that.
(Chiming in late here, sorry!) I think this is a totally valid concern, but I think it’s generally helpful to discuss technical and political challenges separately. I think pessimistic folks often say things like “We have no idea how to align an AI,” and I see this post as a partial counterpoint to that.
In addition to a small alignment tax (as you mention), a couple other ways I could see the political side going well would be (a) an AI project using a few-month lead to do huge amounts of further helpful work (https://www.lesswrong.com/posts/jwhcXmigv2LTrbBiB/success-without-dignity-a-nearcasting-story-of-avoiding#The_deployment_problem); (b) a standards-and-monitoring regime blocking less cautious training and deployment.