I’m not familiar with the ASI alignment literature, but presumably he is. I googled “would have wanted” + “alignment” on this site, and this didn’t seem to turn up much. If this has already been proposed, please let me know in the comments.
related: https://www.lesswrong.com/tag/outer-alignment, https://www.lesswrong.com/tag/coherent-extrapolated-volition, https://carado.moe/surprise-you-want.html
related: https://www.lesswrong.com/tag/outer-alignment, https://www.lesswrong.com/tag/coherent-extrapolated-volition, https://carado.moe/surprise-you-want.html