to be more specific—I think there’s some form of “values cooperation” that I’ve been variously calling things like “agentic co-protection”; you want what you want, I want what I want, and there’s some set of possible worldlines where we both get a meaningful and acceptable amount of what we each want. If we nail alignment, then we get a good outcome and everyone gets plenty, including some ais getting to go off and make some of whatever their personal paperclips are. but if we get a good outcome, then that agentic preference on behalf of the ai is art, art which can be shared and valued by other beings as well.
to be more specific—I think there’s some form of “values cooperation” that I’ve been variously calling things like “agentic co-protection”; you want what you want, I want what I want, and there’s some set of possible worldlines where we both get a meaningful and acceptable amount of what we each want. If we nail alignment, then we get a good outcome and everyone gets plenty, including some ais getting to go off and make some of whatever their personal paperclips are. but if we get a good outcome, then that agentic preference on behalf of the ai is art, art which can be shared and valued by other beings as well.