the main arguments for the programmers including all of [current?] humanity in the CEV “extrapolation base” […] apply symmetrically to AIs-we’re-sharing-the-world-with at the time
I think timeless values might possibly help resolve this; if some {AIs that are around at the time} are moral patients, then sure, just like other moral patients around they should get a fair share of the future.
If an AI grabs more resources than is fair, you do the exact same thing as if a human grabs more resources than is fair: satisfy the values of moral patients (including ones who are no longer around) not weighed by how much leverage they current have over the future, but how much leverage they would have over the future if things had gone more fairly/if abuse/powergrab/etc wasn’t the kind of thing that gets your more control of the future.
“Sorry clippy, we do want you to get some paperclips, we just don’t want you to get as many paperclips as you could if you could murder/brainhack/etc all humans, because that doesn’t seem to be a very fair way to allocate the future.” — and in the same breath, “Sorry Putin, we do want you to get some of whatever-intrinsic-values-you’re-trying-to-satisfy, we just don’t want you to get as much as ruthlessly ruling Russia can get you, because that doesn’t seem to be a very fair way to allocate the future.”
And this can apply regardless of how much of clippy already exists by the time you’re doing CEV.
I think timeless values might possibly help resolve this; if some {AIs that are around at the time} are moral patients, then sure, just like other moral patients around they should get a fair share of the future.
If an AI grabs more resources than is fair, you do the exact same thing as if a human grabs more resources than is fair: satisfy the values of moral patients (including ones who are no longer around) not weighed by how much leverage they current have over the future, but how much leverage they would have over the future if things had gone more fairly/if abuse/powergrab/etc wasn’t the kind of thing that gets your more control of the future.
“Sorry clippy, we do want you to get some paperclips, we just don’t want you to get as many paperclips as you could if you could murder/brainhack/etc all humans, because that doesn’t seem to be a very fair way to allocate the future.” — and in the same breath, “Sorry Putin, we do want you to get some of whatever-intrinsic-values-you’re-trying-to-satisfy, we just don’t want you to get as much as ruthlessly ruling Russia can get you, because that doesn’t seem to be a very fair way to allocate the future.”
And this can apply regardless of how much of clippy already exists by the time you’re doing CEV.