They would do well to taboo the vague “right” and maybe discuss what their respective extrapolated volitions are, whether “maximize pencils” is actually pencil-maximizer’s preference or an unfortunate result of a hardware glitch which on reflection should be corrected, what constitutes a “pencil” for the purpose of proper pencil-maximization, what is the proper attitude towards risk (dependence of utility on the number of pencils can take many shapes, even if it’s a given that it’s monotonic and depends on nothing else), etc.
They would do well to taboo the vague “right” and maybe discuss what their respective extrapolated volitions are, whether “maximize pencils” is actually pencil-maximizer’s preference or an unfortunate result of a hardware glitch which on reflection should be corrected, what constitutes a “pencil” for the purpose of proper pencil-maximization, what is the proper attitude towards risk (dependence of utility on the number of pencils can take many shapes, even if it’s a given that it’s monotonic and depends on nothing else), etc.
...great. Now I’m wondering whether a paperclip made of lead could be called a pencil and thus reconcile their optimization processes.
Excellent idea. This might well be the result of negotiations between two optimizers, preferable to fighting it out for control of the resources.
Two optimizers might negotiate based on the reality of their goals and power, instead of fighting over who owns the label “right”? What crazy talk!
And don’t tell them. I don’t want them joining forces.