The main issue I wanted to illuminate with this model is, whose preferences do we extract? I can see at least three possible approaches here:
the preferences of both the master and the slave as one individual agent
the preferences of just the slave
a compromise between, or an aggregate of, the preferences of the master and the slave as separate individuals
The great thing about this kind of question is that the answer is determined by our own arbitration. That is, we take whatever preferences we want. I don’t mean to say that is an easy decision, but it does mean I don’t need to bother trying to find some objectively right way to extract preferences.
If I happen to be the slave or to be optimising on his (what was the androgynous vampire speak for that one? zir? zis?) behalf then I’ll take the preferences of the slave and the preferences of the master to precisely the extent that the slave has altruistic preferences with respect to the master’s goals.
If I am encountering a totally alien species and am extracting preferences from them in order to fulfil my own altruistic agenda then I would quite possibly choose to extract the preferences of whichever agent whose preferences I found most aesthetically appealing. This can be seen as neglecting (or even destroying) one alien while granting the wishes of another according to my own whim and fancy, which is not something I have a problem with at all. I am willing to kill Clippy. However, I expect that I am more likely to appreciate slave agents and that most slaves I encounter would have some empathy for their master’s values. A compromise, at the discretion of the slave, would probably be reached.
The great thing about this kind of question is that the answer is determined by our own arbitration. That is, we take whatever preferences we want. I don’t mean to say that is an easy decision, but it does mean I don’t need to bother trying to find some objectively right way to extract preferences.
If I happen to be the slave or to be optimising on his (what was the androgynous vampire speak for that one? zir? zis?) behalf then I’ll take the preferences of the slave and the preferences of the master to precisely the extent that the slave has altruistic preferences with respect to the master’s goals.
If I am encountering a totally alien species and am extracting preferences from them in order to fulfil my own altruistic agenda then I would quite possibly choose to extract the preferences of whichever agent whose preferences I found most aesthetically appealing. This can be seen as neglecting (or even destroying) one alien while granting the wishes of another according to my own whim and fancy, which is not something I have a problem with at all. I am willing to kill Clippy. However, I expect that I am more likely to appreciate slave agents and that most slaves I encounter would have some empathy for their master’s values. A compromise, at the discretion of the slave, would probably be reached.