The question of what BPA wants to do to Steve, seems to me to be far more important for Steve’s safety, than the question of what set of rules will constrain the actions of BPA.
BPA shouldn’t be allowed to want anything for Steve. There shouldn’t be a term in its world-model for Steve. This is the goal of cosmic blocking. The BPA can’t even know that Steve exists.
I think the difficult part is when BPA looks at Bob’s preferences (excluding, of course, references to most specific people) and sees preferences for inflicting harm on people-in-general that can be bent just enough to fit into the “not-torture” bucket, and so it synthetically generates some new people and starts inflicting some kind of marginal harm on them.
And I think that this will in fact be a binding constraint on utopia, because most humans will (given the resources) want to make a personal utopia full of other humans that forms a status hierarchy with them at the top. And ‘being forced to participate in a status hierarchy that you are not at the top of’ is a type of ‘generalized consensual harm’.
Even the good old Reedspacer’s Lower Bound fits this model. Reedspacer wants a volcano lair full of catgirls, but the catgirls are consensually participating in a universe that is not optimal for them because they are stuck in the harem of a loser nerd with no other males and no other purpose in life other than being a concubine to Reedspacer. Arguably, that is a form of consensual harm to the catgirls.
So I don’t think there is a neat boundary here. The neatest boundary is informed consent, perhaps backed up by some lower-level tests about what proportion of an entity’s existence is actually miserable.
If Reedspacer beats his catgirls, makes them feel sad all the time, that matters. But maybe if one of them feels a little bit sad for a short moment that is acceptable.
catgirls are consensually participating in a universe that is not optimal for them because they are stuck in the harem of a loser nerd with no other males and no other purpose in life other than being a concubine to Reedspacer
And, the problem with saying “OK let’s just ban the creation of catgirls” is that then maybe Reedspacer builds a volcano lair just for himself and plays video games in it, and the catgirls whose existence you prevented are going to scream bloody murder because you took away from them a very good existence that they would have enjoyed and also made Reedsapcer sad.
BPA shouldn’t be allowed to want anything for Steve. There shouldn’t be a term in its world-model for Steve. This is the goal of cosmic blocking. The BPA can’t even know that Steve exists.
I think the difficult part is when BPA looks at Bob’s preferences (excluding, of course, references to most specific people) and sees preferences for inflicting harm on people-in-general that can be bent just enough to fit into the “not-torture” bucket, and so it synthetically generates some new people and starts inflicting some kind of marginal harm on them.
And I think that this will in fact be a binding constraint on utopia, because most humans will (given the resources) want to make a personal utopia full of other humans that forms a status hierarchy with them at the top. And ‘being forced to participate in a status hierarchy that you are not at the top of’ is a type of ‘generalized consensual harm’.
Even the good old Reedspacer’s Lower Bound fits this model. Reedspacer wants a volcano lair full of catgirls, but the catgirls are consensually participating in a universe that is not optimal for them because they are stuck in the harem of a loser nerd with no other males and no other purpose in life other than being a concubine to Reedspacer. Arguably, that is a form of consensual harm to the catgirls.
So I don’t think there is a neat boundary here. The neatest boundary is informed consent, perhaps backed up by some lower-level tests about what proportion of an entity’s existence is actually miserable.
If Reedspacer beats his catgirls, makes them feel sad all the time, that matters. But maybe if one of them feels a little bit sad for a short moment that is acceptable.
And, the problem with saying “OK let’s just ban the creation of catgirls” is that then maybe Reedspacer builds a volcano lair just for himself and plays video games in it, and the catgirls whose existence you prevented are going to scream bloody murder because you took away from them a very good existence that they would have enjoyed and also made Reedsapcer sad.