There’s something a little rediculous about claiming that every member of a group prefers A to B, but that the group in aggregate does not prefer A to B.
The situation analogous to Simpson’s paradox can only occur if for some reason we care about some people’s opinion more than others in some situations (this is analogous to the situation in Simpson’s paradox where we have more data points in some parts of the table than others. It is a necessary condition for the paradox to occur.)
For example: Suppose Alice (female) values a cure for prostate cancer at 10 utils, and a cure for breast cancer at 15 utils. Bob (male) values a cure for prostate cancer at 100 utils, and a cure for breast cancer at 150 utils. Suppose that because prostate cancer largely affects men and breast cancer largely affects women we value Alice’s opinion twice as much about breast cancer and Bob’s opinion twice as much about prostate cancer. Then in the aggregate curing prostate cancer is 210 utils and curing breast cancer 180 utils, a preference reversal compared to either of Alice or Bob.
This is essentially just an example of Harsanyi’s Theorem in action. And I think it makes a compelling demonstration of why you should not program an AI in that fashion.
To get the effect that we need an optimiser that cares about some people’s opinion more about some things but then for some other things cares about someone else’s opinion. If we just have a utility monster who the optimiser always values more than others we can’t get the effect. The important thing is that it sometimes cares about one person and sometimes cares about someone else.
I don’t see how it’s like Simpson’s paradox, actually. You want to go to Good Hospital instead of Bad Hospital even if more patients who go to Good Hospital die because they get almost the hard cases. Aggregating only hides the information needed to make a properly informed choice. Here, aggregating doesn’t hide any information.
But there are a bunch of other ways things like that can happen.
This very morning I did a nonlinear curvefit on a bunch of repeats of an experiment. One of the parameters that came out had values in the range −1 to +1. I combined the data sets directly and that parameter for the combined set came out around 5.
In a way, this analogy may be even more directly applicable than Simpson’s paradox. Even if A and B are complete specifications (unlike that parameter, which was one of several), the interpersonal reactions to other people can do some very nonlinear things to interpretations of A and B.
That would look a bit like Simpson’s paradox actually.
The situation analogous to Simpson’s paradox can only occur if for some reason we care about some people’s opinion more than others in some situations (this is analogous to the situation in Simpson’s paradox where we have more data points in some parts of the table than others. It is a necessary condition for the paradox to occur.)
For example: Suppose Alice (female) values a cure for prostate cancer at 10 utils, and a cure for breast cancer at 15 utils. Bob (male) values a cure for prostate cancer at 100 utils, and a cure for breast cancer at 150 utils. Suppose that because prostate cancer largely affects men and breast cancer largely affects women we value Alice’s opinion twice as much about breast cancer and Bob’s opinion twice as much about prostate cancer. Then in the aggregate curing prostate cancer is 210 utils and curing breast cancer 180 utils, a preference reversal compared to either of Alice or Bob.
This is essentially just an example of Harsanyi’s Theorem in action. And I think it makes a compelling demonstration of why you should not program an AI in that fashion.
Isn’t that the description of an utility maximizer (or optimizer) taking into account the preferences of an utility monster?
To get the effect that we need an optimiser that cares about some people’s opinion more about some things but then for some other things cares about someone else’s opinion. If we just have a utility monster who the optimiser always values more than others we can’t get the effect. The important thing is that it sometimes cares about one person and sometimes cares about someone else.
I don’t see how it’s like Simpson’s paradox, actually. You want to go to Good Hospital instead of Bad Hospital even if more patients who go to Good Hospital die because they get almost the hard cases. Aggregating only hides the information needed to make a properly informed choice. Here, aggregating doesn’t hide any information.
But there are a bunch of other ways things like that can happen.
This very morning I did a nonlinear curvefit on a bunch of repeats of an experiment. One of the parameters that came out had values in the range −1 to +1. I combined the data sets directly and that parameter for the combined set came out around 5.
In a way, this analogy may be even more directly applicable than Simpson’s paradox. Even if A and B are complete specifications (unlike that parameter, which was one of several), the interpersonal reactions to other people can do some very nonlinear things to interpretations of A and B.