The design of social choice AI faces three sets of decisions: standing, concerning whose ethics views are included; measurement, concerning how their views are identified; and aggregation, concerning how individual views are combined to a single view that will guide AI behavior. [] Each set of decisions poses difficult ethical dilemmas with major consequences for AI behavior, with some decision options yielding pathological or even catastrophic results.
I think it’s slightly lacking in sophistication about aggregation of numerical preferences, and in how revealed preferences indicate that we don’t actually have incommensurable or infinitely-strong preferences, but is overall pretty great.
On the subject of the problem, I don’t think we should program in values that are ad-hoc on the object level (what values to use—trying to program this by hand is destined for failure), or even the meta level (whose values to use). But I do think it’s okay to use an ad-hoc process to try to learn the answers to the meta-level questions. After all, what’s the worst that could happen? (irony). Of course, the ability to do this assumes the solution of other, probably more difficult philosophical/AI problems, like how to refer to peoples’ values in the first place.
Neat paper about the difficulties of specifying satisfactory values for a strong AI. h/t Kaj Sotala.
I think it’s slightly lacking in sophistication about aggregation of numerical preferences, and in how revealed preferences indicate that we don’t actually have incommensurable or infinitely-strong preferences, but is overall pretty great.
On the subject of the problem, I don’t think we should program in values that are ad-hoc on the object level (what values to use—trying to program this by hand is destined for failure), or even the meta level (whose values to use). But I do think it’s okay to use an ad-hoc process to try to learn the answers to the meta-level questions. After all, what’s the worst that could happen? (irony). Of course, the ability to do this assumes the solution of other, probably more difficult philosophical/AI problems, like how to refer to peoples’ values in the first place.
Note that these three things (standing, measurement, and aggregation) are unsolved for human moral decisionmaking as well.