I realize you haven’t been online for a few months, but yes, I do.
Humanity’s desires are not currently consistent. An FAI couldn’t satisfy them all because some of them contradict each other, like Fred’s and Steve’s in your example. There may not even be a way of averaging them out fairly or meaningfully. Either Steve lives or dies: there’s no average or middle ground and Fred is just out of luck.
However, it might be the case that human beings are similar enough that if you extrapolate everything all humans want, you get something consistent. Extrapolation is a tool to resolve inconsistencies and please both Fred and Steve.
I realize you haven’t been online for a few months, but yes, I do.
Humanity’s desires are not currently consistent. An FAI couldn’t satisfy them all because some of them contradict each other, like Fred’s and Steve’s in your example. There may not even be a way of averaging them out fairly or meaningfully. Either Steve lives or dies: there’s no average or middle ground and Fred is just out of luck.
However, it might be the case that human beings are similar enough that if you extrapolate everything all humans want, you get something consistent. Extrapolation is a tool to resolve inconsistencies and please both Fred and Steve.