“good” is determined by a currently opaque but nonetheless finite computation
With a finite definition, but not at all finite or even knowable consequences (they are knowably good, but what they are exactly, one can’t know).
there’s the additional claim that humanity has enough in common that this answer shouldn’t vary between people any significant amount
It’s going to vary a very significant amount, just a lot less than the distance from any other preference we might happen to construct, and as such, for example, creating a FAI modeled on any single person is hugely preferable for other people to letting an arbitrary AGI to develop, even if this AGI was extensively debugged and trained, and looks to possess all the right qualities.
Well, OK, let’s suppose* I agree with that. Could you elaborate on what that means in the context of the post? (Or link to somewhere where you did, if so.)
(*: Even after re-reading the AID post linked by orthonormal, I’m not sure what you mean by “knowably good” above, but I think that answering to the paragraph above would be more helpful than an abstract discussion.)
Whether something is good is also a factual question.
Care to elaborate?
The parent is assuming the naturalistic reduction of morality that EY argued for in the Metaethics Sequence, in which “good” is determined by a currently opaque but nonetheless finite computation (at least for a particular agent, but then there’s the additional claim that humanity has enough in common that this answer shouldn’t vary between people any significant amount).
With a finite definition, but not at all finite or even knowable consequences (they are knowably good, but what they are exactly, one can’t know).
It’s going to vary a very significant amount, just a lot less than the distance from any other preference we might happen to construct, and as such, for example, creating a FAI modeled on any single person is hugely preferable for other people to letting an arbitrary AGI to develop, even if this AGI was extensively debugged and trained, and looks to possess all the right qualities.
Well, OK, let’s suppose* I agree with that. Could you elaborate on what that means in the context of the post? (Or link to somewhere where you did, if so.)
(*: Even after re-reading the AID post linked by orthonormal, I’m not sure what you mean by “knowably good” above, but I think that answering to the paragraph above would be more helpful than an abstract discussion.)