CEV is difficult because our values seem to be very complex.
Value is fragile, OK, but aren’t there easier utopias?
Building an AGI (let alone a super-intelligent AGI) that aimed for an ‘easier utopia’ would have to somehow convince/persuade/align the AI to give up a LOT of value. I don’t think it’s possible without solving alignment anyways. Essentially, it seems like we’d be trying to ‘convince’ the AGI to ‘not go to fast because that might be bad’. The problem is that we don’t know how to precisely what “bad” is anyways.
Many humans would be able to distinguish utopia from dystopia if they saw them
That’s very much not obvious. I don’t think that, e.g. humans from even 100 years ago teleported to today would be able to reliably distinguish the current world from a ‘dystopia’.
I haven’t myself noticed much agreement about the various utopias people have already described! That seems like pretty strong evidence that ‘utopia’ is in fact very hard to specify.
CEV is difficult because our values seem to be very complex.
Building an AGI (let alone a super-intelligent AGI) that aimed for an ‘easier utopia’ would have to somehow convince/persuade/align the AI to give up a LOT of value. I don’t think it’s possible without solving alignment anyways. Essentially, it seems like we’d be trying to ‘convince’ the AGI to ‘not go to fast because that might be bad’. The problem is that we don’t know how to precisely what “bad” is anyways.
That’s very much not obvious. I don’t think that, e.g. humans from even 100 years ago teleported to today would be able to reliably distinguish the current world from a ‘dystopia’.
I haven’t myself noticed much agreement about the various utopias people have already described! That seems like pretty strong evidence that ‘utopia’ is in fact very hard to specify.