Well, the code for the seed should be pretty simple, at least. But I don’t see how that defeats the purpose of my statement; it may be that short of enlisting a superintelligence to help, all current attempts to approximate and extrapolate human preferences in a consistent fashion (e.g. explicit ethical or political theories) might be too crude to have any chance of success (by the standard of actual human preferences) in novel scenarios. I don’t believe this will be the case, but it’s a possibility worth keeping an eye on.
Well, the code for the seed should be pretty simple, at least. But I don’t see how that defeats the purpose of my statement; it may be that short of enlisting a superintelligence to help, all current attempts to approximate and extrapolate human preferences in a consistent fashion (e.g. explicit ethical or political theories) might be too crude to have any chance of success (by the standard of actual human preferences) in novel scenarios. I don’t believe this will be the case, but it’s a possibility worth keeping an eye on.