There’s a recent paper by a human philosopher, which proposes a counterpart of Pascal’s wager, but regarding the possible existence of objective good and evil, rather than the possible existence of God. (The uncertain starting point is called metaethical agnosticism. Ethical agnosticism is where you don’t know what’s right and what’s wrong; metaethical agnosticism is uncertainty about the nature of right and wrong.)
That paper’s proposal is basically that you shouldn’t do things which might be bad, just in case they actually are bad.
Eliezer’s thought process in 2000 starts in the same place, but goes a little differently, because he has the cognitive optimism that a superintelligence might be able to dispel the uncertainty, and actually know for sure, whether or not there are objective rights and wrongs.
So this is undoubtedly a possible thought process. But I’d say it’s far from an inevitable one. It requires a “mind” that has the concept of goal, that is capable of conceiving of objectively valuable or compelling goals, that is capable of being motivationally affected by that possibility—and which doesn’t think up some other clever argument that is even more compelling than this one.
There’s a recent paper by a human philosopher, which proposes a counterpart of Pascal’s wager, but regarding the possible existence of objective good and evil, rather than the possible existence of God. (The uncertain starting point is called metaethical agnosticism. Ethical agnosticism is where you don’t know what’s right and what’s wrong; metaethical agnosticism is uncertainty about the nature of right and wrong.)
That paper’s proposal is basically that you shouldn’t do things which might be bad, just in case they actually are bad.
Eliezer’s thought process in 2000 starts in the same place, but goes a little differently, because he has the cognitive optimism that a superintelligence might be able to dispel the uncertainty, and actually know for sure, whether or not there are objective rights and wrongs.
So this is undoubtedly a possible thought process. But I’d say it’s far from an inevitable one. It requires a “mind” that has the concept of goal, that is capable of conceiving of objectively valuable or compelling goals, that is capable of being motivationally affected by that possibility—and which doesn’t think up some other clever argument that is even more compelling than this one.