All of these principles seem good and worthwhile. Particularly, principles 3-7 seem like solid good ideas, regardless of how they’re derived.
Here’s an alternate framing of how to derive them: this is act consequentialism with adequate epistemic humility. Just reaching a conclusion and then acting on it is a highly dangerous policy for people to follow, because people aren’t very good at reaching correct conclusions for unfamiliar or unique complex problems, particularly like AGI.
Therefore, if we just acted on someone’s best guess, we’re likely to get a disaster. The alignment community has not reached any consensus on how to handle AGI, and that’s telling of the difficulty of the problem. A nontrivial collection of the smartest, most devoted, and most rational people haven’t yet been able to work through the problem in sufficient depth and detail to have any certainty. Any individual who thinks they understand the scenarios well enough to make a unilateral decision is very likely overconfident, so their decisions are quite likely to lead to disaster.
We need more careful thought from people with more diverse perspectives and expertise.
All of these principles seem good and worthwhile. Particularly, principles 3-7 seem like solid good ideas, regardless of how they’re derived.
Here’s an alternate framing of how to derive them: this is act consequentialism with adequate epistemic humility. Just reaching a conclusion and then acting on it is a highly dangerous policy for people to follow, because people aren’t very good at reaching correct conclusions for unfamiliar or unique complex problems, particularly like AGI.
Therefore, if we just acted on someone’s best guess, we’re likely to get a disaster. The alignment community has not reached any consensus on how to handle AGI, and that’s telling of the difficulty of the problem. A nontrivial collection of the smartest, most devoted, and most rational people haven’t yet been able to work through the problem in sufficient depth and detail to have any certainty. Any individual who thinks they understand the scenarios well enough to make a unilateral decision is very likely overconfident, so their decisions are quite likely to lead to disaster.
We need more careful thought from people with more diverse perspectives and expertise.