Two examples of MIRI talking about orthogonality, instrumental convergence, etc.: “Five Theses, Two Lemmas, and a Couple of Strategic Implications” (2013) and “So Far: Unfriendly AI Edition” (2016). The latter is closer to how I’d start a discussion with a random computer scientist today, if they thought AGI alignment isn’t important to work on and I wanted to figure out where the disagreement lies.
I think “Five Theses...” is basically a list of ‘here are the five things Ray Kurzweil is wrong about’. A lot of people interested in AGI early on held Kurzweilian views: humans will self-improve to keep up with AI; sufficiently smart AGI will do good stuff by default; value isn’t fragile; etc. ‘AGI built with no concern for safety or alignment’ is modeled like a person from a foreign culture, or like a sci-fi alien race with bizarre but beautiful cosmopolitan values — not like the moral equivalent of a paperclip maximizer.
I think orthogonality, instrumental convergence, etc. are also the key premises Eliezer needed to learn. Eliezer initially dismissed the importance of alignment research because he thought moral truths were inherently motivating, so any AGI smart enough to learn what’s moral would end up promoting good outcomes. Visualizing human values as just one possible goal in a vast space of possibilities, noticing that there’s no architecture-invariant causal mechanism forcing modeled goals to leak out into held goals, and thinking about obviously bad goals like “just keep making paperclips” helps undo that specific confusion.
I agree that a fair number of people in the early days over-updated based on “other people are wrong” logic.
Two examples of MIRI talking about orthogonality, instrumental convergence, etc.: “Five Theses, Two Lemmas, and a Couple of Strategic Implications” (2013) and “So Far: Unfriendly AI Edition” (2016). The latter is closer to how I’d start a discussion with a random computer scientist today, if they thought AGI alignment isn’t important to work on and I wanted to figure out where the disagreement lies.
I think “Five Theses...” is basically a list of ‘here are the five things Ray Kurzweil is wrong about’. A lot of people interested in AGI early on held Kurzweilian views: humans will self-improve to keep up with AI; sufficiently smart AGI will do good stuff by default; value isn’t fragile; etc. ‘AGI built with no concern for safety or alignment’ is modeled like a person from a foreign culture, or like a sci-fi alien race with bizarre but beautiful cosmopolitan values — not like the moral equivalent of a paperclip maximizer.
I think orthogonality, instrumental convergence, etc. are also the key premises Eliezer needed to learn. Eliezer initially dismissed the importance of alignment research because he thought moral truths were inherently motivating, so any AGI smart enough to learn what’s moral would end up promoting good outcomes. Visualizing human values as just one possible goal in a vast space of possibilities, noticing that there’s no architecture-invariant causal mechanism forcing modeled goals to leak out into held goals, and thinking about obviously bad goals like “just keep making paperclips” helps undo that specific confusion.
I agree that a fair number of people in the early days over-updated based on “other people are wrong” logic.