Thanks for the really insightful answer! I think I’m pretty much convinced on points 1, 2, 5, and 7, mostly agree with you on 6 and 8, and still don’t understand the sheer hopelessness of people who strongly believe 9. Assumptions 3, and 4, however, I’m not sure I fully follow, as it doesn’t seem like a slam dunk that the orthogonality thesis is true, as far as I can tell. I’d expect there to be basins of attraction towards some basic values, or convergence, sort of like carcinisation.
Carcinisation is an excellent metaphor for convergent instrumental values, i.e. values that are desired for ends other than themselves, and which can serve a wide variety of ends, and thus might be expected to occur in a wide variety of minds. In fact, there’s been some research on exactly that by Steve Omohundro, who defined the Omohundro Goals (well worth looking up). These are things like survival and preservation of your other goals, as it’s usually much easier to accomplish a thing if you remain alive to work on it, and continue to value doing so. However, orthogonality doesn’t apply to instrumental goals, which can do a good or bad job of serving as an effective path to other goals, and thus experience selection and carcinisation. Rather, it applies to terminal goals, those things we want purely for their own sake. It’s impossible to judge terminal goals as good or bad (except insofar as they accord or conflict with our own terminal goals, and that’s not a standard an AI automatically has to care about), as they are themselves the standard by which everything else is judged. The researcher Rob Miles has an excellent YouTube video about this you might enjoy entitled Intelligence and Stupidity: the Orthogonality Thesis, which goes into more depth. (Sorry for the lack of direct links; I’m sending this from my phone immediately before going to bed.)
Thanks for the really insightful answer! I think I’m pretty much convinced on points 1, 2, 5, and 7, mostly agree with you on 6 and 8, and still don’t understand the sheer hopelessness of people who strongly believe 9. Assumptions 3, and 4, however, I’m not sure I fully follow, as it doesn’t seem like a slam dunk that the orthogonality thesis is true, as far as I can tell. I’d expect there to be basins of attraction towards some basic values, or convergence, sort of like carcinisation.
Carcinisation is an excellent metaphor for convergent instrumental values, i.e. values that are desired for ends other than themselves, and which can serve a wide variety of ends, and thus might be expected to occur in a wide variety of minds. In fact, there’s been some research on exactly that by Steve Omohundro, who defined the Omohundro Goals (well worth looking up). These are things like survival and preservation of your other goals, as it’s usually much easier to accomplish a thing if you remain alive to work on it, and continue to value doing so. However, orthogonality doesn’t apply to instrumental goals, which can do a good or bad job of serving as an effective path to other goals, and thus experience selection and carcinisation. Rather, it applies to terminal goals, those things we want purely for their own sake. It’s impossible to judge terminal goals as good or bad (except insofar as they accord or conflict with our own terminal goals, and that’s not a standard an AI automatically has to care about), as they are themselves the standard by which everything else is judged. The researcher Rob Miles has an excellent YouTube video about this you might enjoy entitled Intelligence and Stupidity: the Orthogonality Thesis, which goes into more depth. (Sorry for the lack of direct links; I’m sending this from my phone immediately before going to bed.)
• Intelligence And Stupidity by Rob Miles on YouTube
• Orthogonality Thesis on Arbital.com