Thanks for this post! It was especially valuable to see the link to Eliezer’s comments in “I expect that a number of AI safety researchers will deny that such a system will be sufficiently powerful.” It explains some aspects of Eliezer’s worldview that had previously confused me. Personally, I am at the opposite end of the spectrum relative to Eliezer—my intuition is that consequentialist planning and accurate world-modeling are fundamentally different tasks which are likely to stay that way. I’d argue that the history of statistics & machine learning is the history of gradual improvements to accurate world-modeling which basically haven’t shown any tendencies towards greater consequentialism. My default expectation is this trend will continue. The idea that you can’t have one without the other seems anthropomorphic to me.
Thanks for this post! It was especially valuable to see the link to Eliezer’s comments in “I expect that a number of AI safety researchers will deny that such a system will be sufficiently powerful.” It explains some aspects of Eliezer’s worldview that had previously confused me. Personally, I am at the opposite end of the spectrum relative to Eliezer—my intuition is that consequentialist planning and accurate world-modeling are fundamentally different tasks which are likely to stay that way. I’d argue that the history of statistics & machine learning is the history of gradual improvements to accurate world-modeling which basically haven’t shown any tendencies towards greater consequentialism. My default expectation is this trend will continue. The idea that you can’t have one without the other seems anthropomorphic to me.