Thanks for the comment. I agree broadly of course, but the paper says more specific things. For example, agency needs to be prioritized, probably taken outside of standard optimization, otherwise decimating pressure is applied on other concepts including truth and other “human values”. The other part is a empirical one, also related to your concern, namely, human values are quite flexible and biology doesn’t create hard bounds / limits on depletion. If you couple that with ML/AI technologies that will predict what we will do next—then approaches that depend on human intent and values (broadly) are not as safe anymore.
Thanks for the comment. I agree broadly of course, but the paper says more specific things. For example, agency needs to be prioritized, probably taken outside of standard optimization, otherwise decimating pressure is applied on other concepts including truth and other “human values”. The other part is a empirical one, also related to your concern, namely, human values are quite flexible and biology doesn’t create hard bounds / limits on depletion. If you couple that with ML/AI technologies that will predict what we will do next—then approaches that depend on human intent and values (broadly) are not as safe anymore.