I’m sure anyone who’s really thought about the problem of extrapolating volition has been rather frustrated with this at some point. Not only is it uselessly difficult to figure out what we want, it’s incredibly difficult to figure out how to design something that figures out what we want, and this chain of dependencies ends up requiring vast knowledge of mathematics, psychology, philosophy, cosmology, and other fields that any sane agent just shouldn’t need to have mastered in order to introspect on the question of “In the end, what am I trying to do?”.
I’m sure anyone who’s really thought about the problem of extrapolating volition has been rather frustrated with this at some point. Not only is it uselessly difficult to figure out what we want, it’s incredibly difficult to figure out how to design something that figures out what we want, and this chain of dependencies ends up requiring vast knowledge of mathematics, psychology, philosophy, cosmology, and other fields that any sane agent just shouldn’t need to have mastered in order to introspect on the question of “In the end, what am I trying to do?”.