I am currently writing a paper on alternative utility functions for AIXI. Early steps in this direction have been taken for example here by @Anja and here by @AlexMennen—as far as I know the only serious published example is Laurent Orseau’s knowledge-seeking agent.
The reward-seeking formulation of AIXI is a product of its time and not a fundamental feature/constraint—any “continuous, l.s.c.” utility function is fine—the details will be formulated in my paper.
Actually choosing that utility function to be aligned with human values is ~equivalent to the alignment problem. AIXI does not solve it, but does “modularize” it to some extent.
I am currently writing a paper on alternative utility functions for AIXI. Early steps in this direction have been taken for example here by @Anja and here by @AlexMennen—as far as I know the only serious published example is Laurent Orseau’s knowledge-seeking agent.
The reward-seeking formulation of AIXI is a product of its time and not a fundamental feature/constraint—any “continuous, l.s.c.” utility function is fine—the details will be formulated in my paper.
Actually choosing that utility function to be aligned with human values is ~equivalent to the alignment problem. AIXI does not solve it, but does “modularize” it to some extent.