Eliezer: What’s an example of a novel prediction made by the notion of probability?
Richard: Most applications of the central limit theorem.
Eliezer: Then I should get to claim every kind of optimization algorithm which used expected utility, as a successful advance prediction of expected utility?
...
Richard: These seem better than nothing, but still fairly unsatisfying, insofar as I think they are related to more shallow properties of the theory.
This exchange makes me wonder whether Richard would accept the successes of reinforcement learning as “predictions” of the kind he is looking for? Because RL is essentially the straightforward engineering implementation of “expected utility theory”.
I’m still trying to understand the scope of expected utility theory, so examples like this are very helpful! I’d need to think much more about it before I had a strong opinion about how much they support Eliezer’s applications of the theory, though.
I think I might actually be happy to take e.g. the Bellman equation, a fundamental equation in RL, as a basic expression of consistent utilities and thereby claim value iteration, Q-learning, and deep Q-learning all as predictions/applications of utility theory. Certainly this seems fair if you claim applications of the central limit theorem for probability theory.
To expand a bit, the Bellman equation only expresses a certain consistency condition among utilities. The expected utility of this state must equal its immediate utility plus the best expected utility among each possible next state I may choose. Start with some random utilities assigned to states, gradually update them to be consistent, and you get optimal behavior. Huge parts of RL are centered around this equation, including e.g. DeepMind using DQNs to crack Atari games.
I understand Eliezer’s frustration in answering this question. The response to “What predictions/applications does utility theory have?” in regards to intelligent behavior is, essentially, “Everything and nothing.”
Comment after reading section 5.3:
This exchange makes me wonder whether Richard would accept the successes of reinforcement learning as “predictions” of the kind he is looking for? Because RL is essentially the straightforward engineering implementation of “expected utility theory”.
I’m still trying to understand the scope of expected utility theory, so examples like this are very helpful! I’d need to think much more about it before I had a strong opinion about how much they support Eliezer’s applications of the theory, though.
I think I might actually be happy to take e.g. the Bellman equation, a fundamental equation in RL, as a basic expression of consistent utilities and thereby claim value iteration, Q-learning, and deep Q-learning all as predictions/applications of utility theory. Certainly this seems fair if you claim applications of the central limit theorem for probability theory.
To expand a bit, the Bellman equation only expresses a certain consistency condition among utilities. The expected utility of this state must equal its immediate utility plus the best expected utility among each possible next state I may choose. Start with some random utilities assigned to states, gradually update them to be consistent, and you get optimal behavior. Huge parts of RL are centered around this equation, including e.g. DeepMind using DQNs to crack Atari games.
I understand Eliezer’s frustration in answering this question. The response to “What predictions/applications does utility theory have?” in regards to intelligent behavior is, essentially, “Everything and nothing.”