suppose I’m a general trying to maximize my side’s chance of winning a war. Can I evaluate the probability that we win, given all of the information available to me? No—fully accounting for every little piece of info I have is way beyond my computational capabilities. Even reasoning through an entire end-to-end plan for winning takes far more effort than I usually make for day-to-day decisions. Yet I can say that some actions are likely to increase our chances of victory, and I can prioritize actions which are more likely to increase our chances of victory by a larger amount.
So, when and why are we able to get away with doing that?
AFAICT, the formalisms of agents that I’m aware of (Bayesian inference, AIXI etc.) set things up by supposing logical omniscience and that the true world generating our hypotheses is in the set of hypotheses and from there you can show that the agent will maximise expected utilty, or not get dutch booked or whatever. But humans, and ML algorithms for that matter, don’t do that, we’re able to get “good enough” results even when we know our models are wrong and don’t capture a good deal of the underlying process generating our observations. Furthermore, it seems that empirically, the more expressive the model class we use, and the more compute thrown at the problem, the better these bounded inference algorithms work. I haven’t found a good explanation of why this is the case beyond hand wavy “we approach logical omniscience as compute goes to infinity and our hypothesis space grows to encompass all computable hypotheses, so eventually our approximation should work like the ideal Bayesian one”.
I think in part we can get away with it because it’s possible to optimize for things that are only usually decidable.
Take winning the war for example. There may be no computer program that could look at any state of the world and tell you who won the war—there are lots of weird edge cases that could cause a Turing machine to not return a decision. But if we expect to be able to tell who won the war with very high probability (or have a model that we think matches who wins the war with high probability), then we can just sort of ignore the weird edge cases and model failures when calculating an expected utility.
So, when and why are we able to get away with doing that?
AFAICT, the formalisms of agents that I’m aware of (Bayesian inference, AIXI etc.) set things up by supposing logical omniscience and that the true world generating our hypotheses is in the set of hypotheses and from there you can show that the agent will maximise expected utilty, or not get dutch booked or whatever. But humans, and ML algorithms for that matter, don’t do that, we’re able to get “good enough” results even when we know our models are wrong and don’t capture a good deal of the underlying process generating our observations. Furthermore, it seems that empirically, the more expressive the model class we use, and the more compute thrown at the problem, the better these bounded inference algorithms work. I haven’t found a good explanation of why this is the case beyond hand wavy “we approach logical omniscience as compute goes to infinity and our hypothesis space grows to encompass all computable hypotheses, so eventually our approximation should work like the ideal Bayesian one”.
I think in part we can get away with it because it’s possible to optimize for things that are only usually decidable.
Take winning the war for example. There may be no computer program that could look at any state of the world and tell you who won the war—there are lots of weird edge cases that could cause a Turing machine to not return a decision. But if we expect to be able to tell who won the war with very high probability (or have a model that we think matches who wins the war with high probability), then we can just sort of ignore the weird edge cases and model failures when calculating an expected utility.
Perhaps...
As the approximation gets closer to the ideal, the results do as well. (The Less Wrong quote seems relevant.)