You are a soldier, planning to attack an enemy position. You have a foolproof strategy for taking positions held by ordinary enemies, which will get you killed if you use it on positions held by snipers. You can call in an airstrike that will reliably devastate the enemy position, but you’d prefer not to if you don’t have to. Using an airstrike will overextend the army’s air support; there’s a Q% chance this will lead to a fellow soldier’s death.
X% of enemies are snipers.
Snipers hit their target Y% of the time.
Ordinary enemies hit their target Z% of the time.
[Poke a helmet on a stick out of the trench]
[Call in an airstrike] [Attack!]
I made a miniature interactive scenario to help people develop and/or evaluate their understanding of Bayes’ Theorem. I intend to use it to teach a class a week from now, and I’m hoping to get feedback before then. Help me (and possibly yourself) out?
This is fun! I agree with others that the added question of Is it better to kill an ally with probability P or die myself with probability Q makes the whole problem slightly less straightforward. In an extreme case, if I just want to survive, I might as well always call an airstrike, missing the whole teaching of this problem. Maybe you could tweak the scenario a bit to remove this dilemma?
The jankiness here is deliberate (which doesn’t preclude it from being a mistake). My class on Bayesianism is intended to also be a class on the limitations thereof: that it fails when you haven’t mapped out the entire sample space, that it doesn’t apply ‘cleanly’ to any but the most idealised use cases, and that once you’ve calculated everything out you’ll still be left with irreducible judgement calls.
I think this scenario is really valuable and gives you a nice intuitive feel for Bayesian updates.
I wish there were more “Bayesian toy models” like this.
(I have the “show P(sniper)” feature always enabled to “train” my neural network on this data, rather than trying to calculate this in my head)
That’s among the intended use cases; I’m pleased to see someone thought of it independently.
It was fun to actually get out Bayes rule in a bayesian reasoning challenge, and gratifying to see that I got the same number as reveal P(Sniper). When I clicked “stick out helmet” a second time, I had already clicked “reveal P(sniper)” to check my work from the single shot calculation, and it live updated to the two shot calculation- spoilers?
I feel like I rolled “hard mode” the first time I loaded the page: 50% are snipers, 60% sniper hit rate, 40% regular hit rate (so no difference on priors and not much to tell the difference between them), and then they only deigned to take two shots at my helmet (one hit, one miss) before catching on to the ruse.
I guess “sometimes the world doesn’t provide convenient data” is a valid part of the lesson. But if I were tweaking the variables I might patch in a higher minimum number of shots against the helmet (I did see it become willing to take many more on subsequent re-rolls)
Might also be more satisfying if the air strike option revealed whether it really was a sniper—even when I’m confident they were, I find myself repeating the scenario and choosing “attack” to check.
Do you recall what the probability of a fellow soldier dying due to calling in air strike was? In my tests so far it was never greater than 20%, so you should definitely call in an air strike in your scenario since the false positive outcome is not as costly as the false negative.
Edit: It looks like it sometimes goes up to 30%, but the conclusion still holds.
Edit2: If you have a sufficiently “selfish” utility function, it would be short-term rational to always click the airstrike button. The scenario doesn’t outline any larger picture in terms of consequences for you personally.
Ah, apparently I rolled maximum hard mode that time, as it was indeed 30% chance of fellow soldier death
I reasoned similarly that the cost of a FP was less than for a FN and called in the air strike; it told me some other guy died. I reloaded the same scenario and tried a direct attack; I got shot by a sniper.