It seems to me like the whistler is saying that the probability of saving knuth people for $5 is exactly 1/knuth after updating for the Matrix Lord’s claim, not before the claim, which seems surprising.
Also, it’s not clear that we need to make an FAI resistant to very very unlikely scenarios.
Also, if the AI spreads widely and is around for a long time, it will eventually run into very unlikely scenarios. Not 1/3^^^3 unlikely, but pretty unlikely.
It seems to me like the whistler is saying that the probability of saving knuth people for $5 is exactly 1/knuth after updating for the Matrix Lord’s claim, not before the claim, which seems surprising. Also, it’s not clear that we need to make an FAI resistant to very very unlikely scenarios.
I’m a lot more worried about making an FAI behave correctly if it encounters a scenario which we thought was very very unlikely.
Also, if the AI spreads widely and is around for a long time, it will eventually run into very unlikely scenarios. Not 1/3^^^3 unlikely, but pretty unlikely.