Ordinary probability theory and expected utility are sufficient to handle this puzzle. You just have to calculate the expected utility of each strategy before choosing a strategy. In this puzzle a strategy is more complicated than simply putting some number of coins in the machine: it requires deciding what to do after each coin either succeeds or fails to succeed in releasing two coins.
In other words, a strategy is a choice of what you’ll do at each point in the game tree—just like a strategy in chess.
We don’t expect to do well at chess if we decide on a course of action that ignores our opponent’s moves. Similarly, we shouldn’t expect to do well in this probabilistic game if we only consider strategies that ignore what the machine does. If we consider all strategies, compute their expected utility based on the information we have, and choose the one that maximizes this, we’ll do fine.
I’m saying essentially the same thing Jeremy Salwen said.
Thanks for writing this—I’ve added links in my article recommending that people read yours.