Does that make sense?
In a word, no.
I believe you are thinking of infinity as a number, and that’s always a mistake. I think that what you’re trying to say with your left-hand graph is that, given infinite utility, probability is a tiebreaker, but all infinite-utility options dominate all finite utilities. But this treats “infinity” as a binary quality which an option either has or not.
Consider two different Pascal’s muggers: One offers you a 1% probability of utility increasing linearly in time, the other, a 1% chance of utility increasing exponentially with time. Clearly both options “are infinite”; equally clearly, you prefer the second one even though the probabilities are the same. They occupy the same point on your left-hand graph. But by your suggested decision procedure you would choose the linearly-increasing option if the first mugger offered even an epsilon increase in probability; and this is obviously Weird. It gives you a smaller expected utility at almost all points in time!
Mu: Question cannot be answered because “win” is not defined. Does winning require
a) Dictating terms, in the style of Versailles?
b) As above, but also not burning to cinders the social technology that allowed you to fight such a war in the first place? (As happened to the OTL victors.)
c) As above, but also getting some kind of actual net benefit either in geopolitical-power terms or in goods for your citizens? (As very noticeably did not occur for the OTL “victors”.)
d) A negotiated peace in which it’s generally recognised that you had the upper hand and got most of the surplus? (Surplus relative to continuing the war, that is.) Same variants as above.
e) Any peace that avoids the total collapse of OTL Germany and resulting even-more-disastrous war?
f) Any peace in which the prewar decision makers emerge with their personal power and prestige enhanced, whatever happens in Germany at large?
g) Avoiding the conflict entirely? (Best option! Investing in productive assets will get you a lot more benefit than trying to win a massively negative-sum game!)