If you find yourself thinking about the differences between geometric expected utility and expected utility in terms of utility functions, remind yourself that, for any utility function, one can choose* either* averaging method.
No, you can only use the geometric expected utility for nonnegative utility functions.
This doesn’t completely explain the trick, though. In the step where you write f=(1-I)^{-1} 0, if you interpret I as an operator then you get f=0 as the result. To get f=Ce^x you need to have f=(1-I)^{-1} C in that step instead. You can get this by replacing \int f by If+C at the beginning.