Infinite utilities violate VNM-rationality. Unbounded utility functions do too, because they allow you to construct gambles that have infinite utility. For instance, if the utility function is unbounded, then there exists a sequence of outcomes such that for each n, the utility of the nth outcome is at least 2^n. Then the utility of the gamble that, for each positive integer n, gives you a 1/2^n chance of getting the nth outcome, has infinite utility.
In the case of utility functions that are bounded but do not have a maximum, the problem is not particularly worrying. If you pick a tiny amount of utility epsilon, you can ensure that you will never sacrifice more than epsilon utility. An agent that does this, while not optimal, will be pretty good provided that it actually does always choose tiny values of epsilon.
Infinite utilities violate VNM-rationality. Unbounded utility functions do too, because they allow you to construct gambles that have infinite utility. For instance, if the utility function is unbounded, then there exists a sequence of outcomes such that for each n, the utility of the nth outcome is at least 2^n. Then the utility of the gamble that, for each positive integer n, gives you a 1/2^n chance of getting the nth outcome, has infinite utility.
In the case of utility functions that are bounded but do not have a maximum, the problem is not particularly worrying. If you pick a tiny amount of utility epsilon, you can ensure that you will never sacrifice more than epsilon utility. An agent that does this, while not optimal, will be pretty good provided that it actually does always choose tiny values of epsilon.