We could have a contest, where each contestant tries to describe a scenario that has the largest utility to a judge. I bet that after a few rounds of this, we’ll converge on some scenario of maximum utility, no matter who the judge is.
Does this show that utility can’t go arbitrarily high?
ETA: The above perhaps only shows the difficulty of not getting stuck in a local maximum. Maybe a better argument is that a human mind can only consider a finite subset of configuration space. The point in that subset with the largest utility must be the maximum utility for that mind.
We could have a contest, where each contestant tries to describe a scenario that has the largest utility to a judge. I bet that after a few rounds of this, we’ll converge on some scenario of maximum utility, no matter who the judge is.
Does this show that utility can’t go arbitrarily high?
ETA: The above perhaps only shows the difficulty of not getting stuck in a local maximum. Maybe a better argument is that a human mind can only consider a finite subset of configuration space. The point in that subset with the largest utility must be the maximum utility for that mind.