Essentially, take advantage of the fact that we are finite state probabilistic machines (or analogous to that), and therefore there is a maximum to the number of choices we can expect to make. So our option set is actually finite (though brutally large).
I’m referring to an infinity of possible outcomes, not an infinity of possible choices. This problem still applies if the agent must pick from a finite list of actions.
Specifically, I’m referring to the problem discussed in this paper, which is mostly the same problem as Pascal’s mugging.
Interesting problem, thanks! I personally felt that there could be a good case made for insisting your utility be bounded, and that paper’s an argument in that direction.
Pascal’s mugging is less of a problem if your utility function is bounded, and it completely goes away if the bound is reasonably low, since then there just isn’t any amount of utility that would outweight the improbability of the mugger being truthful.
I was working on a way to do that—but I’m also aware that not all divergent expectations can be compared, and so that there might be a case to avoid using unbounded utilities.
The convergence can be solved using the arguments I presented in: http://lesswrong.com/lw/giu/naturalism_versus_unbounded_or_unmaximisable/
Essentially, take advantage of the fact that we are finite state probabilistic machines (or analogous to that), and therefore there is a maximum to the number of choices we can expect to make. So our option set is actually finite (though brutally large).
I’m referring to an infinity of possible outcomes, not an infinity of possible choices. This problem still applies if the agent must pick from a finite list of actions.
Specifically, I’m referring to the problem discussed in this paper, which is mostly the same problem as Pascal’s mugging.
Interesting problem, thanks! I personally felt that there could be a good case made for insisting your utility be bounded, and that paper’s an argument in that direction.
Pascal’s mugging is less of a problem if your utility function is bounded, and it completely goes away if the bound is reasonably low, since then there just isn’t any amount of utility that would outweight the improbability of the mugger being truthful.
Weren’t you working on ways to compare infinite/divergent expectations? I’m confused that you’re now writing as if the problem is new to you...
I was working on a way to do that—but I’m also aware that not all divergent expectations can be compared, and so that there might be a case to avoid using unbounded utilities.