I was actually reading this post and I was trying to find a solution to the coalition problem where Eliezer wonders how rational agents can solve a problem with the potential for an infinite loop, which lead me to what I’ll call the Waiting Game, where you can wait n units of time and gain n utility for any finite n, which then led me to this post.
Suppose instead that the game is “gain n utility”. No need to speak the number, wait n turns, or even to wait for a meat brain to make a decision or comprehend the number.
I posit that a perfectly rational, disembodied agent would decide to select an n such that there exists no n higher. If there is a possible outcome that such an agent prefers over all other possible outcomes, then by the definition of utility such an n exists.
Spoilers, haha.
I was actually reading this post and I was trying to find a solution to the coalition problem where Eliezer wonders how rational agents can solve a problem with the potential for an infinite loop, which lead me to what I’ll call the Waiting Game, where you can wait n units of time and gain n utility for any finite n, which then led me to this post.
Suppose instead that the game is “gain n utility”. No need to speak the number, wait n turns, or even to wait for a meat brain to make a decision or comprehend the number.
I posit that a perfectly rational, disembodied agent would decide to select an n such that there exists no n higher. If there is a possible outcome that such an agent prefers over all other possible outcomes, then by the definition of utility such an n exists.
Not quite. There is no reason inherent in the definition that utility has to be bounded.