Sometimes it seems that a commenter did not slow down enough to read the whole paper, or read it carefully enough, and I find myself forced to rewrite the entire paper in a comment.
The basic story is that your hypothetical internal monologue from the AGI, above, did not seem to take account of ANY of the argument in the paper. The goal of the paper was not to look inside the AGI’s thoughts, but to discuss its motivation engine. The paper had many constructs and arguments (scattered all over the place) that would invalidate the internal monologue that you wrote down, so it seemed you had not read the paper.
Sometimes it seems that a commenter did not slow down enough to read the whole paper, or read it carefully enough, and I find myself forced to rewrite the entire paper in a comment.
The basic story is that your hypothetical internal monologue from the AGI, above, did not seem to take account of ANY of the argument in the paper. The goal of the paper was not to look inside the AGI’s thoughts, but to discuss its motivation engine. The paper had many constructs and arguments (scattered all over the place) that would invalidate the internal monologue that you wrote down, so it seemed you had not read the paper.