The crucial point is: when the untrusted model gets to step 6, it sees R in the history instead of A, so it knows its attempt was replaced.
Why does it know its attempt was replaced? Because it can guess it wouldn’t have produced R? If it can tell that, how come it can’t guess it specifically would have produced A?
Why does it know its attempt was replaced? Because it can guess it wouldn’t have produced R?
Yep.
If it can tell that, how come it can’t guess it specifically would have produced A?
I’m just hypothesizing that the first guessing problem is easier than the second; I intuitively expect that it’s a lot easier to notice that something isn’t your style than to guess what exactly you would have done (among other things, there are many more bits in “what action did I take” than “did I take this action”).
Why does it know its attempt was replaced? Because it can guess it wouldn’t have produced R? If it can tell that, how come it can’t guess it specifically would have produced A?
Yep.
I’m just hypothesizing that the first guessing problem is easier than the second; I intuitively expect that it’s a lot easier to notice that something isn’t your style than to guess what exactly you would have done (among other things, there are many more bits in “what action did I take” than “did I take this action”).