I started asking for a chess example because you implied that the reasoning in the top-level comment stops being sane in iterated games.
In a simple iteration of Troll bridge, whether we’re dumb is clear after the first time we cross the bridge.
Right, OK. I would say “sequential” rather than “iterated”—my point was about making a weird assessment of your own future behavior, not what you can do if you face the same scenario repeatedly. IE: Troll Bridge might be seen as artificial in that the environment is explicitly designed to punish you if you’re “dumb”; but, perhaps a sequential game can punish you more naturally by virtue of poor future choices.
Suppose my chess skill varies by day. If my last few moves were dumb, I shouldn’t rely on my skill today. I don’t see why I shouldn’t deduce this ahead of time
Yep, I agree with this.
I concede the following points:
If there is a mistake in the troll-bridge reasoning, predicting that your next actions are likely to be dumb conditional on a dumb-looking action is not an example of the mistake.
Furthermore, that inference makes perfect sense, and if it is as analogous to the troll-bridge reasoning as I was previously suggesting, the troll-bridge reasoning makes sense.
However, I still assert the following:
Predicting that your next actions are likely to be dumb conditional on a dumb looking action doesn’t make sense if the very reason why you think the action looks dumb is that the next actions are probably dumb if you take it.
IE, you don’t have a prior heuristic judgement that a move is one which you make when you’re dumb; rather, you’ve circularly concluded that the move would be dumb—because it’s likely to lead to a bad outcome—because if you take that move your subsequent moves are likely to be bad—because it is a dumb move.
I don’t have a natural setup which would lead to this, but the point is that it’s a crazy way to reason rather than a natural one.
The question, then, is whether the troll-bridge reasoning is analogous to to this.
I think we should probably focus on the probabilistic case (recently added to the OP), rather than the proof-based agent. I could see myself deciding that the proof-based agent is more analogous to the sane case than the crazy one. But the probabilistic case seems completely wrong.
In the proof-based case, the question is: do we see the Löbian proof as “circular” in a bad way? It makes sense to conclude that you’d only cross the bridge when it is bad to do so, if you can see that proving it’s a good idea is inconsistent. But does the proof that that’s inconsistent “go through” that very inference? We know that the troll blows up the bridge if we’re dumb, but that in itself doesn’t constitute outside reason that crossing is dumb.
But I can see an argument that our “outside reason” is that we can’t know that crossing is safe, and since we’re a proof-based agent, would never take the risk unless we’re being dumb.
However, this reasoning does not apply to the probabilistic agent. It can cross the bridge as a calculated risk. So its reasoning seems absolutely circular. There is no “prior reason” for it to think crossing is dumb; and, even if it did think it more likely dumb than not, it doesn’t seem like it should be 100% certain of that. There should be some utilities for the three outcomes which preserve the preference ordering but which make the risk of crossing worthwhile.
Right, OK. I would say “sequential” rather than “iterated”—my point was about making a weird assessment of your own future behavior, not what you can do if you face the same scenario repeatedly. IE: Troll Bridge might be seen as artificial in that the environment is explicitly designed to punish you if you’re “dumb”; but, perhaps a sequential game can punish you more naturally by virtue of poor future choices.
Yep, I agree with this.
I concede the following points:
If there is a mistake in the troll-bridge reasoning, predicting that your next actions are likely to be dumb conditional on a dumb-looking action is not an example of the mistake.
Furthermore, that inference makes perfect sense, and if it is as analogous to the troll-bridge reasoning as I was previously suggesting, the troll-bridge reasoning makes sense.
However, I still assert the following:
Predicting that your next actions are likely to be dumb conditional on a dumb looking action doesn’t make sense if the very reason why you think the action looks dumb is that the next actions are probably dumb if you take it.
IE, you don’t have a prior heuristic judgement that a move is one which you make when you’re dumb; rather, you’ve circularly concluded that the move would be dumb—because it’s likely to lead to a bad outcome—because if you take that move your subsequent moves are likely to be bad—because it is a dumb move.
I don’t have a natural setup which would lead to this, but the point is that it’s a crazy way to reason rather than a natural one.
The question, then, is whether the troll-bridge reasoning is analogous to to this.
I think we should probably focus on the probabilistic case (recently added to the OP), rather than the proof-based agent. I could see myself deciding that the proof-based agent is more analogous to the sane case than the crazy one. But the probabilistic case seems completely wrong.
In the proof-based case, the question is: do we see the Löbian proof as “circular” in a bad way? It makes sense to conclude that you’d only cross the bridge when it is bad to do so, if you can see that proving it’s a good idea is inconsistent. But does the proof that that’s inconsistent “go through” that very inference? We know that the troll blows up the bridge if we’re dumb, but that in itself doesn’t constitute outside reason that crossing is dumb.
But I can see an argument that our “outside reason” is that we can’t know that crossing is safe, and since we’re a proof-based agent, would never take the risk unless we’re being dumb.
However, this reasoning does not apply to the probabilistic agent. It can cross the bridge as a calculated risk. So its reasoning seems absolutely circular. There is no “prior reason” for it to think crossing is dumb; and, even if it did think it more likely dumb than not, it doesn’t seem like it should be 100% certain of that. There should be some utilities for the three outcomes which preserve the preference ordering but which make the risk of crossing worthwhile.