Wait, where? I think the objection to “Doing that is quite hard” is not an objection to “it’s not obviously true that such algorithms are actually “achievable” for SGD”—it’s an objection to the conclusion that model would try hard enough to justify arguments about deception from weak statement about loss decreasing during training.
an objection to the conclusion that model would try hard enough to justify arguments about deception from weak statement about loss decreasing during training.
That’s what I thought he was saying previously, but he objected to that characterization in his most recent comment.
Wait, where? I think the objection to “Doing that is quite hard” is not an objection to “it’s not obviously true that such algorithms are actually “achievable” for SGD”—it’s an objection to the conclusion that model would try hard enough to justify arguments about deception from weak statement about loss decreasing during training.
This is… roughly one point I was making, yes.