Sorry if this wasn’t clear in the post but when I say “we’re” trying to protect some submodule, I don’t mean us as the engineers who want to make sure the model doesn’t change a submodule (we could do that trivially, just add a stop gradient), but rather from the perspective of a gradient hacker that needs to protect its own logic from being disabled by SGD.
Sorry if this wasn’t clear in the post but when I say “we’re” trying to protect some submodule, I don’t mean us as the engineers who want to make sure the model doesn’t change a submodule (we could do that trivially, just add a stop gradient), but rather from the perspective of a gradient hacker that needs to protect its own logic from being disabled by SGD.
Ah, that makes sense.