Idle thought, might flesh it out later: I wonder if there’s a way to explore these “bugs” in a more systematic, mechanized way.
Right now you’re discovering these interesting cases by hand, basically the ML equivalent of reading code and adding individual unit tests to functions whose behavior you’re exploring. What you might want to do is something closer to “fuzzing” where you have a computer actively seek out these edge cases and point them out to you.
Now, actual fuzzers for real programs are based on coverage, which doesn’t trivially work with neural network because every weight is essentially visited once per token.
Still, there might be an analogy to be found? Maybe a fuzzer could try exploring “paths” that have very low attention scores somehow?
Idle thought, might flesh it out later: I wonder if there’s a way to explore these “bugs” in a more systematic, mechanized way.
Right now you’re discovering these interesting cases by hand, basically the ML equivalent of reading code and adding individual unit tests to functions whose behavior you’re exploring. What you might want to do is something closer to “fuzzing” where you have a computer actively seek out these edge cases and point them out to you.
Now, actual fuzzers for real programs are based on coverage, which doesn’t trivially work with neural network because every weight is essentially visited once per token.
Still, there might be an analogy to be found? Maybe a fuzzer could try exploring “paths” that have very low attention scores somehow?
Each relu is a branch. I don’t see why you couldn’t do branch-guided coverage with relus as branches.
Sure, waves hands, something like that.