I agree, but I don’t see why that’s relevant? The point of the “Adversarial Spheres” paper is not that the dataset is realistic, of course, but that studying an unrealistically simple dataset might offer generalizable insights. If the ground truth decision boundary is a sphere, but your neural net learns a “squiggly” ellipsoid that admits adversarial examples (because SGD is just brute-forcing a fit rather than doing something principled that could notice hypotheses on the order of, “hey, it’s a sphere”), that’s a clue that when the ground truth is something complicated, your neural net is also going to learn something squiggly that admits adversarial examples (where the squiggles in your decision boundary predictably won’t match the complications in your dataset, even though they’re both not-simple).
My suspicion is that for a lot of categorization problems we care about, there isn’t a nice smooth boundary between categories such that an adversarially robust classifier is possible, so the failure of the “lol stack more layers” approach to find such boundaries in the rare cases where those boundaries do exist isn’t super impactful.
Strong belief weakly held on the “most real-world category boundaries are not smooth”.
Sorry, this doesn’t make sense to me. The boundary doesn’t need to be smooth in an absolute sense in order to exist and be learnable (whether by neural nets or something else). There exists a function from business plans to their profitability. The worry is that if you try to approximate that function with standard ML tools, then even if your approximation is highly accurate on any normal business plan, it’s not hard to construct an artificial plan on which it won’t be. But this seems like a limitation of the tools; I don’t think it’s because the space of business plans is inherently fractally complex and unmodelable.
I genuinely think that the space of “what level of success will a business have if it follows its business plan” is inherently fractal in the same way that “which root of a polynomial will repeated iteration of Newton’s method converge to” is inherently fractal. For some plans, a tiny change to the plan can lead to a tiny change in behavior, which can lead to a giant change in outcome.
Which is to say “it is, at most points it doesn’t matter that it is, but if the point is adversarially selected you once again have to care”.
All that said, this is a testable hypothesis. I can’t control the entire world closely enough to run tiny variations on a business plan and plot the results on a chart, but I could do something like “take the stable diffusion text encoder, encode three different prompts (e.g. ‘a horse’, ‘a salad bowl’, ‘a mountain’) and then, holding the input noise steady, generate an image for each blend, classify the output images, and plot the results”. Do you have strong intuitions about what the output chart would look like?
I agree, but I don’t see why that’s relevant? The point of the “Adversarial Spheres” paper is not that the dataset is realistic, of course, but that studying an unrealistically simple dataset might offer generalizable insights. If the ground truth decision boundary is a sphere, but your neural net learns a “squiggly” ellipsoid that admits adversarial examples (because SGD is just brute-forcing a fit rather than doing something principled that could notice hypotheses on the order of, “hey, it’s a sphere”), that’s a clue that when the ground truth is something complicated, your neural net is also going to learn something squiggly that admits adversarial examples (where the squiggles in your decision boundary predictably won’t match the complications in your dataset, even though they’re both not-simple).
My suspicion is that for a lot of categorization problems we care about, there isn’t a nice smooth boundary between categories such that an adversarially robust classifier is possible, so the failure of the “lol stack more layers” approach to find such boundaries in the rare cases where those boundaries do exist isn’t super impactful.
Strong belief weakly held on the “most real-world category boundaries are not smooth”.
Sorry, this doesn’t make sense to me. The boundary doesn’t need to be smooth in an absolute sense in order to exist and be learnable (whether by neural nets or something else). There exists a function from business plans to their profitability. The worry is that if you try to approximate that function with standard ML tools, then even if your approximation is highly accurate on any normal business plan, it’s not hard to construct an artificial plan on which it won’t be. But this seems like a limitation of the tools; I don’t think it’s because the space of business plans is inherently fractally complex and unmodelable.
I genuinely think that the space of “what level of success will a business have if it follows its business plan” is inherently fractal in the same way that “which root of a polynomial will repeated iteration of Newton’s method converge to” is inherently fractal. For some plans, a tiny change to the plan can lead to a tiny change in behavior, which can lead to a giant change in outcome.
Which is to say “it is, at most points it doesn’t matter that it is, but if the point is adversarially selected you once again have to care”.
All that said, this is a testable hypothesis. I can’t control the entire world closely enough to run tiny variations on a business plan and plot the results on a chart, but I could do something like “take the stable diffusion text encoder, encode three different prompts (e.g. ‘a horse’, ‘a salad bowl’, ‘a mountain’) and then, holding the input noise steady, generate an image for each blend, classify the output images, and plot the results”. Do you have strong intuitions about what the output chart would look like?