Thought-provoking post, though as you hinted it’s not fair to directly compare “classification accuracy” with “accuracy at avoiding catastrophe”. Humans are probably less reliable than deep learning systems at this point in terms of their ability to classify images and understand scenes, at least given < 1 second of response time. Instead, human ability to avoid catastrophe is an ability to generate conservative action sequences in response to novel physical and social situations—e.g. if I’m driving and I see something I don’t understand up ahead I’ll slow down just in case.
I imagine if our goal was “never misclassify an MNIST digit” we could get to 6-7 nines of “worst-case accuracy” even out of existing neural nets, at the cost of saying “I don’t know” for the confusing 0.2% of digits.
>I imagine if our goal was “never misclassify an MNIST digit” we could get to 6-7 nines of “worst-case accuracy” even out of existing neural nets, at the cost of saying “I don’t know” for the confusing 0.2% of digits.
Er, how? I haven’t seen anyone describe a way to do this. Getting a neural network to meaningfully say “I don’t know” is very much cutting-edge research as far as I’m aware.
You’re right that it’s an ongoing research area but there’s a number of approaches that work relatively well. This NeurIPS tutorial describes a few. Probably the easiest thing is to use one of the calibration methods mentioned there to get your classifier to output calibrated uncertainties for each class, then say “I don’t know” if the network isn’t at least 90% confident in one of the 10 classes.
OK, thanks for linking that. You’re probably right in the specific example of MNIST. I’m less convinced about more complicated tasks—it seems like each individual task would require a lot of engineering effort.
One thing I didn’t see—is there research which looks at what happens if you give neural nets more of the input space as data? Things which are explicitly out-of-distribution, random noise, abstract shapes, or maybe other modes that you don’t particularly care about performance on, and label it all as “garbage” or whatever. Essentially, providing negative as well as positive examples, given that the input spaces are usually much larger than the intended distribution.
Humans are probably less reliable than deep learning systems at this point in terms of their ability to classify images and understand scenes, at least given < 1 second of response time.
Another way to frame this point is that humans are always doing multi-modal processing in the background, even for tasks which require only considering one sensory modality. Doing this sort of multi-modal cross checking by default offers better edge case performance at the cost of lower efficiency in the average case.
I’ll add to this: humans make errors all the time, but we have institutions and systems in place to ensure that those errors don’t lead to catastrophe, by having additional safeguards beyond just hoping that one person doesn’t screw up.
Thought-provoking post, though as you hinted it’s not fair to directly compare “classification accuracy” with “accuracy at avoiding catastrophe”. Humans are probably less reliable than deep learning systems at this point in terms of their ability to classify images and understand scenes, at least given < 1 second of response time. Instead, human ability to avoid catastrophe is an ability to generate conservative action sequences in response to novel physical and social situations—e.g. if I’m driving and I see something I don’t understand up ahead I’ll slow down just in case.
I imagine if our goal was “never misclassify an MNIST digit” we could get to 6-7 nines of “worst-case accuracy” even out of existing neural nets, at the cost of saying “I don’t know” for the confusing 0.2% of digits.
>I imagine if our goal was “never misclassify an MNIST digit” we could get to 6-7 nines of “worst-case accuracy” even out of existing neural nets, at the cost of saying “I don’t know” for the confusing 0.2% of digits.
Er, how? I haven’t seen anyone describe a way to do this. Getting a neural network to meaningfully say “I don’t know” is very much cutting-edge research as far as I’m aware.
You’re right that it’s an ongoing research area but there’s a number of approaches that work relatively well. This NeurIPS tutorial describes a few. Probably the easiest thing is to use one of the calibration methods mentioned there to get your classifier to output calibrated uncertainties for each class, then say “I don’t know” if the network isn’t at least 90% confident in one of the 10 classes.
OK, thanks for linking that. You’re probably right in the specific example of MNIST. I’m less convinced about more complicated tasks—it seems like each individual task would require a lot of engineering effort.
One thing I didn’t see—is there research which looks at what happens if you give neural nets more of the input space as data? Things which are explicitly out-of-distribution, random noise, abstract shapes, or maybe other modes that you don’t particularly care about performance on, and label it all as “garbage” or whatever. Essentially, providing negative as well as positive examples, given that the input spaces are usually much larger than the intended distribution.
Another way to frame this point is that humans are always doing multi-modal processing in the background, even for tasks which require only considering one sensory modality. Doing this sort of multi-modal cross checking by default offers better edge case performance at the cost of lower efficiency in the average case.
I’ll add to this: humans make errors all the time, but we have institutions and systems in place to ensure that those errors don’t lead to catastrophe, by having additional safeguards beyond just hoping that one person doesn’t screw up.