I resent the implication that I need to “read the literature” or “do my homework” before I can meaningfully contribute to a problem of this sort.
The title of my post is “how 2 tell if ur input is out of distribution given only model weights”. That is, given just the model, how can you tell which inputs the model “expects” more? I don’t think any of the resources you refer to are particularly helpful there.
We propose leveraging [diverse image and text] data to improve deep anomaly detection by training anomaly detectors against an auxiliary dataset of outliers, an approach we call Outlier Exposure (OE). This enables anomaly detectors to generalize and detect unseen anomalies.
Paper 2 just says “softmax classifers tend to make more certain predictions on in-distribution inputs”. I should certainly hope so. (Of course, not every model is a softmax classifer.)
We present a simple baseline that utilizes probabilities from softmax distributions. Correctly classified examples tend to have greater maximum softmax probabilities than erroneously classified and out-of-distribution examples, allowing for their detection.
Paper 3 requires you to know the training set, and also it only works on models that happen to be softmax classifiers.
Paper 4 requires a dataset of in-distribution data, it requires you to train a classifier for every model you want to use their methods with, and it looks like it requires the data to be separated into various classes.
Paper 5 is basically the same as Paper 2, except it says “logits” instead of “probabilities”, and includes more benchmarks.
We [...] find that a surprisingly simple detector based on the maximum logit outperforms prior methods in all the large-scale multi-class, multi-label, and segmentation tasks.
Paper 6 only works for classifiers and it also requires you to provide an in-distribution dataset.
We obtain the class conditional Gaussian distributions with respect to (low- and upper-level) features of the deep models under Gaussian discriminant analysis, which result in a confidence score based on the Mahalanobis distance.
It seems that all of the six methods you referred me to either (1) require you to bring a dataset, or (2) reduce to “Hey guys, classifiers make less confident predictions OOD!”. Therefore, I feel perfectly fine about failing to acknowledge the extant academic literature here.
(Additionally, the methods in my post were also replicated in language models by @voooooogel:
There’s a literature on this topic. (paper list, lecture/slides/homework)
I resent the implication that I need to “read the literature” or “do my homework” before I can meaningfully contribute to a problem of this sort.
The title of my post is “how 2 tell if ur input is out of distribution given only model weights”. That is, given just the model, how can you tell which inputs the model “expects” more? I don’t think any of the resources you refer to are particularly helpful there.
Your paper list consists of six arXiv papers (1, 2, 3, 4, 5, 6).
Paper 1 requires you to bring a dataset.
Paper 2 just says “softmax classifers tend to make more certain predictions on in-distribution inputs”. I should certainly hope so. (Of course, not every model is a softmax classifer.)
Paper 3 requires you to know the training set, and also it only works on models that happen to be softmax classifiers.
Paper 4 requires a dataset of in-distribution data, it requires you to train a classifier for every model you want to use their methods with, and it looks like it requires the data to be separated into various classes.
Paper 5 is basically the same as Paper 2, except it says “logits” instead of “probabilities”, and includes more benchmarks.
Paper 6 only works for classifiers and it also requires you to provide an in-distribution dataset.
It seems that all of the six methods you referred me to either (1) require you to bring a dataset, or (2) reduce to “Hey guys, classifiers make less confident predictions OOD!”. Therefore, I feel perfectly fine about failing to acknowledge the extant academic literature here.
(Additionally, the methods in my post were also replicated in language models by @voooooogel: