Instead you could select maximally informative data points to ask them about.
In this case, information is measured by how much of thingspace would be sheared if it turned out that a data point should be classified as ‘unknown’. It isn’t immediately clear how to find this without a tractable thingspace-volume-subroutine, but I think this would be computationally-efficient for both of our ideas.
I’ll bet you could do some math to determine how to get the strongest statistical guarantees with the minimum amount of money spent on MTurk too.
The technique you’re probably looking for is called Bayesian Optimization. Aside: at my school, ‘Optimization’ - not ‘Conspiracy’ - is unfortunately the word which most frequently follows ‘Bayesian’.
If the dog is represented using a convex polytope instead of a sphere, you might even reverse engineer the corners of your current classifier region, and then display them all to the user to show how expansive the classifier’s notion of “dog” is. But the map is not the territory: It’s possible that in some cases, the shape the user wants is actually concave.
Even an imperfect estimate of the volume would be useful: for example, perhaps we only find some of the edges and conclude the volume is some fraction of its true value. I have the distinct sense of talking past the point you were trying to make, though.
Even an imperfect estimate of the volume would be useful: for example, perhaps we only find some of the edges and conclude the volume is some fraction of its true value. I have the distinct sense of talking past the point you were trying to make, though.
In this case, information is measured by how much of thingspace would be sheared if it turned out that a data point should be classified as ‘unknown’. It isn’t immediately clear how to find this without a tractable thingspace-volume-subroutine, but I think this would be computationally-efficient for both of our ideas.
The technique you’re probably looking for is called Bayesian Optimization. Aside: at my school, ‘Optimization’ - not ‘Conspiracy’ - is unfortunately the word which most frequently follows ‘Bayesian’.
Even an imperfect estimate of the volume would be useful: for example, perhaps we only find some of the edges and conclude the volume is some fraction of its true value. I have the distinct sense of talking past the point you were trying to make, though.
No, that sounds more or less right.