In this case, with only selectivity ratings, you can’t do better than 50% accuracy (any person wants to date any other person with 50% probability). But with eye-color ratings, you can get it perfect.
Edit: I initially misread your remark. I tried to clarify the setup with:
In this blog post I’m restricting consideration to signals of the partners’ general selectivity and general desirability, without considering how their traits interact.
Is this ambiguous?
My impression is that there are significant structural correlations in your data that I don’t really understand the impact of. (For instance, at least if everyone rates everyone, I think the correlation of attr with attrAvg is guaranteed to be positive, even if attr is completely random.)
I may not fully parse what you have in mind, but I excluded the rater and ratee from the averages. This turns out not to be enough to avoid contamination for subtle reasons, so I made a further modification. I’ll be discussing this later, but if you’re wondering about this particular point, I’d be happy to now.
The relevant code is here. Your remark prompted me to check my code by replacing the ratings with random numbers drawn from a normal distribution. Using 7 ratings and 7 averages, the mean correlation is 0.003, with 23 negative and 26 positive.
Nitpick: There are only 10 distinct such correlations that are not 1 by definition.
Thanks, that was an oversight on my part. I’ve edited the text.
Model accuracy actually isn’t actually a great measure of predictive power, because it’s sensitive to base rates. (You at least mentioned the base rates, but it’s still hard to know how much to correct for the base rates when you’re interpreting the goodness of a classifier.)
I suppressed technical detail in this first post to make it more easily accessible to a general audience. I’m not sure whether this answers your question, but I used log loss as a measure of accuracy. The differentials were (approximately, the actual final figures are lower):
For Men: ~0.690 to ~0.500.
For Women: ~0.635 to ~0.567.
For Matches: ~0.432 to ~0.349
I’ll also be giving figures within the framework of recommendation systems in a later post.
As far as I know, if you don’t have a utility function, scoring classifiers in an interpretable way is still kind of an open problem, but you could look at ROC AUC as a still-interpretable but somewhat nicer summary statistic of model performance.
It wasn’t clear that this applied to the statement “we couldn’t improve on using these” (mainly because I forgot you weren’t considering interactions).
I excluded the rater and ratee from the averages.
Okay, that gets rid of most of my worries. I’m not sure it account for covariance between correlation estimates of different averages, so I’d be interested in seeing some bootstrapped confidence intervals). But perhaps I’m preempting future posts.
Also, thinking about it more, you point out a number of differences between correlations, and it’s not clear to me that those differences are significant as opposed to just noise.
I’m not sure whether this answers your question, but I used log loss as a measure of accuracy.
I was using “accuracy” in the technical sense, i.e., one minus what you call “Total Error” in your table. (It’s unfortunate that Wikipedia says scoring rules like log-loss are a measure of the “accuracy” of predictions! I believe the technical usage, that is, percentage properly classified for a binary classifier, is a more common usage in machine learning.)
The total error of a model is in general not super informative because it depends on the base rate of each class in your data, as well as the threshold that you choose to convert your probabilistic classifier into a binary one. That’s why I generally prefer to see likelihood ratios, as you just reported, or ROC AUC scores (which integrates over a range of thresholds).
(Although apparently using AUC for model comparison is questionable too, because it’s noisy and incoherent in some circumstances and doesn’t penalize miscalibration, so you should use the H measure instead. I mostly like it as a relatively interpretable, utility-function-independent rough index of a model’s usefulness/discriminative ability, not a model comparison criterion.)
Also, thinking about it more, you point out a number of differences between correlations, and it’s not clear to me that those differences are significant as opposed to just noise.
Thanks Ben!
Edit: I initially misread your remark. I tried to clarify the setup with:
In this blog post I’m restricting consideration to signals of the partners’ general selectivity and general desirability, without considering how their traits interact.
Is this ambiguous?
I may not fully parse what you have in mind, but I excluded the rater and ratee from the averages. This turns out not to be enough to avoid contamination for subtle reasons, so I made a further modification. I’ll be discussing this later, but if you’re wondering about this particular point, I’d be happy to now.
The relevant code is here. Your remark prompted me to check my code by replacing the ratings with random numbers drawn from a normal distribution. Using 7 ratings and 7 averages, the mean correlation is 0.003, with 23 negative and 26 positive.
Thanks, that was an oversight on my part. I’ve edited the text.
I suppressed technical detail in this first post to make it more easily accessible to a general audience. I’m not sure whether this answers your question, but I used log loss as a measure of accuracy. The differentials were (approximately, the actual final figures are lower):
For Men: ~0.690 to ~0.500. For Women: ~0.635 to ~0.567. For Matches: ~0.432 to ~0.349
I’ll also be giving figures within the framework of recommendation systems in a later post.
Thanks, I’ve been meaning to look into this.
It wasn’t clear that this applied to the statement “we couldn’t improve on using these” (mainly because I forgot you weren’t considering interactions).
Okay, that gets rid of most of my worries. I’m not sure it account for covariance between correlation estimates of different averages, so I’d be interested in seeing some bootstrapped confidence intervals). But perhaps I’m preempting future posts.
Also, thinking about it more, you point out a number of differences between correlations, and it’s not clear to me that those differences are significant as opposed to just noise.
I was using “accuracy” in the technical sense, i.e., one minus what you call “Total Error” in your table. (It’s unfortunate that Wikipedia says scoring rules like log-loss are a measure of the “accuracy” of predictions! I believe the technical usage, that is, percentage properly classified for a binary classifier, is a more common usage in machine learning.)
The total error of a model is in general not super informative because it depends on the base rate of each class in your data, as well as the threshold that you choose to convert your probabilistic classifier into a binary one. That’s why I generally prefer to see likelihood ratios, as you just reported, or ROC AUC scores (which integrates over a range of thresholds).
(Although apparently using AUC for model comparison is questionable too, because it’s noisy and incoherent in some circumstances and doesn’t penalize miscalibration, so you should use the H measure instead. I mostly like it as a relatively interpretable, utility-function-independent rough index of a model’s usefulness/discriminative ability, not a model comparison criterion.)
More to follow (about to sleep), but regarding
What do you have in mind specifically?