Sociosexual Orientation Inventory, or failing to perform basic sanity checks
I just did some reading about “Sociosexual Orientation Inventory”, a simple 7-item test designed to measure one’s openness to sex without love and long term commitment.
Here are the questions. How long will it take you to spot the huge problem ahead...
With how many different partners have you had sex (sexual intercourse) within the last year.
How many different partners do you foresee yourself having sex with during the next five years? (Please give a specific, realistic estimate)
With how many different partners have you had sex on one and only one occasion?
How often do (did) you fantasize about having sex with someone other than your current (most recent) dating partner? (1 never … 8 at least once a day)
“Sex without love is OK” (1 strongly disagree … 9 stronly agree)
“I can imagine myself being comfortable and enjoying `casual’ sex with different partners (1 strongly disagree … 9 stronly agree)
“I would have to be closely attached to someone (both emotionally and psychologially) before I could feel comfortable and fully enjoy having sex with him or her” (1 strongly disagree … 9 stronly agree)
Score is: 5 x item1 + 1 x item2 (capped at 30) + 5 x item3 + 4 x item4 + 2 x (mean of item5, item6, and reversed item7)
Do you see the problem already?
Quite predictably, researchers report that men have much higher SOI scores than women in all countries. But the first three questions (ignoring non-1:1 gender ratios, differently biased sampling for different genders, different rates of homosexuality between genders, different behaviour of homosexuals of different genders, 30 partners cap on the second item, differently biased forecasts of the second item and other small details that won’t affect the score much) - simply have to be identical for men and women, so the entire difference would have to be explained by items 4 to 7, which have relatively low weights!
The differences between men and women can be really extreme for some countries, Ukraine has 50.79±28.92 (mean±sd) for men, and 17.36±8.65 for women, which means that either Ukrainian men, or Ukrainian women, or both, are notoriously lying when asked about past and future sex partners. In most countries the differences are more moderate, with total 48-country sample’s scores being 46.67±29.68 for men, and 27.34±19.55 for women. Latvia leads the way with smallest difference, and so most likely greatest honesty, with 49.42±23.61 for men, and 41.68±26.68 for women, what can be plausibly explained by just differences in attitude. (fake lie detector experiments have shown it’s almost exclusively women who are lying when answering questions like that)
SOI seems to be considered quite useful by psychologists, it correlates with many nice things, not only other questionnaires, but country SOI averages correlate with various demographic, economic, and health scores in quite systematic way. Still, I cannot read papers about it without asking myself—why didn’t they bother to perform this basic sanity check—which would detect huge number of outright lies in answers. And more importantly—what proportion of “serious science” suffers from problems like that?
References: The 48 country SOI study, fake lie detectors shows which gender lies more.
Item 4 is the killer. It is actually heavily weighted in the paper you link, though not in all versions of the SOI. You can score a 50 without ever having had sex or ever expecting to have sex, and up to 32 of those points can come from a response to 4. 4 is obviously not zero sum and need not average out.
There is very good reason to believe that men fantasize about sex more often than women (who buys most pornography?), and there is also reason to believe men are more likely to fantasize about persons other than their current partner. It’s also possible that “once per day” is actually relatively infrequent given how often men think about sex. Put these together and it can explain a lot of the variance. The paper does not show breakdown by answers, at least not that I saw.
It’s also worth noting they had some significant response-rate issues (RR < .5 for some demographics). If women who like casual sex, are promiscuous, or are prostitutes are less likely to respond than all other groups, that could explain the variance.
Also, there are other versions of the ROI that don’t need to average out, as they lump large numbers together (i.e. “More than 8”).
I don’t see that as obvious. The obvious counter-argument is “who buys most romance novels?”, which are basically a different medium for porn.
I would love to see breakdown by question, if anyone can find a paper with that.
There’s a social norm supporting female “purity” that’s still lingering after a couple hundred years. There is a lot of guilt (particularly of the religious variety) directed towards sexuality, and particularly female sexuality since, (A) the church was run and funded by men and (B) women can get pregnant. Conversely, standards of masculinity developed for men that were rather explicitly pro-sexual, starting at least in the early 1900′s and remaining quite prevalent today.
Society continues to have very different standards for appropriate sexual behaviour for men and women. The anecdotal and cultural evidence (e.g. there’s no female James Bond, at least not in terms of promiscuity) that supports substantially different cultural pressures is overwhelming and too extensive to enumerate here. It’s my understanding that a lot of women (obviously not all, but probably a much larger proportion than men) may have fantasies that are more romantic than actively sexual as a result of developing their sexuality under existing social pressure.
Also, if you believe love is necessary to be comfortable with sex, sexual fantasies about non-partners seem much less likely, and based on substantial anecdotal experience, as well as rather clear cultural norms, men are less likely to believe love is a prerequisite for sex.
There are also plenty of ev-psych reasons why it would be more adaptive for men to be deeply interested in sex and women to be less interested in it / more interested in romance. I don’t put much weight in this view, given the complex history of how men and women’s sexuality has been viewed and expressed, but there it is.
I certainly agree with the usefulness of a breakdown by question, though.
Psychohistorian nails it. Items 4-7 could plausibly account for the wide gender differences. And even if they didn’t completely, then we still have the response rate issue, and differences in optimism in 2.
You never say what the problems are. So what if the subjects lie? Maybe “orientation” as measured by what people want to project is more useful than their actual behavior.
This reminds me of Shalizi’s complaints about IQ (not his complaints about intergroup differences that Johnicholas linked). One difference is that for SOI there’s a definite number (actual turnover of sex partners) that could be confused with it. But the name seems designed to discourage that confusion.
These are real problems, but it’s not obvious that the “basic sanity checks” you suggest would lead to better measures.
The problem is that aggregate SOI must be very similar for men as a group and for women as a group, in particular answers to questions 1-3 must be extremely close (as they measure behaviour, not orientation, and there’s one man and one woman in every sex pairing of the past year, every sex pairing of the next five years, and every one night stand, other than some tiny effects). The data makes it extremely obvious that they’re not, so there’s spectacular amount of systemic lying going on.
Doing anything with answers to questions you know people systemically lie about, you need to ask yourself what are you really measuring.
Yeah, there could be some lying going on (though there doesn’t have to be a “spectacular” amount; see Psychohistorian’s response).
However, just because people tend to lie about a certain behavior, it doesn’t make it useless to try to measure it. Rather than just giving up, psychologists often employ measures that will detect deceptiveness or social desirability bias such as the Marlowe-Crowne scale.
True. But at least in this case, people who underreport on this scale probably have less of what it’s actually trying to measure than people with the same behavior who report accurately. Since the SOI is about orientation, then how forthcoming and proud you are of the behavior it measures could be seen as part of that orientation.
For one amount of lying changes drastically depending on tiny details of how the test is administered. If you know about widespread lying is, and want to include it, you need to standardize testing conditions.
Those are good points. They definitely could produce better measures.
Shalizi’s complaints are semi-valid, that if you throw a huge amount of somewhat correlated data at PCA, you will most likely get a small number of components, with one explaining most of the variance. And when you start removing data that doesn’t correlate highly enough (as obviously “testing something else”), the leading component will only seem statistically stronger.
I’m quite surprised but it mirrors very closely what I think about the Big Five personality traits—factors on their own don’t really prove anywhere as much as is commonly stated, and can as easily be statistical artifacts.
This criticism doesn’t mean that either IQ or big 5 are invalid, but it does mean that the case should be made for them independently of “they show up as big factors in PCA”. It seems to be so for IQ, and I’m not that terribly convinced it’s also true for the Big Five.
I am reminded of a great paper, which isn’t online:
Biddlecom, Ann E., and Laurie F. DeRose. “A Cross-National Study of Virgin Births.” Journal of Irreproducible Results 38, no. 3 (May/June 1993): 9-14.
It is now. It can be found reproduced within a 1000-page screed of crackpottery here. Go down to page 211 (as reported in the page header) or 168 (as labelled within the document).
Amusing paper. For those uninterested in downloading 100M of crackpottery (as indeed it is), I’ve cut out the relevant bit as a standalone PDF: https://dl.dropboxusercontent.com/u/85192141/1993-biddlecom.pdf
Money quote:
I agree with the others who have said that there doesn’t actually seem to be a problem here.
OK, so people lie. That’s only a problem if the people who designed the test intended their measure to be based on actual number of partners, but I don’t see what reason there is to think that. Someone who is more willing to admit to having many partners, or who thinks having many partners is something to aspire to, is most likely more “open” to the idea of sex-without-commitment than someone who thinks having many partners is something to be ashamed of.
Indeed. Does it actually matter if responses don’t reflect actual sexual behavior?
Psychologists have noticed that there are sex differences on questions like 1-3, even though there cannot be sex differences in those behaviors, and have tried to explain that finding. See, for instance, this paper (pdf) by Baumeister, Catanese, & Vohs (2001) which argues that men have a stronger sex drive than women. They write (p. 250):
They also note a study showing that men were more likely than women to classify oral or manual stimulation as sex.
I expect they did. But there is little benefit to them is there in emphasising that particular aspect of their findings.
If people have biased expectations of the future amount of sex they’ll have, male/female answers for question 2 could differ.
I mentioned “differently biased forecasts of the second item”. This is one of many ways you can have noise, but the question was about realistic expectation, not how many they’d like to have, so if you’re going too far away from even answer you’re either dishonest with the researchers, or with yourself.
It’s not at all obvious that any of the small factors mentioned should bias the test to give men higher scores, other than outright lying.
Recall that in the planning fallacy, there was no difference between the “realistic” forecast and the “best case” forecast. This form of self-deception extends beyond the area of sexuality; what question 2 actually measures is a person’s vision of their next five years of sexual relationships, and how many and how enduring they’d want them to be. (The “realistic” disclaimer is commented out in conscious human forecasting.)
And, given the cultural incentives and evolutionary psychology, why would it be surprising for males’ imagined “best case” to skew much higher than their actual future, and females’ imagined “best case” to skew much lower?
There’s another problem with this scale—which I think is an actual problem, unlike the one taw complained about. The name of the scale suggests it’s meant to measure attitudes but 1 and 2 also measure opportunity. Someone might be very open in principle to sex without love or commitment, but find that no one (or no one they find sufficiently appealing) wants to have sex with them.
(Of course it is possible to buy sex for money, but there are reasons why someone might not want to do that that have nothing to do with “sociosexual openness”.)
This is called ‘face validity’. It’s why people get upset at things like this, but not at neologisms like ‘philanthrosexual’.
There is rather good debate on the topic of counting sexual partners and the reliability of survey responses here: http://www.nybooks.com/articles/1889
I’m afraid some of the material may be gated.
Volume 42, Number 9 · May 25, 1995 ‘Sex, Lies, and Social Science’: An Exchange By John H. Gagnon, Edward O. Laumann, Robert T. Michael, Stuart Michaels, Richard Sennett, Reply by Richard C. Lewontin
In response to Sex, Lies, and Social Science* (April 20, 1995)
I thought you were going to say that the problem was that men would be more optimistic.
It’s not the difference in optimism that matters; it’s the difference in what counts as optimal that matters.
And I’m pretty sure men and women differ on the optimal number of sex parters over the next five years...
And what would be the basis of it? They do NOT differ on the actual number of sex partners. So mens’ perfect number would be let’s say 10 partners, while womens’ 5 they would admit to having + 5 they would rather not?
My optimal number of gallons of ice cream to eat over the next five years is ten gallons.
In the next five years, I will most likely eat more than ten gallons of ice cream. Why?
Got an answer? Now, map it to the present problem.
Also, one thing to note that can mess up the “male sexual partners must equal female sexual partners” is the occurence of rare extreme female outliers. That is, a small number of women with far more sexual partners than average (usually but not necessarily prostitutes) who are unlikely to be found by these survey takers.
We can make up explanations like prostitutes and whatnot, but we have pretty good evidence that it’s simply women lying.
I’ve lost it, but I once ran across a paper that surveyed prostitutes and concluded they explained the entire difference. Most anomalies in social science that people are willing to admit exist in the data are explained many times over by lone papers. Sometimes even by whole bodies of literature that fail to acknowledge the other school of thought.
(ETA: here is a paper along these lines. here is a popular account. This paper is really not about under-sampling prostitutes, but about how prostitutes massively underreport.)
We also have pretty solid evidence for the existence of prostitutes and researchers’ failure to find them.
We also have pretty solid evidence for the existence of pick up artists. The linked SOI paper blatantly threw away top 1% of men with highest scores without any explanation.
I don’t know how much of this (link to NSFW site) is made up, but she claims to have had 152 male sexual partners and 23 female sexual partners from 1999 to 2007.
I seriously doubt these due to incest, but such numbers are perfectly plausible for either a man or a woman.
Plausible for a given individual, but quite an outlier from the average man or woman, I’d expect. Even so, though, how many of her partners also had 10-20 partners per year? If few of them, this survey, if she and all her partners filled it out, would skew heavily female in question one. I don’t think we can rule out the idea that question one is likely to produce very different results for male and female, even if everyone answered truthfully. It’s not immediately implausible, at least, as you imply.
Differences in question one are not only implausible, they’re impossible.
If such extreme female outliers really existed, at least some surveys would show number of partners for women being massively higher than for men, just by including one such person by chance. There’s no statistical way around it, unless you postulate 1 in 100,000 female outliers with millions of distinct male partners each, what’s quite ridiculous.
And we have pretty good evidence that women lie anyway.
The survey methodology could systematically fail to include outliers. That is the prostitute theory of the discrepancy, not just that there are female outliers, but that there is a reason that they aren’t sampled. “There’s no statistical way around it” if the samples are not biased.
...
I just showed how it isn’t impossible. Further, while you apparently assume that I meant only that women could have more partners on average than men, I didn’t imply that the actual survey would skew either way (on average), only that we have an example of where it would skew female.
I think it might actually be the case, in at least some societies, that fewer men than women have sex at all, on average, due to men who can have sex at all having sex with several or many women each. That’s an example of how it could easily skew male on question one, and I don’t really get how you believe that it’s impossible to have a higher population of one sex than the other that doesn’t have sex at all. It seems likely that there would be at least some difference.
But don’t surveys generally throw out outlier responses?
Define outlier. I have thrown out, or capped, data that is logically impossible, but I’ve also knowingly, and unknowingly included outliers I was skeptical of because there is no widely agreed upon standard for treating them. I could try many different standards/assumptions, and see if they affect my analysis but this type of work is time consuming and would not be of less interest to my colleagues than the work I would have to sacrifice. I realize this is sometimes problematic and I will make some effort to shift the standards in my discipline, sociology.
It would be a stupid thing to do. In any case that 48-nation survey mentioned throwing away highest 1% of male responses, as supposed outliers, and nothing about throwing away any female responses, which is extremely incompatible with “female outliers” theory.