I think the common meaning of the phrase “research shows X” is “research shows X is true”.
If that’s not what you want to argue you can use phrases like “research suggests X”.
If the thing you want to teach is evidence based reasoning it would be still useful to explain a naive audience the strength of the evidence.
A post either has mistakes or it doesn’t. The point of steelmanning is to change someone’s argument to make it better. I’m not denying that it’s possible to steelman the claim that polyamory increases happiness or openness but that’s not what I care about here. The thing that matter is whether or not this post encourages critical thinking. Using “show” when you mean “suggest” doesn’t help for that goal.
I agree that the common meaning of the phrase “research shows X” is “research shows X is true.” However, can you clarify to me where you believe the post makes mistakes in its use of “show” vs. “suggest”?
The statement I made in the salient blog post on the Intentional Insights website was as follows “research on poly relationships show that people with consensual non-monogamous relationships are happier.” I believe that the evidence I cited there supports the notion that “research shows X is true” when X = “on poly relationships shows that people with consensual non-monogamous relationships are happier.” For examples of such evidence, see this article, or this book or this article.
If I did make a mistake, as you suggest, I’d be glad to update my beliefs—I love to be convinced to do so :-) Indeed, I acknowledged in an earlier comment that the post went a little too strongly into using cognitive ease strategies to make its points. So we’ll work to tone down the cognitive ease strategies in the future, and thank you for being one of the people to help update my beliefs incrementally.
My core issue isn’t about the availability of research but about the research that you cited. If you want to teach evidence based reasoning than you have to give people good sources. You don’t want to teach people to treat media articles titled 12 surprising facts about X as real evidence. It would raise the sanity waterline to teach people that such articles don’t constitute evidence.
You cited a link citing the first article. From Ioannidis we know that in general observational studies are pretty unreliable.
This article isn’t even a good observational studies. The gave their questionnaire only to Swingers and then compare those Swingers with external values from the literature. There no reason to believe that’s a good idea for a value like self reported happiness.
As far as the second article goes. First as far as I see you didn’t cite it. If you think that it’s an important article that someone should use for forming an evidence based view on the subject, you should cite it.
Secondly it says: “However, inconsistent with our predictions, anxiety was unrelated to current relationship
status”. I don’t see how that translates into them being more happy.
As far as the book goes, it doesn’t seem to be a peer reviewed study. Even when it might reference them.
Can you clarify why you believe that such research is unacceptable as a form of Level III evidence? After all, some evidence is better than no evidence for actually changing one’s mind. Polyamory research is a very new field, and descriptive studies are pretty much the best there is at this stage. But if you find other relevant research on polyamory that we missed, I would be happy to update my beliefs.
As that comment illuminates, I used the kind of evidence available currently.
For the second article, it states “a sizeable minority of people engage in CNM and report high levels of satisfaction.” CNM is the common acronym for consensual non-monogamous relationships. To me, “high levels of satisfaction” equates to being “happy.” However, I accept that we might interpret the term “high levels of satisfaction” differently—to me, it equates to being “happy,” but it might not to you. I think that’s primarily a semantic issue, though, and would rather not pursue it.
I am confused by your claims that the book is not a peer reviewed study. It is published by Rowman & Littlefield, a well-known and well-respected publisher, which has a solid peer review process. I would appreciate your clarification on what you meant when you suggested it was not.
The link you give for Level III evidence means that it’s “controlled trial without randomization”. Controlling against a literature value isn’t a controlled trial.
Just given your questionaire to swingers that want to take a questionaire about swingers and presumably give the world a good impression about what it means to be a swinger to some average census value or literature value doesn’t work.
The feminist paper says that it uses a qualitative approach. In the hierarchy that you linked that’s Level VI.
As that comment illuminates, I used the kind of evidence available currently.
To the extend that currently there isn’t strong evidence available, you shouldn’t use definitive language like “show” but suggest to illustrate that the evidence isn’t strong.
For the second article, it states “a sizeable minority of people engage in CNM and report high levels of satisfaction.”
When reading that your first reaction should be “How do they know?” In this case this is part of their discussion of previous work and a result that comes out of the data they gathered and that they discuss in the methods section.
The first paper I looked at from that pile contains lines like “A social constructionist discourse analytic approach was taken to the data.”, so again qualitative. The second is also again qualitative. If it’s multiple qualitative studies that’s in your linked scheme level V.
“Sizable minority” is also a term that doesn’t tell you whether the average member of the population is better or worth then average.
I am confused by your claims that the book is not a peer reviewed study.
It’s a book and no study.Books are generally not primary sources.
Regarding the book, I want to draw your attention to the content of my comment above:
I am confused by your claims that the book is not a peer reviewed study. It is published by Rowman & Littlefield, a well-known and well-respected publisher, which has a solid peer review process. I would appreciate your clarification on what you meant when you suggested it was not.
Can you please point out to me where we disagree regarding the book being a peer-reviewed study? That was the issue you raised, and that was what I responded to. So please let me know what your thoughts are about this matter.
Books often help to understand a subject matter in more depth. On the other hand they are usually no primary sources and therefore not that good for demonstrating the evidence for particular claims.
Has primary sources and is a primary source are two different things. When determining a specific claim such as is there research that proves that polyamorous people are happy than I care about the specific set up of a study to evaluate the evidence.
In general in academia everybody who runs a decent study wants to have the study in a citable paper and that paper is usually the best source for understanding what was done. In addition to his book Kahneman published a variety of papers.
I don’t have an issue with the sentence: “A major 15-year ethnographic research project showed the richness and diversity of poly families, within which individuals form relationships with a wide variety of partners and enjoy emotional and sexual freedom. ”
Showing richness and diversity is something that a qualitative ethnographic research project can do.
I have expressed confusion above over your claim regarding books, where you stated that they are “not that good for demonstrating the evidence for particular claims.” Can you please respond to that specific request for clarification?
You made an argument that books are not helpful for demonstrating claims. First, you stated that the book I cited was not peer reviewed. I pointed out that the book was actually peer reviewed. Would you please respond to my statement and avoid making further claims until you actually respond to my statement?
I pointed out a number of books, including my book and Daniel Kahneman’s book, that are based on primary sources. There is no difference, in fact, between a peer-reviewed academic article and a book—they are both peer-reviewed pieces, and are both based on primary source evidence. Do you disagree? If so, please explain your disagreement. After all, if you do not take books to be good evidence, that is a basic difference in our worldviews, and I have no interest in further engaging in this matter.
First, you stated that the book I cited was not peer reviewed.
I said “Peer reviewed study”. That phrase not only contains “peer reviewed” but also study. In general high quality studies get published in papers. A good scientific textbook refers to a bunch of peer reviewed studies but it usually isn’t the primary source.
In this case it might be that some gender study folks have different idea of how to do science. After all they want to do science in a feminist framework. That might result in a study really being published in a book.
I think most people on LW do have alarm bells that ring if a gender study folks say they don’t like the way science is done traditionally and they rather want to follow a feminist framework.
I pointed out a number of books, including my book and Daniel Kahneman’s book, that are based on primary sources.
A book that is based on primary sources is a secondary source, it’s not itself a primary source. A much better secondary source than a click-bait mainstream media article but still a secondary source.
I think the common meaning of the phrase “research shows X” is “research shows X is true”. If that’s not what you want to argue you can use phrases like “research suggests X”.
If the thing you want to teach is evidence based reasoning it would be still useful to explain a naive audience the strength of the evidence.
A post either has mistakes or it doesn’t. The point of steelmanning is to change someone’s argument to make it better. I’m not denying that it’s possible to steelman the claim that polyamory increases happiness or openness but that’s not what I care about here. The thing that matter is whether or not this post encourages critical thinking. Using “show” when you mean “suggest” doesn’t help for that goal.
I agree that the common meaning of the phrase “research shows X” is “research shows X is true.” However, can you clarify to me where you believe the post makes mistakes in its use of “show” vs. “suggest”?
The statement I made in the salient blog post on the Intentional Insights website was as follows “research on poly relationships show that people with consensual non-monogamous relationships are happier.” I believe that the evidence I cited there supports the notion that “research shows X is true” when X = “on poly relationships shows that people with consensual non-monogamous relationships are happier.” For examples of such evidence, see this article, or this book or this article.
If I did make a mistake, as you suggest, I’d be glad to update my beliefs—I love to be convinced to do so :-) Indeed, I acknowledged in an earlier comment that the post went a little too strongly into using cognitive ease strategies to make its points. So we’ll work to tone down the cognitive ease strategies in the future, and thank you for being one of the people to help update my beliefs incrementally.
My core issue isn’t about the availability of research but about the research that you cited. If you want to teach evidence based reasoning than you have to give people good sources. You don’t want to teach people to treat media articles titled 12 surprising facts about X as real evidence. It would raise the sanity waterline to teach people that such articles don’t constitute evidence.
You cited a link citing the first article. From Ioannidis we know that in general observational studies are pretty unreliable. This article isn’t even a good observational studies. The gave their questionnaire only to Swingers and then compare those Swingers with external values from the literature. There no reason to believe that’s a good idea for a value like self reported happiness.
As far as the second article goes. First as far as I see you didn’t cite it. If you think that it’s an important article that someone should use for forming an evidence based view on the subject, you should cite it.
Secondly it says: “However, inconsistent with our predictions, anxiety was unrelated to current relationship status”. I don’t see how that translates into them being more happy.
As far as the book goes, it doesn’t seem to be a peer reviewed study. Even when it might reference them.
It’s great to see we are both committed to using evidence, and the debate is now focusing on the nature of the evidence.
I believe I had earlier stated the following in my comment above
As that comment illuminates, I used the kind of evidence available currently.
For the second article, it states “a sizeable minority of people engage in CNM and report high levels of satisfaction.” CNM is the common acronym for consensual non-monogamous relationships. To me, “high levels of satisfaction” equates to being “happy.” However, I accept that we might interpret the term “high levels of satisfaction” differently—to me, it equates to being “happy,” but it might not to you. I think that’s primarily a semantic issue, though, and would rather not pursue it.
I am confused by your claims that the book is not a peer reviewed study. It is published by Rowman & Littlefield, a well-known and well-respected publisher, which has a solid peer review process. I would appreciate your clarification on what you meant when you suggested it was not.
The link you give for Level III evidence means that it’s “controlled trial without randomization”. Controlling against a literature value isn’t a controlled trial.
Just given your questionaire to swingers that want to take a questionaire about swingers and presumably give the world a good impression about what it means to be a swinger to some average census value or literature value doesn’t work.
The feminist paper says that it uses a qualitative approach. In the hierarchy that you linked that’s Level VI.
To the extend that currently there isn’t strong evidence available, you shouldn’t use definitive language like “show” but suggest to illustrate that the evidence isn’t strong.
When reading that your first reaction should be “How do they know?” In this case this is part of their discussion of previous work and a result that comes out of the data they gathered and that they discuss in the methods section. The first paper I looked at from that pile contains lines like “A social constructionist discourse analytic approach was taken to the data.”, so again qualitative. The second is also again qualitative. If it’s multiple qualitative studies that’s in your linked scheme level V.
“Sizable minority” is also a term that doesn’t tell you whether the average member of the population is better or worth then average.
It’s a book and no study.Books are generally not primary sources.
Regarding the book, I want to draw your attention to the content of my comment above:
Can you please point out to me where we disagree regarding the book being a peer-reviewed study? That was the issue you raised, and that was what I responded to. So please let me know what your thoughts are about this matter.
Books often help to understand a subject matter in more depth. On the other hand they are usually no primary sources and therefore not that good for demonstrating the evidence for particular claims.
I notice I’m confused over your claim about books and primary sources.
My brief monograph, for example, is flush with primary sources. The specific book I cited earlier, the one from the Intentional Insights blog post on polyamory, is a 15-year ethnographic study, and is thus quite full of primary sources. Daniel Kahneman’s Thinking, Fast and Slow is also a book, and has plenty of primary sources.
So can you please clarify to me your comments about books? Appreciate it :-)
Has primary sources and is a primary source are two different things. When determining a specific claim such as is there research that proves that polyamorous people are happy than I care about the specific set up of a study to evaluate the evidence.
In general in academia everybody who runs a decent study wants to have the study in a citable paper and that paper is usually the best source for understanding what was done. In addition to his book Kahneman published a variety of papers.
I don’t have an issue with the sentence: “A major 15-year ethnographic research project showed the richness and diversity of poly families, within which individuals form relationships with a wide variety of partners and enjoy emotional and sexual freedom. ”
Showing richness and diversity is something that a qualitative ethnographic research project can do.
I think especially the claim about polyamory people being happier is not established. I opened a question on skeptics stackexchange about that claim. http://skeptics.stackexchange.com/questions/23807/are-polyamorous-people-happier
I have expressed confusion above over your claim regarding books, where you stated that they are “not that good for demonstrating the evidence for particular claims.” Can you please respond to that specific request for clarification?
You made an argument that books are not helpful for demonstrating claims. First, you stated that the book I cited was not peer reviewed. I pointed out that the book was actually peer reviewed. Would you please respond to my statement and avoid making further claims until you actually respond to my statement?
I pointed out a number of books, including my book and Daniel Kahneman’s book, that are based on primary sources. There is no difference, in fact, between a peer-reviewed academic article and a book—they are both peer-reviewed pieces, and are both based on primary source evidence. Do you disagree? If so, please explain your disagreement. After all, if you do not take books to be good evidence, that is a basic difference in our worldviews, and I have no interest in further engaging in this matter.
I said “Peer reviewed study”. That phrase not only contains “peer reviewed” but also study. In general high quality studies get published in papers. A good scientific textbook refers to a bunch of peer reviewed studies but it usually isn’t the primary source.
In this case it might be that some gender study folks have different idea of how to do science. After all they want to do science in a feminist framework. That might result in a study really being published in a book.
I think most people on LW do have alarm bells that ring if a gender study folks say they don’t like the way science is done traditionally and they rather want to follow a feminist framework.
A book that is based on primary sources is a secondary source, it’s not itself a primary source. A much better secondary source than a click-bait mainstream media article but still a secondary source.