Show me some peer-reviewed research (please, not clinical case studies).
This seems like an unreasonable thing to ask of a non-academic. Based on what I hear of academia, pjeby doesn’t have a good chance of obtaining funding for a controlled study nor of publishing his results in a respectable journal even if they are as good as he claims. Or am I wrong? It would be nice if I were incorrect on either of those things.
You are probably right. It was an overly onerous requirement on my part. However, peer-reviewed is our best stamp of quality research we have and a meta-analysis is even better, comprised of hundreds of peer-reviewed research. I am passionate about science, well aware of the limitations of clincial expert opinion, and was probably too strident.
In truth, it is almost impossible for a sole practitioner to discern whether the efficaciousness of their treatment is due to the treatment itself or other apparently non-relevant aspects, such as the placebo effect or the personality of the clinician. There are some really effective clinicians out there who are successful through their innate ability to inspire. You need to do or rely on research to determine what is really going on (i.e., evidence based treatment). There really isn’t any other way (really, really, really), and unless he gets this, there is nothing he will personally experience that will make him change his mind. This isn’t new though. Research has repeated shown statistical analysis beats clinical opinion pretty much everytime (here’s one from Paul Meehl, who I studied under and was both a clinician and statistican:
http://www.psych.umn.edu/faculty/grove/114meehlscontributiontoclinical.pdf).
This type of issue is never going go away though. We have everything from homeopathy to applied kinesiology, all of which where appears to work because people believe it works. The only way to separate out whether the motivational treatment is inherently effective is through research. If it is the placebo effect and you are happy with that being the source of whatever change you are seeing, then add a lot more pomp and ceremony—it ups the effect.
There are some really effective clinicians out there who are successful through their innate ability to inspire.
Heh. Doesn’t apply in my case, unless mere text on a screen qualifies as innate ability to inspire. (Most of my client work is done in text format, and I mostly try to teach people techniques which they can apply themselves.)
Really, if these clinicians are successful for this reason, then why isn’t there any research identifiying what this “innate ability” consists of, so that other clinicians can be taught to be inspiring, or conversely, there can be some sort of inspirational ability test made a qualification of licensing?
A phrase like “innate abiliity to inspire” is bad science and bad reductionism.
You need to do or rely on research to determine what is really going on (i.e., evidence based treatment).
Ah, that’s why auto mechanics have peer-reviewed journals in order to notice whether they can really fix cars, or just have an innate ability to inspire the cars. ;-)
Can a mechanic be wrong about why a car started working, or how it was broken? Absolutely. Does it matter to the mechanic? To the car’s owner? Not very much.
I wrote a response to your post above, but the site sits and spins for several minutes every time I submit it; I guess perhaps it’s too long. I referred back to various other postings on this site, so you could get an idea of how strict LessWrong’s standards of reductionism and word usage actually are, and showing why individual falsifiability is a higher standard than peer-reviewed research, if you want a car that starts.
The type of research-based advice you’re touting, doesn’t rise to the level of individual falsifiability, because you can still say “it’s proven science” even when it doesn’t work for that particular individual. I don’t have that retreat, because I only accept as “technique” processes which can be unequivocally stated as having worked or not worked for a particular application.
My longer post also detailed the likely areas where placebo effects could exist in my work, and described some of the difficulties in formulating an appropriate control placebo for same.
So, I do understand the difference between chemistry and auto mechanics, and I’m not claiming to be a good chemist, or that you’re a bad one. But I am saying that chemists who haven’t actually opened the hood and gotten their hands dirty might not be the best people to write owners’ manuals, even if they might be a good technical reviewer for such a manual.
Conversely, auto mechanics shouldn’t write chemistry textbooks, and I don’t have any delusions in that regard.
(Hopefully, this comment is short enough to actually get posted.)
This seems like an unreasonable thing to ask of a non-academic. Based on what I hear of academia, pjeby doesn’t have a good chance of obtaining funding for a controlled study nor of publishing his results in a respectable journal even if they are as good as he claims. Or am I wrong? It would be nice if I were incorrect on either of those things.
You are probably right. It was an overly onerous requirement on my part. However, peer-reviewed is our best stamp of quality research we have and a meta-analysis is even better, comprised of hundreds of peer-reviewed research. I am passionate about science, well aware of the limitations of clincial expert opinion, and was probably too strident.
In truth, it is almost impossible for a sole practitioner to discern whether the efficaciousness of their treatment is due to the treatment itself or other apparently non-relevant aspects, such as the placebo effect or the personality of the clinician. There are some really effective clinicians out there who are successful through their innate ability to inspire. You need to do or rely on research to determine what is really going on (i.e., evidence based treatment). There really isn’t any other way (really, really, really), and unless he gets this, there is nothing he will personally experience that will make him change his mind. This isn’t new though. Research has repeated shown statistical analysis beats clinical opinion pretty much everytime (here’s one from Paul Meehl, who I studied under and was both a clinician and statistican: http://www.psych.umn.edu/faculty/grove/114meehlscontributiontoclinical.pdf).
This type of issue is never going go away though. We have everything from homeopathy to applied kinesiology, all of which where appears to work because people believe it works. The only way to separate out whether the motivational treatment is inherently effective is through research. If it is the placebo effect and you are happy with that being the source of whatever change you are seeing, then add a lot more pomp and ceremony—it ups the effect.
Heh. Doesn’t apply in my case, unless mere text on a screen qualifies as innate ability to inspire. (Most of my client work is done in text format, and I mostly try to teach people techniques which they can apply themselves.)
Really, if these clinicians are successful for this reason, then why isn’t there any research identifiying what this “innate ability” consists of, so that other clinicians can be taught to be inspiring, or conversely, there can be some sort of inspirational ability test made a qualification of licensing?
A phrase like “innate abiliity to inspire” is bad science and bad reductionism.
Ah, that’s why auto mechanics have peer-reviewed journals in order to notice whether they can really fix cars, or just have an innate ability to inspire the cars. ;-)
Can a mechanic be wrong about why a car started working, or how it was broken? Absolutely. Does it matter to the mechanic? To the car’s owner? Not very much.
I wrote a response to your post above, but the site sits and spins for several minutes every time I submit it; I guess perhaps it’s too long. I referred back to various other postings on this site, so you could get an idea of how strict LessWrong’s standards of reductionism and word usage actually are, and showing why individual falsifiability is a higher standard than peer-reviewed research, if you want a car that starts.
The type of research-based advice you’re touting, doesn’t rise to the level of individual falsifiability, because you can still say “it’s proven science” even when it doesn’t work for that particular individual. I don’t have that retreat, because I only accept as “technique” processes which can be unequivocally stated as having worked or not worked for a particular application.
My longer post also detailed the likely areas where placebo effects could exist in my work, and described some of the difficulties in formulating an appropriate control placebo for same.
So, I do understand the difference between chemistry and auto mechanics, and I’m not claiming to be a good chemist, or that you’re a bad one. But I am saying that chemists who haven’t actually opened the hood and gotten their hands dirty might not be the best people to write owners’ manuals, even if they might be a good technical reviewer for such a manual.
Conversely, auto mechanics shouldn’t write chemistry textbooks, and I don’t have any delusions in that regard.
(Hopefully, this comment is short enough to actually get posted.)