So Vassar was right, we have reached a crisis. A self-help sales pitch with allegations of first-percentile utility right here on LW. This gets my downvote on good old Popperian grounds.
You say this stuff helps with akrasia? However hot your enthusiasm burns, you don’t get to skip the “controlled study” part. Come back with citations. At this point you haven’t even ruled out the placebo effect, for Bayes’ sake!
However hot your enthusiasm burns, you don’t get to skip the “controlled study” part.
While I agree with some of what you’re saying, it isn’t like “cached thoughts” or many of Eliezer’s other classics come with references to controlled studies, either. Like Robin Hanson pointed out in response to my own critique of evpsych:
claims can be “tested” via almost any connection they make with other claims that connect etc. to things we see. This is what intellectual exploration looks like.
No, Eby’s article didn’t have direct references to empirical work establishing the connection between PCT and akrasia, but it did build on enough existing work about PCT to make the connection plausible and easy to believe. If this were a peer-reviewed academic journal, that wouldn’t be enough, and it’d have to be backed with experimental work. But I see no reason to require LW posts to adhere to the same standard as an academic journal—this is also a place to simply toss out interesting and plausible-seeming ideas, so that they can be discussed and examined and somebody can develop them further, up to the point of gathering that experimental evidence.
You say this stuff helps with akrasia? However hot your enthusiasm burns, you don’t get to skip the “controlled study” part. Come back with citations. At this point you haven’t even ruled out the placebo effect, for Bayes’ sake!
The term “placebo effect” was coined to refer to phsychological effects intruding on non-psychological studies. In this case, since the desired effect is purely psychological, it’s meaningless at best and misleading at worst. There is no self-help advice equivalent to a sugar pill. The closest thing to a sugar pill available is known-bad advice, and giving known-bad advice to a control group strikes me as decidedly unethical.
So, if you have an experimental procedure, go ahead and suggest it. Absent that, the only available data comes from self-experimentation and anecdotes.
What if you’re wrong? What if the most effective anti-procrastination technique is tickling your left foot in exactly the right manner, and this works regardless of whether you believe in its efficacy, or even know about it? That (predicated on a correct theory of human motivation) is the kind of stuff we’re looking for.
There is no self-help advice equivalent to a sugar pill. The closest thing to a sugar pill available is known-bad advice, and giving known-bad advice to a control group strikes me as decidedly unethical.
You’re saying that there’s no neutral (non-positive and non-negative) self-help advice? That’s a pretty weird statement to make. Some advice is good, some is bad; why do you suspect a gap at zero? Failing all else, you could refrain from telling the subjects that the study is about self-control and anti-procrastination, just tell them to blindly follow some instructions and measure the effects covertly.
No, I have no experimental protocol ready yet, but have the impudence to insist that we as a community should create one or shut up.
That (predicated on a correct theory of human motivation) is the kind of stuff we’re looking for.
You don’t know what “we” are looking for. There is no one thing “we” are looking for. Some of us may be interested in plausible, attested-to self-help methods, even without experimental support.
Some of us may be interested in plausible, attested-to self-help methods, even without experimental support.
Without experimental support is fine. But without extraordinary support isn’t. Something must make the plausibility of a particular thing stand out, because you can’t be interested in all the 1000 of equally plausible things unless you devote all your time to that.
No, I have no experimental protocol ready yet, but have the impudence to insist that we as a community should create one or shut up.
I certainly agree with the ‘create one’ part of what you’re saying. Not so much the ‘shut up’. Talking about the topic (and in so doing dragging all sorts of relevant knowledge from the community) and also self experimenting has its use. Particularly in as much as it can tell us whether something is worth testing.
I do note that there are an awful lot of posts here (and on Overcoming Bias) which do not actually have controlled studies backing them. Is there a reason why Kaj’s post requires a different standard to be acceptable? (And I ask that non-rhetorically, I can see reasons why you may reasonably do just that.)
The closest thing to a sugar pill available is known-bad advice, and giving known-bad advice to a control group strikes me as decidedly unethical.
It would seem ethically acceptable to give groups advice selected from common social norms. For example, give one group some “Getting Things Done”, another group nothing at all, a third some instruction on calculus (irrelevant but still high status attention and education), a fifth a drill sergeant motivational yelling at and the fourth group gets PJEby’s system.
The closest thing to a sugar pill available is known-bad advice,
One example of a control group in a psychological study (can’t find reference): researchers compared freudian psychoanalysis to merely sitting there and listening.
sugar has physiological effects, so you can’t really assume a sugar pill is neutral with no side-effects
However hot your enthusiasm burns, you don’t get to skip the “controlled study” part.
To what end do you not get to skip it? Others may legitimately have lower standards for something being interesting, as Kaj said, or for a technique being worth a try.
Honestly, it sounds more like you’re trying to take down Kaj for getting uppity and violating the norms of Science, than like you’re trying to contribute to finding truth or usefulness.
This gets my downvote on good old Popperian grounds. … you don’t get to skip the “controlled study” part. Come back with citations.
I’m afraid you have Popper all turned around. According to Popper, one should make claims that are testable, and then it’s the job of (usually other) scientists to perform experiments to try to tear them apart.
If you’re a Popperian and you disagree, go ahead and perform the experiment. If your position is that the relevant claim isn’t testable, that’s a different complaint entirely.
You’re supposed to try to tear apart your own claims, first. Making random but testable assertions for no particular reason is not part of the methodology.
Yes, I’m a Popperian. Yes, people should make testable claims and other people should test them. That’s how everything is supposed to work. All right so far.
As to the nature of my complaint… Here’s a non-trivial question: how do we rigorously test Kaj and Eby’s assertions about akrasia? I took Vassar’s words very seriously and have been trying to think up an experiment that would (at least) properly control for the belief effect, but came up empty so far. If I manage to solve this problem, I’ll make a toplevel post about that.
Why is it so difficult? Even a head to head test between PJ’s magic and an arbitrarily selected alternative would provide valuable information. Given the claims made for, as you pointed out, first percentile utility, it seems that just a couple of tests against arbitrary alternatives should be expected to show drastic differences and at least tell us whether it is worth thinking harder.
So Vassar was right, we have reached a crisis. A self-help sales pitch with allegations of first-percentile utility right here on LW. This gets my downvote on good old Popperian grounds.
You say this stuff helps with akrasia? However hot your enthusiasm burns, you don’t get to skip the “controlled study” part. Come back with citations. At this point you haven’t even ruled out the placebo effect, for Bayes’ sake!
While I agree with some of what you’re saying, it isn’t like “cached thoughts” or many of Eliezer’s other classics come with references to controlled studies, either. Like Robin Hanson pointed out in response to my own critique of evpsych:
No, Eby’s article didn’t have direct references to empirical work establishing the connection between PCT and akrasia, but it did build on enough existing work about PCT to make the connection plausible and easy to believe. If this were a peer-reviewed academic journal, that wouldn’t be enough, and it’d have to be backed with experimental work. But I see no reason to require LW posts to adhere to the same standard as an academic journal—this is also a place to simply toss out interesting and plausible-seeming ideas, so that they can be discussed and examined and somebody can develop them further, up to the point of gathering that experimental evidence.
The term “placebo effect” was coined to refer to phsychological effects intruding on non-psychological studies. In this case, since the desired effect is purely psychological, it’s meaningless at best and misleading at worst. There is no self-help advice equivalent to a sugar pill. The closest thing to a sugar pill available is known-bad advice, and giving known-bad advice to a control group strikes me as decidedly unethical.
So, if you have an experimental procedure, go ahead and suggest it. Absent that, the only available data comes from self-experimentation and anecdotes.
What if you’re wrong? What if the most effective anti-procrastination technique is tickling your left foot in exactly the right manner, and this works regardless of whether you believe in its efficacy, or even know about it? That (predicated on a correct theory of human motivation) is the kind of stuff we’re looking for.
You’re saying that there’s no neutral (non-positive and non-negative) self-help advice? That’s a pretty weird statement to make. Some advice is good, some is bad; why do you suspect a gap at zero? Failing all else, you could refrain from telling the subjects that the study is about self-control and anti-procrastination, just tell them to blindly follow some instructions and measure the effects covertly.
No, I have no experimental protocol ready yet, but have the impudence to insist that we as a community should create one or shut up.
You don’t know what “we” are looking for. There is no one thing “we” are looking for. Some of us may be interested in plausible, attested-to self-help methods, even without experimental support.
Without experimental support is fine. But without extraordinary support isn’t. Something must make the plausibility of a particular thing stand out, because you can’t be interested in all the 1000 of equally plausible things unless you devote all your time to that.
I certainly agree with the ‘create one’ part of what you’re saying. Not so much the ‘shut up’. Talking about the topic (and in so doing dragging all sorts of relevant knowledge from the community) and also self experimenting has its use. Particularly in as much as it can tell us whether something is worth testing.
I do note that there are an awful lot of posts here (and on Overcoming Bias) which do not actually have controlled studies backing them. Is there a reason why Kaj’s post requires a different standard to be acceptable? (And I ask that non-rhetorically, I can see reasons why you may reasonably do just that.)
It would seem ethically acceptable to give groups advice selected from common social norms. For example, give one group some “Getting Things Done”, another group nothing at all, a third some instruction on calculus (irrelevant but still high status attention and education), a fifth a drill sergeant motivational yelling at and the fourth group gets PJEby’s system.
One example of a control group in a psychological study (can’t find reference): researchers compared freudian psychoanalysis to merely sitting there and listening.
sugar has physiological effects, so you can’t really assume a sugar pill is neutral with no side-effects
And when you are testing the psychological effects of urea based salts you can’t really assume lithium salts are neutral with no side-effects.
Is it how the real studies view the situation?
To what end do you not get to skip it? Others may legitimately have lower standards for something being interesting, as Kaj said, or for a technique being worth a try.
Honestly, it sounds more like you’re trying to take down Kaj for getting uppity and violating the norms of Science, than like you’re trying to contribute to finding truth or usefulness.
I’m afraid you have Popper all turned around. According to Popper, one should make claims that are testable, and then it’s the job of (usually other) scientists to perform experiments to try to tear them apart.
If you’re a Popperian and you disagree, go ahead and perform the experiment. If your position is that the relevant claim isn’t testable, that’s a different complaint entirely.
You’re supposed to try to tear apart your own claims, first. Making random but testable assertions for no particular reason is not part of the methodology.
Yes, I’m a Popperian. Yes, people should make testable claims and other people should test them. That’s how everything is supposed to work. All right so far.
As to the nature of my complaint… Here’s a non-trivial question: how do we rigorously test Kaj and Eby’s assertions about akrasia? I took Vassar’s words very seriously and have been trying to think up an experiment that would (at least) properly control for the belief effect, but came up empty so far. If I manage to solve this problem, I’ll make a toplevel post about that.
Why is it so difficult? Even a head to head test between PJ’s magic and an arbitrarily selected alternative would provide valuable information. Given the claims made for, as you pointed out, first percentile utility, it seems that just a couple of tests against arbitrary alternatives should be expected to show drastic differences and at least tell us whether it is worth thinking harder.