However, I actively avoid or give little consideration to other people’s self-experiments, because I think that we intuitively give stories way more credit than they deserve (shameless plug for my blog). For the same reason I don’t write about my own self-experiments. They may persuade some people, but even the smallest controlled study achieves much better evidence.
I think we better focus on empirical data or at least a good theory (like exploiting cognitive biases) instead.
However, nearly all productivity advice is anecdotal. My guess is that most of the tips work as a placebo or don’t do anything at all.
Luke Muehlhauser wrote a great post on self-help in general. Maybe we can extend the section on productivity and look on which part of the advice is actually supported by evidence. Maybe we can also rank it by confidence.
Sounds like a fun project to do some digging to produce a literature review. Is that something you’d be interested in doing? I’d definitely love to see the final product!
Awesome. Let me know if there’s anything I can do to support you—if you want to talk methodology or anything, or if I’d just be a useful commitment device.
Academic journals not only publish studies with multiple participants. From time to time they also publish case studies.
Case studies are useful for understanding an intervention.
With productive advice it’s likely that someone reads a “productivity tip” and then tries to implement it but does something different than intended. Stories help illustrate what we are actually talking about in a way that’s easier to understand.
I agree that there is some value in stories. They are a tool to communicate the idea of more reliable information sources to a greater public.
The problem is that we trust stories based on their coherence and the coherence of a story has little to no correlation with its validity. Or: “Everyone can tell a great story, but a great story is not necessarily true”.
The other problem is that we trust personal anecdotes way too much.
We say Ben Franklin/Steve Jobs/Warren Buffet does X and he is successful. Therefore, if I want to be successful I have to do X. That is a formal fallacy.
A better approach is to say “Millions of successful people do X, therefore I should do X”. This is better than anecdotal evidence, but still only a correlation. May successful people buy expensive cars. Does buying an expensive car make you successful? Of course not. It’s only the result of having more money.
The best kind of evidence comes from controlled studies. Set up a group with people that use time-tracking tools and one without. See who gets more stuff done. Try to have at least a 95% confidence level. Report that in a journal. Done.
Sorry if my explanation is overly long and obvious to many of you (I don’t know, is it?).
Does the average person who reads a story about productivity tips automatically implement the tip because the trust it? Usually not. Usually people just read and then do nothing.
Don’t underrate the effect of telling stories in a way that get people to change their behavior.
I think it’s likely that my main challenges don’t lie with having the right evidence about which productivity technique is best but with akrasia around implementing techniques.
I am also a big fan of self-experiments.
However, I actively avoid or give little consideration to other people’s self-experiments, because I think that we intuitively give stories way more credit than they deserve (shameless plug for my blog). For the same reason I don’t write about my own self-experiments. They may persuade some people, but even the smallest controlled study achieves much better evidence.
I think we better focus on empirical data or at least a good theory (like exploiting cognitive biases) instead.
However, nearly all productivity advice is anecdotal. My guess is that most of the tips work as a placebo or don’t do anything at all.
Luke Muehlhauser wrote a great post on self-help in general. Maybe we can extend the section on productivity and look on which part of the advice is actually supported by evidence. Maybe we can also rank it by confidence.
Best regards.
Sounds like a fun project to do some digging to produce a literature review. Is that something you’d be interested in doing? I’d definitely love to see the final product!
Sounds like a good idea.
It may take some time. I’ll let you know once I am finished.
Awesome. Let me know if there’s anything I can do to support you—if you want to talk methodology or anything, or if I’d just be a useful commitment device.
Academic journals not only publish studies with multiple participants. From time to time they also publish case studies. Case studies are useful for understanding an intervention.
With productive advice it’s likely that someone reads a “productivity tip” and then tries to implement it but does something different than intended. Stories help illustrate what we are actually talking about in a way that’s easier to understand.
I agree that there is some value in stories. They are a tool to communicate the idea of more reliable information sources to a greater public.
The problem is that we trust stories based on their coherence and the coherence of a story has little to no correlation with its validity. Or: “Everyone can tell a great story, but a great story is not necessarily true”.
The other problem is that we trust personal anecdotes way too much.
We say Ben Franklin/Steve Jobs/Warren Buffet does X and he is successful. Therefore, if I want to be successful I have to do X. That is a formal fallacy.
A better approach is to say “Millions of successful people do X, therefore I should do X”. This is better than anecdotal evidence, but still only a correlation. May successful people buy expensive cars. Does buying an expensive car make you successful? Of course not. It’s only the result of having more money.
The best kind of evidence comes from controlled studies. Set up a group with people that use time-tracking tools and one without. See who gets more stuff done. Try to have at least a 95% confidence level. Report that in a journal. Done.
Sorry if my explanation is overly long and obvious to many of you (I don’t know, is it?).
What do you actually mean with “trust”?
Does the average person who reads a story about productivity tips automatically implement the tip because the trust it? Usually not. Usually people just read and then do nothing.
Don’t underrate the effect of telling stories in a way that get people to change their behavior.
I think it’s likely that my main challenges don’t lie with having the right evidence about which productivity technique is best but with akrasia around implementing techniques.
With trusting stories I meant something like “we assign them a higher probability to be true than they actually have”.
Something along Kahneman’s theory that we default to believe everything we hear. That disbelieving is harder (a System 2 activity).
But akrasia is also a good point! I agree that many people already have all the information they need and just need to take action.