From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit—you can plan out what is necessary, and it’s clear when you succeeded. A literature review isn’t really either. So I’m not sure a prize in general is succeeded for the task.
My personal problems are that:
I can’t bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large—I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it’s one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I’d feel really terrible!
The difficult questions are:
How much does spaced repetition actually help memory?
Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it’s not clear what is even meant by ‘how much’ - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?
There’s probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?
New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.
The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.
Great comment. Perhaps I should be reading the prize literature! I’ll think about how to structure this better.
On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.
Just to make sure I’m clear, I’m not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.
I actually hadn’t read your page before, so thanks. I actually think your page is a fairly good start.
Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of ‘prize’ and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.
It’s funny you linked to my post on prizes, which brought me here, because I’m actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it’s potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html
You guys need to get more rational about the value and importance of what you’re doing in this community. You’re arguing over trivial amounts of money here. I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
I’m putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn’t changed. Maybe just need to wait for a bit.]
Finally, I’m only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.
! You’re Arvind? I had no idea. Gosh, the past few days have been interesting for me—first Piotr Wozniak tells me he’s incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here’s Arvind Narayanan.
I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
On one hand, I agree—if a main page post improves one high-value person’s time, that alone is probably a few tens of thousands’ worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet… So what’s the marginal return on another literature survey?
A few comments:
You mention creatine on your blog. Creatine only works in specific demographics; I don’t know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in… At this point, I’m not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.
Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I’m not vegetarian but my meat consumption is lower than average, especially when I’m not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn’t worth it.
From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit—you can plan out what is necessary, and it’s clear when you succeeded. A literature review isn’t really either. So I’m not sure a prize in general is succeeded for the task.
My personal problems are that:
I can’t bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large—I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it’s one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I’d feel really terrible!
The difficult questions are:
How much does spaced repetition actually help memory?
Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it’s not clear what is even meant by ‘how much’ - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?
There’s probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?
New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.
The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.
Great comment. Perhaps I should be reading the prize literature! I’ll think about how to structure this better.
On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.
Just to make sure I’m clear, I’m not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.
I actually hadn’t read your page before, so thanks. I actually think your page is a fairly good start.
Do you have a good starting point for reading on the prize literature?
Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of ‘prize’ and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.
ok, thanks.
It’s not a literature review, but http://33bits.org/2011/06/06/the-surprising-effectiveness-of-prizes-as-catalysts-of-innovation/ does link to a McKinsey report on prizes, which ought to be good reading if their reputation can be relied upon.
It’s funny you linked to my post on prizes, which brought me here, because I’m actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it’s potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html
You guys need to get more rational about the value and importance of what you’re doing in this community. You’re arguing over trivial amounts of money here. I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
I’m putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn’t changed. Maybe just need to wait for a bit.]
Finally, I’m only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.
! You’re Arvind? I had no idea. Gosh, the past few days have been interesting for me—first Piotr Wozniak tells me he’s incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here’s Arvind Narayanan.
On one hand, I agree—if a main page post improves one high-value person’s time, that alone is probably a few tens of thousands’ worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet… So what’s the marginal return on another literature survey?
A few comments:
You mention creatine on your blog. Creatine only works in specific demographics; I don’t know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in… At this point, I’m not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.
Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I’m not vegetarian but my meat consumption is lower than average, especially when I’m not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn’t worth it.
Note that posts aren’t voted up to anywhere; they’re promoted by editors.
Ah, I see, thanks.