I’d like to work on this (be paid to substantially improve my http://www.gwern.net/Mnemosyne ? Where’s the downside?), but you ask hard questions which would require a lot of research to be done to my satisfaction and during which I could be scooped. I’m not sure a race is the best way to run this.
Why do you care if it is done to your satisfaction when the prize is awarded based on other’s satisfaction with it?
This is an expected value problem. Decide how much a unit of your time is worth, how much time you are willing to invest and then (the hard part) estimate your likelihood for success.
So, if you value your time at $10/hr, are willing to invest 10 hours and estimate you will win the $155 X% of the time then we get this equation:
P(winning)$amount won—P(losing)$amount not won = initial investment
x$155 - (1-x)$0 = $10/hr * 10 hours
solve for x
x = 64.5%, in other words, to “break even” you need to be sure that if you invest 10 hours of your time in this project that you will win it at least 64.5% of the time.
A race may not be the best way to run this for you, since I suspect that you value your time a high rate relative to the potential payoff. But someone who values their time less (or is more productive than you per unit time) may think a race is a wonderful idea.
Why do you care if it is done to your satisfaction when the prize is awarded based on other’s satisfaction with it?
Why do I care… what I care? Seriously?
But I do agree with your expected utility calculations. The problem is, I can’t motivate myself to write under conditions of uncertainty like that. I understand that shouldn’t matter, especially since I consider such a project worthwhile even in the absence of $155, but it does.
Getting it done to other’s satisfaction and getting it done to your own are not mutually exclusive. You can work quickly to win the prize and then go back and expand.
Could you specify the drawbacks you see to a race? Does something else seem like it would work better? Does a dated prize (where the best entry is selected on a specific date) or a work contract seem like it would work significantly better? I’m Just Trying Things right now, and I’m not especially attached to this format.
Which are the especially hard questions? What is the primary source of difficulty?
From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit—you can plan out what is necessary, and it’s clear when you succeeded. A literature review isn’t really either. So I’m not sure a prize in general is succeeded for the task.
My personal problems are that:
I can’t bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large—I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it’s one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I’d feel really terrible!
The difficult questions are:
How much does spaced repetition actually help memory?
Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it’s not clear what is even meant by ‘how much’ - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?
There’s probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?
New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.
The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.
Great comment. Perhaps I should be reading the prize literature! I’ll think about how to structure this better.
On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.
Just to make sure I’m clear, I’m not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.
I actually hadn’t read your page before, so thanks. I actually think your page is a fairly good start.
Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of ‘prize’ and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.
It’s funny you linked to my post on prizes, which brought me here, because I’m actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it’s potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html
You guys need to get more rational about the value and importance of what you’re doing in this community. You’re arguing over trivial amounts of money here. I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
I’m putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn’t changed. Maybe just need to wait for a bit.]
Finally, I’m only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.
! You’re Arvind? I had no idea. Gosh, the past few days have been interesting for me—first Piotr Wozniak tells me he’s incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here’s Arvind Narayanan.
I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
On one hand, I agree—if a main page post improves one high-value person’s time, that alone is probably a few tens of thousands’ worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet… So what’s the marginal return on another literature survey?
A few comments:
You mention creatine on your blog. Creatine only works in specific demographics; I don’t know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in… At this point, I’m not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.
Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I’m not vegetarian but my meat consumption is lower than average, especially when I’m not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn’t worth it.
I’d like to work on this (be paid to substantially improve my http://www.gwern.net/Mnemosyne ? Where’s the downside?), but you ask hard questions which would require a lot of research to be done to my satisfaction and during which I could be scooped. I’m not sure a race is the best way to run this.
Why do you care if it is done to your satisfaction when the prize is awarded based on other’s satisfaction with it?
This is an expected value problem. Decide how much a unit of your time is worth, how much time you are willing to invest and then (the hard part) estimate your likelihood for success.
So, if you value your time at $10/hr, are willing to invest 10 hours and estimate you will win the $155 X% of the time then we get this equation:
P(winning)$amount won—P(losing)$amount not won = initial investment
x$155 - (1-x)$0 = $10/hr * 10 hours
solve for x
x = 64.5%, in other words, to “break even” you need to be sure that if you invest 10 hours of your time in this project that you will win it at least 64.5% of the time.
A race may not be the best way to run this for you, since I suspect that you value your time a high rate relative to the potential payoff. But someone who values their time less (or is more productive than you per unit time) may think a race is a wonderful idea.
Why do I care… what I care? Seriously?
But I do agree with your expected utility calculations. The problem is, I can’t motivate myself to write under conditions of uncertainty like that. I understand that shouldn’t matter, especially since I consider such a project worthwhile even in the absence of $155, but it does.
Getting it done to other’s satisfaction and getting it done to your own are not mutually exclusive. You can work quickly to win the prize and then go back and expand.
No, I can’t. That’s my problem, as I’ve been explaining in my comments.
Perhaps you two should collaborate and split the prize in some way.
I think it’s a bit late for that. But I’ve PMed Duke.
Could you specify the drawbacks you see to a race? Does something else seem like it would work better? Does a dated prize (where the best entry is selected on a specific date) or a work contract seem like it would work significantly better? I’m Just Trying Things right now, and I’m not especially attached to this format.
Which are the especially hard questions? What is the primary source of difficulty?
Duplicated effort, some of which is presumably invisible.
From what I remember of the prize literature, prizes work best for well-delineated, objectively judgeable accomplishments. Like putting a rocket into orbit—you can plan out what is necessary, and it’s clear when you succeeded. A literature review isn’t really either. So I’m not sure a prize in general is succeeded for the task.
My personal problems are that:
I can’t bring myself to start a new project like intensively researching spaced repetition to establish exactly how good the evidence is, how effective it is, and what domains it does and does not work in, without having a good idea of what I might receive in turn. Uncertainty is paralyzing for me.
This is especially true because your prize is large—I felt guilty taking a bitcoin from Kiba for Bitcoin is Worse is Better because I knew it was incomplete in several respects, although it’s one of my best essays and gave him a lot of traffic; and a bitcoin is only worth $20, so how would I feel delivering something bad for $155? I’d feel really terrible!
The difficult questions are:
How much does spaced repetition actually help memory?
Quantifying the advantage in any more than an ad hoc way like http://www.gwern.net/Mnemosyne#fn3 would be difficult since it requires delving into the specifics of the data, and forgetting curves are non-linear so it’s not clear what is even meant by ‘how much’ - memories per review at 1 year? At 10 years? At 50 years? Integrated over an actuarially projected lifespan?
Does spaced repetition have hidden benefits or costs?
There’s probably no direct research on this, so now one has to delve into related research fields and look for very weak evidence or correlations, like maybe high memory correlated with longevity in old people.
Is there any interesting ongoing research? If so, what is it on?
New research tends to be hard to get (= wasted time going around paywalls) and requires following a lot of citations in bibliographies and checking out blogs and emails to really get a sense of where a field is now.
The best way is probably to informally or formally contract with someone like lukeprog (who does this sort of literature review routinely) to conduct the review.
Great comment. Perhaps I should be reading the prize literature! I’ll think about how to structure this better.
On your personal problems: consider finding someone who has more time and acting as a consultant for them. For example, Duke, who comments elsewhere in this thread.
Just to make sure I’m clear, I’m not trying to gain a detailed or very specific understanding of the literature on SR, but to figure out how confident I should be in recommending SR to others and to myself and what I should recommend others and I do and what I should expect it to help and how much.
I actually hadn’t read your page before, so thanks. I actually think your page is a fairly good start.
Do you have a good starting point for reading on the prize literature?
Hm, not really. What I would do is google Robin Hanson, Anders Broberg, and LW for mentions of ‘prize’ and use them as entrance into the academic literature, and spider authors & citations & bibliographies from there.
ok, thanks.
It’s not a literature review, but http://33bits.org/2011/06/06/the-surprising-effectiveness-of-prizes-as-catalysts-of-innovation/ does link to a McKinsey report on prizes, which ought to be good reading if their reputation can be relied upon.
It’s funny you linked to my post on prizes, which brought me here, because I’m actually very interested in spaced repetition. In fact just a couple of weeks ago I mentioned it’s potentially worth millions of dollars to me: http://arvindn.livejournal.com/132233.html
You guys need to get more rational about the value and importance of what you’re doing in this community. You’re arguing over trivial amounts of money here. I’d say that any post that’s good enough to get voted up to the main page is worth tens of thousands of dollars to mankind. Easily.
I’m putting my money where my mouth is chipping in $100 for the new prize. [Edit. Hmm, I got a paypal receipt but the total on chipin hasn’t changed. Maybe just need to wait for a bit.]
Finally, I’m only a sporadic reader, but allow me to express my deep gratitude and admiration for the community.
! You’re Arvind? I had no idea. Gosh, the past few days have been interesting for me—first Piotr Wozniak tells me he’s incrementally reading 8 articles mentioning or by me, then Danny Hillis replies to a comment of mine over at the Technium, and now here’s Arvind Narayanan.
On one hand, I agree—if a main page post improves one high-value person’s time, that alone is probably a few tens of thousands’ worth. The other part of me is cynically remarking that Wozniak and psychologists in general have been advocating spaced repetition for the last >2 decades / centuries (respectively), the research is unassailable, and yet… So what’s the marginal return on another literature survey?
A few comments:
You mention creatine on your blog. Creatine only works in specific demographics; I don’t know whether they include you.
You mention dual n-back; the results seem to keep getting weaker since Jaeggi 2008, and Jaeggi 2011 seems extremely weak. I used to be extremely enthusiastic, but as the studies keep coming in… At this point, I’m not sure how much time is worth putting into DNB for just the WM benefits. Probably not much past D5B though.
Thanks for the information! That does look pretty damning for dual n-back. As for creatine, I’m not vegetarian but my meat consumption is lower than average, especially when I’m not weight training. At any rate, I stopped taking it after a couple of months because I decided the unknown long-term health risk from increased renal stress wasn’t worth it.
Note that posts aren’t voted up to anywhere; they’re promoted by editors.
Ah, I see, thanks.