First off, some context: this post is a submission for a contest I and others sponsored.
What you have written is a useful LW post, but I do not think it fills the requirements of the contest. In case it is not clear: I do not want to come across as an enemy; I bear you no ill feelings. I have decided that the original format of the contest was poorly chosen for the reasons Gwern mentioned, and I am going to restart the contest using a more traditional format. If you feel treated unfairly, please email me and we will arranged a video chat to discuss it (my email is my username at gmail.com).
My concerns in the order I noted them:
Your introduction seems a bit odd and confusing. I think it would have been significantly clearer to directly address the prize you were responding to and the purpose of the post.
You only cite one academic source (granted that it’s a survey paper). How hard did you look for other academic sources? You don’t mention this.
You don’t explain the structure of the evidence: for example you might have said something like ” there is quite a bit of related old academic research and one new literature review but little ongoing research. There is a good amount of both older and ongoing non-academic research”
your quoted equations look like they mean R = e—d t instead of R = exp(-dt)
you don’t seem to address several of my questions. In all cases it is perfectly OK to say something like ‘academic research does not seem to address the question of X. non-academic research and writing has Y to say about the issue’ (when it’s true anyway). If there is no good evidence on a particular topic, I’d like to know about that.
My questions were:
What spacing is best? (you do address this significantly; I would have also liked to hear about quantitative discussion about how large spacing should be (if there was any))
How much does spaced repetition actually help memory? (I see your anecdotal evidence, which is good, but I would also like to hear about any non-anecdotal evidence)
Does spaced repetition have hidden benefits or costs? (no discussion, but maybe not a good question)
Does the effectiveness vary across domains? How much? (no discussion)
Is there research on the kinds of questions that work best? (some discussion of non-academic research, but no mention of academic research)
What questions do researchers think are most important? (no discussion)
Is there any interesting ongoing research? If so, what is it on? (no discussion)
What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person? (no discussion)
What does spaced repetition not do that people might expect it to? (no discussion)
I think it is unfair of you to post a public critique of my submission since this is a contest. I have effectively been penalized for being first. Every submission that follows will have the benefit of seeing this critique.
I am also concerned that you have decided to change the contest format immediately following my submission. In my estimation, you had either already decided to change the format prior to my submission (clearly a major disadvantage to me), or you decided to change the format based on my submission, which, again, effectively penalizes me for being first.
If we are comparing fairness against some platonic ideal, I would point out that you already started with a significant advantage—being able to draw upon my Mnemosyne article advocating SRS and covering many of the questions jsalvatier asked. (You quote me, most obviously, but I also suspect you came to Wozniak’s equations by way of my own discussion of Wozniak’s equations for calculating the five-minute rule, among other things.)
I did not want to be first to criticize this because it would look like sour grapes for not having my act together enough to submit my own article (I was waiting for you to email or IRC me & I only have 4.6k karma anyway), but your article is exactly what you suggested in the contest thread: something rushed together, and a good example of what I meant by a contest not being a good incentive structure for such a spaced repetition article. For example, lukeprog suggested a number of solid useful references, which you did not use, and the review article you did link includes discussion and references for some of the questions jsalvatier is most interested in!
That said, I do understand why you are upset, and this is part of why I was against a contest format. If jsalvatier denies you the $300 or whatever, then you will feel aggrieved that a promise was broken and yourself deprived of a pretty substantial sum. If jsalvatier awards it to you, then competitors like myself will feel aggrieved that a low-quality product won and that the community was not as well-informed as it could be.
Currently, this seems to be a moot point. The terms of the contest were that the winner was the article be promoted to the main page, not merely present in article-space. Right now, after roughly 3 days, this article is still lingering at 10 points. I have the impression that articles tend to level off after a few days, having done most of their rising or falling by that point. So purely on the karma aspect, it seems pretty unlikely that this article will be promoted and so you would not have/will not won/win the contest (jsalvatier’s criticism not having affected the outcome either way).
I agree that changing the contest immediately following your submission negatively affects you. Basically you provided me with information about how this kind of contest would work and have not been compensated for it. Please note that you can still participate in the reworked contest. I’d like to address this, but feel moderately uncomfortable discussing this in public (not sure why), would you email me (my username at gmail) ?
I disagree that critiquing your submission in public gives you an especially unfair disadvantage since in the original contest there was a big first mover advantage and because you have personalized feedback about how to modify your submission.
After some deliberation, I’ve decided to withdraw this request. I am content with my submission. I am also content not to receive the prize or any portion thereof.
I want to express my approval. You practiced the difficult discipline of taking some time to deliberate on something that clearly engaged your emotions strongly, and the yet more difficult art of actually changing your mind.
No upvote, though; a more advanced rationalist wouldn’t get an upvote because they would have remembered to deliberate before expressing their first opinion.
First off, some context: this post is a submission for a contest I and others sponsored.
What you have written is a useful LW post, but I do not think it fills the requirements of the contest. In case it is not clear: I do not want to come across as an enemy; I bear you no ill feelings. I have decided that the original format of the contest was poorly chosen for the reasons Gwern mentioned, and I am going to restart the contest using a more traditional format. If you feel treated unfairly, please email me and we will arranged a video chat to discuss it (my email is my username at gmail.com).
My concerns in the order I noted them:
Your introduction seems a bit odd and confusing. I think it would have been significantly clearer to directly address the prize you were responding to and the purpose of the post.
You only cite one academic source (granted that it’s a survey paper). How hard did you look for other academic sources? You don’t mention this.
You don’t explain the structure of the evidence: for example you might have said something like ” there is quite a bit of related old academic research and one new literature review but little ongoing research. There is a good amount of both older and ongoing non-academic research”
your quoted equations look like they mean R = e—d t instead of R = exp(-dt)
you don’t seem to address several of my questions. In all cases it is perfectly OK to say something like ‘academic research does not seem to address the question of X. non-academic research and writing has Y to say about the issue’ (when it’s true anyway). If there is no good evidence on a particular topic, I’d like to know about that. My questions were:
What spacing is best? (you do address this significantly; I would have also liked to hear about quantitative discussion about how large spacing should be (if there was any))
How much does spaced repetition actually help memory? (I see your anecdotal evidence, which is good, but I would also like to hear about any non-anecdotal evidence)
Does spaced repetition have hidden benefits or costs? (no discussion, but maybe not a good question)
Does the effectiveness vary across domains? How much? (no discussion)
Is there research on the kinds of questions that work best? (some discussion of non-academic research, but no mention of academic research)
What questions do researchers think are most important? (no discussion)
Is there any interesting ongoing research? If so, what is it on? (no discussion)
What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person? (no discussion)
What does spaced repetition not do that people might expect it to? (no discussion)
I think it is unfair of you to post a public critique of my submission since this is a contest. I have effectively been penalized for being first. Every submission that follows will have the benefit of seeing this critique.
I am also concerned that you have decided to change the contest format immediately following my submission. In my estimation, you had either already decided to change the format prior to my submission (clearly a major disadvantage to me), or you decided to change the format based on my submission, which, again, effectively penalizes me for being first.
If we are comparing fairness against some platonic ideal, I would point out that you already started with a significant advantage—being able to draw upon my Mnemosyne article advocating SRS and covering many of the questions jsalvatier asked. (You quote me, most obviously, but I also suspect you came to Wozniak’s equations by way of my own discussion of Wozniak’s equations for calculating the five-minute rule, among other things.)
I did not want to be first to criticize this because it would look like sour grapes for not having my act together enough to submit my own article (I was waiting for you to email or IRC me & I only have 4.6k karma anyway), but your article is exactly what you suggested in the contest thread: something rushed together, and a good example of what I meant by a contest not being a good incentive structure for such a spaced repetition article. For example, lukeprog suggested a number of solid useful references, which you did not use, and the review article you did link includes discussion and references for some of the questions jsalvatier is most interested in!
That said, I do understand why you are upset, and this is part of why I was against a contest format. If jsalvatier denies you the $300 or whatever, then you will feel aggrieved that a promise was broken and yourself deprived of a pretty substantial sum. If jsalvatier awards it to you, then competitors like myself will feel aggrieved that a low-quality product won and that the community was not as well-informed as it could be.
Currently, this seems to be a moot point. The terms of the contest were that the winner was the article be promoted to the main page, not merely present in article-space. Right now, after roughly 3 days, this article is still lingering at 10 points. I have the impression that articles tend to level off after a few days, having done most of their rising or falling by that point. So purely on the karma aspect, it seems pretty unlikely that this article will be promoted and so you would not have/will not won/win the contest (jsalvatier’s criticism not having affected the outcome either way).
I agree that changing the contest immediately following your submission negatively affects you. Basically you provided me with information about how this kind of contest would work and have not been compensated for it. Please note that you can still participate in the reworked contest. I’d like to address this, but feel moderately uncomfortable discussing this in public (not sure why), would you email me (my username at gmail) ?
I disagree that critiquing your submission in public gives you an especially unfair disadvantage since in the original contest there was a big first mover advantage and because you have personalized feedback about how to modify your submission.
I request arbitration and that it be done publicly.
OK, choose someone with 5k+ Karma who is willing to arbitrate (or I can choose someone if you prefer) and have them post here.
After some deliberation, I’ve decided to withdraw this request. I am content with my submission. I am also content not to receive the prize or any portion thereof.
I want to express my approval. You practiced the difficult discipline of taking some time to deliberate on something that clearly engaged your emotions strongly, and the yet more difficult art of actually changing your mind.
No upvote, though; a more advanced rationalist wouldn’t get an upvote because they would have remembered to deliberate before expressing their first opinion.
I’m not asking for a Rationalist of the Month Award, just a measly upvote.