I still believe that the spaced repetition dataset produced by memosyne is underanalysed. Gwern put effort into bringing that dataset into a good format.
In particular:
Is there a way to calculate the brain power for an individual day? Reviews made at a high brainpower day would have a higher successful. Having such a metric would be very useful for Quantified Self (QS) purposes because it means that everybody who uses Anki or memosyne daily would get a free QS metric.
Is there a better way to to calculate the interval for cards than the supermemo algorithm that can be proven to be better based on the data from the memosyne set? If so how much better? (If you find something that shows real improvement over the status quo, the next step would be to write an Anki addon that does AB split testing with it)
I think that especially the first task has the chance to provide a decently cited academic paper.
I don’t know how much effort either of those tasks takes and whether it would go beyond a project for a college course, but both projects should be interesting and highly useful. You might also find other interesting question when you investigate the data set.
Thanks, that looks great! ~60% probability I’ll do my project on this, actually.
The second one actually looks more interesting and useful to me, since it would directly lead to improved scheduling. There’s a lot of literature on the spacing effect, and it doesn’t look like anyone’s actually done empirical analysis of it on this scale before. (And I do a lot of reviews daily, so I wouldn’t be surprised if this project actually took negative net time!) There’s also disagreement between e.g. Supermemo and Anki about which algorithm is best, so the issue isn’t very settled.
The first one (calculating brainpower for a day) seems easy to do to some extent—just look at the average time each review took, or some function of time-per-review and number of reviews. I’m doubtful about whether you could get more reliability out of looking at e.g. card ratings. Perhaps a better way to measure brainpower would be n-back, or Seth Roberts’s arithmetic test.
There’s also disagreement between e.g. Supermemo and Anki about which algorithm is best, so the issue isn’t very settled.
Yes.
Wozniak who wrote Supermemo did nearly all his work on his own. I think there a good chance that he missed significant things that are known in 2014 about machine learning in his work.
Anki and Mnemosyne are also both written by people without strong knowledge of machine learning.
Maybe some deep learning algorithm is simply better than Wozniaks idea.
The first one (calculating brainpower for a day) seems easy to do to some extent—just look at the average time each review took, or some function of time-per-review and number of reviews.
The problem is that not every card is similar. If I add 100 new cards in a single day and go through them and they are all relatively easy compared to the cards I usually answer, I will effect the brainpower score if you simply calculate it the easy way. Thinking up a way that’s robust to such effect is where it get’s tricky.
Perhaps a better way to measure brainpower would be n-back, or Seth Roberts’s arithmetic test.
Doing n-back or arithmetic tests means that you need to spend additional time. I have put quite a lot of thought in the issue and even did arithmetic tests via a self written android app for over a year and I have come to the conclusion that we simply won’t get a significant number of people to do this. A lot of people are already doing Anki or Mnemosyne and would get free data without spending additional time or mental effort.
Seth Roberts arithmetic test has the problem that it doesn’t tell you how to treat a speed up at the cost of more errors. If I remember right button produced speed up but raised the error rate slightly. The same problem comes with Anki. I often observed speed ups in answering cards that come along with higher error rates.
Long term memory brainpower is also an interesting metric. As far as I know there are no good tests for it. At present psychologists do have tests for short term memory and tests for reaction time and tasks like arithmetic.
Having a good way to measure long term memory brainpower at daily resolution might be useful for research about diseases like alzheimers and detecting it in it’s early stages. Researchers cite their tools, so a relevant paper that gives them a metric for long term memory brainpower has a good chance of being cited widely.
Having a brain power metric would also allow for better scheduling. Let’s say one day you are ill and your brain isn’t working properly. You do you reviews. Various cards which you reviewed successfully don’t get the boost that they would get if your brain was working properly.
After a month has passed an algorithm that detects days with low brain power. It detects the problem. It could reduce the intervals of the cards reviewed on that day that aren’t already reviewed.
That means you get a better scheduling algorithm when you first solve the brainpower issue.
I’m not sure about the effect size but Anki Droid has >1,000,000 downloads. A lot of smart people use Anki from an effective altruism perspective, making smart people more effective is highly useful.
I still believe that the spaced repetition dataset produced by memosyne is underanalysed. Gwern put effort into bringing that dataset into a good format.
In particular:
Is there a way to calculate the brain power for an individual day? Reviews made at a high brainpower day would have a higher successful. Having such a metric would be very useful for Quantified Self (QS) purposes because it means that everybody who uses Anki or memosyne daily would get a free QS metric.
Is there a better way to to calculate the interval for cards than the supermemo algorithm that can be proven to be better based on the data from the memosyne set? If so how much better? (If you find something that shows real improvement over the status quo, the next step would be to write an Anki addon that does AB split testing with it)
I think that especially the first task has the chance to provide a decently cited academic paper.
I don’t know how much effort either of those tasks takes and whether it would go beyond a project for a college course, but both projects should be interesting and highly useful. You might also find other interesting question when you investigate the data set.
Thanks, that looks great! ~60% probability I’ll do my project on this, actually.
The second one actually looks more interesting and useful to me, since it would directly lead to improved scheduling. There’s a lot of literature on the spacing effect, and it doesn’t look like anyone’s actually done empirical analysis of it on this scale before. (And I do a lot of reviews daily, so I wouldn’t be surprised if this project actually took negative net time!) There’s also disagreement between e.g. Supermemo and Anki about which algorithm is best, so the issue isn’t very settled.
The first one (calculating brainpower for a day) seems easy to do to some extent—just look at the average time each review took, or some function of time-per-review and number of reviews. I’m doubtful about whether you could get more reliability out of looking at e.g. card ratings. Perhaps a better way to measure brainpower would be n-back, or Seth Roberts’s arithmetic test.
Yes. Wozniak who wrote Supermemo did nearly all his work on his own. I think there a good chance that he missed significant things that are known in 2014 about machine learning in his work. Anki and Mnemosyne are also both written by people without strong knowledge of machine learning.
Maybe some deep learning algorithm is simply better than Wozniaks idea.
The problem is that not every card is similar. If I add 100 new cards in a single day and go through them and they are all relatively easy compared to the cards I usually answer, I will effect the brainpower score if you simply calculate it the easy way. Thinking up a way that’s robust to such effect is where it get’s tricky.
Doing n-back or arithmetic tests means that you need to spend additional time. I have put quite a lot of thought in the issue and even did arithmetic tests via a self written android app for over a year and I have come to the conclusion that we simply won’t get a significant number of people to do this. A lot of people are already doing Anki or Mnemosyne and would get free data without spending additional time or mental effort.
Seth Roberts arithmetic test has the problem that it doesn’t tell you how to treat a speed up at the cost of more errors. If I remember right button produced speed up but raised the error rate slightly. The same problem comes with Anki. I often observed speed ups in answering cards that come along with higher error rates.
Long term memory brainpower is also an interesting metric. As far as I know there are no good tests for it. At present psychologists do have tests for short term memory and tests for reaction time and tasks like arithmetic.
Having a good way to measure long term memory brainpower at daily resolution might be useful for research about diseases like alzheimers and detecting it in it’s early stages. Researchers cite their tools, so a relevant paper that gives them a metric for long term memory brainpower has a good chance of being cited widely.
Having a brain power metric would also allow for better scheduling. Let’s say one day you are ill and your brain isn’t working properly. You do you reviews. Various cards which you reviewed successfully don’t get the boost that they would get if your brain was working properly.
After a month has passed an algorithm that detects days with low brain power. It detects the problem. It could reduce the intervals of the cards reviewed on that day that aren’t already reviewed. That means you get a better scheduling algorithm when you first solve the brainpower issue.
I’m not sure about the effect size but Anki Droid has >1,000,000 downloads. A lot of smart people use Anki from an effective altruism perspective, making smart people more effective is highly useful.