Memory, Spaced Repetition and Life
I have made the case that with the advent of the internet went the need to memorize anything. Why worry about memorizing when I’ll never be tested for a grade and can access knowledge nearly instantaneously? As well, I reasoned, I have probably already memorized everything I need to. I focused my time instead on learning thinking techniques, such as Bayesian calculations, expected value calculations and various things for improving emotional control.
But after reading this a couple months back I decided to experiment with Anki, a digital flashcard program which exploits a cognitive phenomenon called the Spacing Effect by implementing a memorization technique called Spaced Repetition. The Spacing Effect is the widely observed tendency for people to recall information better when studied a few times over a long period than when studied many times over a short period. Balota et al (2007):
Spacing effects occur across domains (e.g., learning perceptual motor tasks vs. learning lists of words), across species (e.g., rats, pigeons, and humans), across age groups and individuals with different memory impairments, and across retention intervals of seconds to months.
Gwern analogizes the spacing effect with radioactive decay:
You can think of the ‘forgetting curve’ as being like a chart of radioactive half-lives: each review bumps your memory up in strength 50% of the chart, say, but review doesn’t do very much in the early days because the memory simply hasn’t decayed very much! (Chart)
One consequence of the spacing effect is that cramming is useful for recalling things shortly after memorizing them; however, if those crammed memories are not eventually refreshed then they are likely to decay to nothing. From this observation came Spaced Repetition: a memorization technique using flashcards (usually) shown at increasing intervals of time to optimize the relationship between number of reviews and strength of memory. The PC explosion was a boon to Spaced Repetition since storing and showing flashcards as well as physically calculating their frequencies were delegated to the computer. The program Anki, for instance, permits the user to generate flashcard decks, specify study session length and frequency, specify how many new cards are introduced per session and specify the frequency of the cards based on the user’s input. Hard material is shown more often than easy material, with the ease or difficulty being determined directly by the user selecting buttons marked “again,” “easy,” “good,” and “hard.”
That sounds nifty, but how well does it work? As for myself, using the Anki default settings, I was able to thoroughly memorize a deck of 80 cognitive biases and related terms (160 cards total, name to definition and vice avers) in about three weeks using Anki ~15 minutes/day. Since the cards are pushed back further and further for review as I progressed, I have only five cards to review today. The first one, Endowment Effect, came instantly to me so I selected the “easy” button. Now, as a result of the Anki algorithm, I won’t see that card for 1.3 months. My low expectations for the Anki experiment were exceeded.
Piotr Wozniak, who designed the first SuperMemo algorithm in the early 80′s (of which later versions are still in use in SuperMemo as well as Anki), and devoted enormous energies to studying modern computer aided self-instruction systems promotes spaced repetition. He and two others developed a two-variable framework for memorization which they built upon to examine a way of optimizing interval spacing in Spaced Repetition. The first variable, memory retrieval (R), is the probability of recalling something and is approximated by an exponential decay function, while the second variable, memory stability (S), measures how long a memory lasts before it is forgotten entirely. Wozniak et al, expressed S as the inter-repetition interval time that produces R = 90% (the likelihood of recall being a 9 out of 10 chance) and concluded the following:
We express the changes in retrievability as:
(3.1) R=e-d*t
where:
t—time
R—probability of recall at time t (retrievability)
d—decay constant dependent on the stability
We can replace the constant d dependent on stability, with a constant k that is independent of stability:
(3.2) R=e-k*t/S
where:
t—time
R—probability of recall at time t
S—stability expressed by the inter-repetition interval that results in retrievability of 90% (i.e. R=0.9)
k—constant independent of stability
Drawing on analysis of large data sets cultivated from SuperMemo, Wozniak et al provide empirical evidence that memory decay matches their exponential decay approximation. The goal of Wozniak’s SuperMemo algorithm is to optimize inter-repetition spacing by (theoretically) refreshing a memory the moment before it decays totally, thus spiking that memory until it decays near totally again and gets spiked again. (Although, depending on the importance of being able to recall of a piece of information, it can be used, theoretically, to spike a memory every time it decays to likelihood of recall of 90%, 80%, 70%, etc.) Interestingly, in a meta-analysis by Balota et al (2007) , the authors conclude that while spaced repetition is certainly better than massed practice (studying all at once and then not reviewing again), spaced repetition shows no advantage over static spaced repetition (holding intervals constant)! Since most studies cited in the meta-analysis used a small number (usually three) retrieval attempts, the authors suggest that future research should expand this number to better reflect the way people can practically use spaced repetition. In my estimation, when it comes to memorization, given the ease of use of these digital flashcards programs, the specific algorithm design is a secondary concern to being personally disciplined to consistently review material until you think you’ve internalized it fully.
Another consideration beyond algorithm design is formulating a usable flashcard deck: simplifying the information and implementing techniques to enhance recall. Wozniak et al found evidence that it was harder to recall information the more complex it was. Hence, Wozniak recommends 20 rules for formatting knowledge to make flashcards more digestible during reviews. The first three rules are standard: understand before you learn, learn before you memorize and build on the basics. The remaining rules are specific to developing and maintaining flashcard decks, such as simplifying questions, using clozed deletion (a sentence missing a part replaced by three dots), including images, avoiding sets, etc. If you are planning on creating your own deck then familiarize yourself with these rules.
Additionally, I recommend including hyperlinks, if available, in your cards to sources with thorough explanations of the topics, and to be careful doing Wikipedia-based decks. Having small previous knowledge of cognitive biases when I started, it was essential to read expanded explanations on many of the terms to understand them completely, so I actually updated the deck, which someone else created, with hyperlinks on every card. I think it greatly enhanced the usability and effectiveness of the deck. Incidentally, when I went to Wikipedia to better grasp many of the terms, I found several entries there lacking in credibility. On at least two occasions, after being skeptical of an entry on a term, I Googled the term and found every other mention of it on the internet was either sourced to Wikipedia or directly copied from there.
This all still sounds nifty, but, I’ll repeat, why worry about memorizing when I’ll never be tested for a grade and can access knowledge nearly instantaneously? As for standard trivia type information (state capitals, etc), memorization is virtually a total waste of time (thank you, technology!). Instant recall of facts, except on Jeopardy or when using a foreign language, is generally not of value. Think of a time when your inability to instantly recall a fact resulted in a financial loss for you—I can’t. On the other hand, every damn day I am confronted with dynamic situations where I am forced to make quick decisions that vary in effectiveness based on how well I analyze what is happening and construct counter-strategies that maximize my utility. In these moments, when the necessary facts are right in front of us, what we usually don’t have is a comprehensive database of methodologies, heuristics and other decision theoretic knowledge to surf through and use for calculating useful outputs. You might be familiar with Bayesianism, Nonviolent Communication (NVC), PUA, logical fallacies and the like, but it is unlikely you have internalized the concepts to the point where even in the face of chaos or emotional turmoil (when it likely matters most) you can implement them to the best of your mental ability. Thus, I recommend using digital flashcards employing Spaced Repetition to memorize a relatively small set of widely applicable methods (and related knowledge) for use in dynamic situations that require instant or near-instant action.
NVC is the poster-child because it is an easy to remember step-by-step process which does not require complicated inputs for any of the steps; virtually anyone can observe a situation, dissect relevant information from it and then run it through the NVC process. While simple, NVC might be most valuable in chaotic or emotionally-charged social situation when minds are thrust into primate mode, making it that more important to ingrain thoroughly. Divia, who created an NVC deck and several other useful Anki decks, recounts successfully using NVC on a train when a drunk sports fan near her was acting belligerent (imagine that!). In my experience, having internalized a bunch of cognitive biases, I’m feel like I am vigilant about recognizing them in my thoughts and behaviors and in those of others, without devoting much conscious effort to doing so. I expect that databasing of logical fallacies and human behavioral cues will have the same effect. Please list other methods or knowledge that you think would be worth devoting time to memorize.
In sum, spaced repetition for memorization is superior to massed consumption without further review, although it is undetermined what inter-repetition algorithm is best. It seems that having discipline and consistency in review is more important than the inter-repetition spacing, as even static spacing works well. Also important is the design and maintenance of the flash card decks used for spaced repetition exercise, with an emphasis on simplifying the information presented. Lastly, be thoughtful about what things use spend time memorizing. Almost all information is just as useful to us wherever it currently is, especially if it is on the internet, than it would be if we had it memorized. Thus, I suggest using Anki or other spaced repetition software to memorize methods, concepts and knowledge can be deployed in dynamic situations where we are forced to make important decisions in an instant or near-instant.
REFERENCES
Balota, D.A., Duchek, J.M., & Logan, J.M. (2007). Is expanded retrieval practice a superior form of spaced retrieval? A critical review of the extant literature. In J. Nairne (Ed.), The Foundations of Remembering: Essays in Honor of Henry L. Roediger III, (pp. 83-106), Psychology Press, New York.
- Spaced Repetition literature review prize: And the winner is... by (19 Aug 2011 20:35 UTC; 37 points)
- Spaced repetition review (my entry) by (8 Aug 2011 15:48 UTC; 28 points)
- 's comment on Welcome to Less Wrong! (2010-2011) by (3 Jul 2011 4:18 UTC; 3 points)
- 's comment on [prize] Spaced Repetition literature review by (9 Jun 2011 18:58 UTC; 2 points)
- 's comment on Less Wrong lacks direction by (26 May 2015 22:14 UTC; 0 points)
If a piece of information isn’t accessible for your mind, you can’t notice unexpected connections between things.
For instance, some time ago I came up with the hypothesis that happiness might be an evolutionary mechanism that made us take more risks when we had spare resources and could risk doing so. But as someone pointed out, the opposite interpretation sounds just as plausible, if not more so. That is, when you have lots of resources you should concentrate on not losing them, and when you have few, you should take more risks until you’re in safer waters.
Now, later on I ran across an article with research about how moods affect our decision-making. Happy moods are related with heuristic, “business as usual” kind of decision-making; sadness triggers a more systematic analysis of the situation. That seems more in line with the interpretation that’s opposite to my original one.
Now, “happy moods are linked with heuristic processing, sad moods with systematic” is a piece of trivia. But it happens that this piece of trivia helps tremendously in evaluating my original hypothesis. Had I read that article earlier, and memorized that piece of trivia, I might have gotten my hypothesis right from the get-go.
For that matter, a lot of novel hypotheses seem to basically just involve the putting together of a large number of trivia. Somebody learns of one thing, and then of another seemingly unrelated thing, and then of a third thing, and then she notices the general pattern connecting all of them and formulates it out loud. But this requires that all of those trivia pieces are actually in your head, so that your brain can find the connections. If you need to know three different pieces of trivia in order to solve the problem, the fact that you could look them up at will doesn’t help if it never occurs to you that these are the ones you should be looking at.
My intuition is that we should be spending much more time memorizing trivia than we are. Not stuff like state capitals, obviously, but possibly useful details from whatever fields of knowledge we’re interested in and might want to contribute to.
I agree with all of this. Maybe it doesn’t come across clearly in my post, but I tried to differentiate between rank trivia and applicable knowledge, such as cognitive biases, decision theory concepts, logical fallacies, stuff you listed, etc. I don’t know what exactly differentiates applicable knowledge with near-worthless trivia, however.
Alright. I suspected that that might have been the case, but your post was a bit ambiguous.
Hope I managed what you had in mind. I had to manually re-insert your links, so you may want to check and see I got them all.
I was trying to avoid re-inserting the links, so thanks so much for your time!
First off, some context: this post is a submission for a contest I and others sponsored.
What you have written is a useful LW post, but I do not think it fills the requirements of the contest. In case it is not clear: I do not want to come across as an enemy; I bear you no ill feelings. I have decided that the original format of the contest was poorly chosen for the reasons Gwern mentioned, and I am going to restart the contest using a more traditional format. If you feel treated unfairly, please email me and we will arranged a video chat to discuss it (my email is my username at gmail.com).
My concerns in the order I noted them:
Your introduction seems a bit odd and confusing. I think it would have been significantly clearer to directly address the prize you were responding to and the purpose of the post.
You only cite one academic source (granted that it’s a survey paper). How hard did you look for other academic sources? You don’t mention this.
You don’t explain the structure of the evidence: for example you might have said something like ” there is quite a bit of related old academic research and one new literature review but little ongoing research. There is a good amount of both older and ongoing non-academic research”
your quoted equations look like they mean R = e—d t instead of R = exp(-dt)
you don’t seem to address several of my questions. In all cases it is perfectly OK to say something like ‘academic research does not seem to address the question of X. non-academic research and writing has Y to say about the issue’ (when it’s true anyway). If there is no good evidence on a particular topic, I’d like to know about that. My questions were:
What spacing is best? (you do address this significantly; I would have also liked to hear about quantitative discussion about how large spacing should be (if there was any))
How much does spaced repetition actually help memory? (I see your anecdotal evidence, which is good, but I would also like to hear about any non-anecdotal evidence)
Does spaced repetition have hidden benefits or costs? (no discussion, but maybe not a good question)
Does the effectiveness vary across domains? How much? (no discussion)
Is there research on the kinds of questions that work best? (some discussion of non-academic research, but no mention of academic research)
What questions do researchers think are most important? (no discussion)
Is there any interesting ongoing research? If so, what is it on? (no discussion)
What, if any, questions do researchers think it is important to answer? Are there other unanswered questions that would jump out at a smart person? (no discussion)
What does spaced repetition not do that people might expect it to? (no discussion)
I think it is unfair of you to post a public critique of my submission since this is a contest. I have effectively been penalized for being first. Every submission that follows will have the benefit of seeing this critique.
I am also concerned that you have decided to change the contest format immediately following my submission. In my estimation, you had either already decided to change the format prior to my submission (clearly a major disadvantage to me), or you decided to change the format based on my submission, which, again, effectively penalizes me for being first.
If we are comparing fairness against some platonic ideal, I would point out that you already started with a significant advantage—being able to draw upon my Mnemosyne article advocating SRS and covering many of the questions jsalvatier asked. (You quote me, most obviously, but I also suspect you came to Wozniak’s equations by way of my own discussion of Wozniak’s equations for calculating the five-minute rule, among other things.)
I did not want to be first to criticize this because it would look like sour grapes for not having my act together enough to submit my own article (I was waiting for you to email or IRC me & I only have 4.6k karma anyway), but your article is exactly what you suggested in the contest thread: something rushed together, and a good example of what I meant by a contest not being a good incentive structure for such a spaced repetition article. For example, lukeprog suggested a number of solid useful references, which you did not use, and the review article you did link includes discussion and references for some of the questions jsalvatier is most interested in!
That said, I do understand why you are upset, and this is part of why I was against a contest format. If jsalvatier denies you the $300 or whatever, then you will feel aggrieved that a promise was broken and yourself deprived of a pretty substantial sum. If jsalvatier awards it to you, then competitors like myself will feel aggrieved that a low-quality product won and that the community was not as well-informed as it could be.
Currently, this seems to be a moot point. The terms of the contest were that the winner was the article be promoted to the main page, not merely present in article-space. Right now, after roughly 3 days, this article is still lingering at 10 points. I have the impression that articles tend to level off after a few days, having done most of their rising or falling by that point. So purely on the karma aspect, it seems pretty unlikely that this article will be promoted and so you would not have/will not won/win the contest (jsalvatier’s criticism not having affected the outcome either way).
I agree that changing the contest immediately following your submission negatively affects you. Basically you provided me with information about how this kind of contest would work and have not been compensated for it. Please note that you can still participate in the reworked contest. I’d like to address this, but feel moderately uncomfortable discussing this in public (not sure why), would you email me (my username at gmail) ?
I disagree that critiquing your submission in public gives you an especially unfair disadvantage since in the original contest there was a big first mover advantage and because you have personalized feedback about how to modify your submission.
I request arbitration and that it be done publicly.
OK, choose someone with 5k+ Karma who is willing to arbitrate (or I can choose someone if you prefer) and have them post here.
After some deliberation, I’ve decided to withdraw this request. I am content with my submission. I am also content not to receive the prize or any portion thereof.
I want to express my approval. You practiced the difficult discipline of taking some time to deliberate on something that clearly engaged your emotions strongly, and the yet more difficult art of actually changing your mind.
No upvote, though; a more advanced rationalist wouldn’t get an upvote because they would have remembered to deliberate before expressing their first opinion.
I’m not asking for a Rationalist of the Month Award, just a measly upvote.
I like that you specifically noted an exception in the case of foreign languages, as it was the one salient point I would have raised otherwise. Not that I think it might be the only point in contention to be raised, merely that it would be the only one I would have brought up, though. I wish you had emphasized it a little more heavily in your rhetoric, though that might just be my own biases in play.
The link is outdated. The decks can now be found here.
In an effort to learn Mandarin, I started to use Anki. At the time the android app kept crashing. (Reviews seem to say it’s better now.) I also had a doubt that the current two variable algorithm is actually optimal.
So I set up a webpage with a mobile interface that lets you import decks and study using a variation of SuperMemo’s SM-2 algorithm. It has a small Gaussian randomness built into the easiness factor (decay constant). This might help determine if the algorithm should change.
I’m also worked on a way of sorting a language corpus. If you have sentences translated into another language, it’s not useful to have a long sentence presented, when you don’t know ANY of the vocabulary. Effectively, the algorithm runs through the items and sorts the items so that the challenge/novelty is fairly constant; the number of new words is distributed evenly over the entire set. As you might guess, this is iterative and still processor intensive… so I need to optimize it some more before letting web users run that function.
The other function still in the pipeline is a ‘remember on but not after’ deadline. I think this would only be useful for students, who need to remember something for testing, but can then forget it and use the internet if it ever comes up again.
Anyway, if interested: http://www.superbrain.me
http://www.superbrain.me looks useful. How does it differ from Anki—just that it’s web-based rather than needing to be installed? (I’m wondering how well it will work for those of us on slow connections.
I see from doing a site-search with Google that you can export to CSV—nice. I assume it’s possible to import the same way? And it looks like we can import from Anki—very nice.
Downloaded and set up with a couple of Divia’s decks. How many decks do you recommend working through at one time? For reference, I’m currently doing one deck on the default settings, which works out to ~40 cards a day (20 new, ~20 review) and takes 5-7 minutes.
It depends on how much time you are willing to devote. I spend 0-3 minutes per day on the cognitive biases deck which allows me to spend the additional 17-30 minutes I have allocated for Anki working on the NVC deck, which is massive. I plan to add a spanish vocab deck soon.
I’m liking this—A nice, practical rationality implementation technique.
In a broader sense, perhaps if one had the time, it might be good to have Agile, five forces and Business Generation Models on cards too. I’d posit those consulting-style problem frameworks would have more real world value if one could summon them for use with any given situation without the paper/screen aid.
Might try that cognitive bias stack first! (Am already carrying BGM frameworks in my notebook)
The maths sub-Reddit had a post on maths flash cards which prompted me to write a long comment worrying about the relationship between memorisation and understanding.
One thing I’ve thought would be good to have is a program that takes math formulas and damages them, to produce plausible, similar-looking formulas but with terms missing or altered. This would be used to make a set of flash cards where you have to distinguish between real and damaged formulas.
I have written a program (in the form of a web page) which does a specialized form of this. It has a set of generators of formulas and damaged formulas, and presents you with a list containing several formulas of the same type (e.g. ∫ 2x dx = x^2 + C) but with one damaged (e.g. ∫ 2x dx = 2x^2 + C).
You have to choose the incorrect formula (on the principle that finding errors in mostly-correct reasoning is more challenging and relevant than the usual multiple-choice approach of a bunch of often-obviously wrong items), and you are scored on the logarithm of the number of choices left when you pick the right item — choosing an undamaged item does not make you fail the problem set. This is not based on any particular model of learning; it’s just what I decided would minimize the annoyance/tedium of such a quiz for me.
It’s quite lacking in features and good architecture in its current state, as I only worked on it for a couple of days, but I just published the source code; you can also play it online in its current state.
There are no instructions and no options; all you can do is do your best to choose only incorrect formulas. It only generates integer factoring and differentiation-or-indefinite-integration problems.
I think that you shouldn’t keep false formulas so as to not accidentally learn them. In general, this sounds like you could hit on memetically strong corruptions which could contaminate your knowledge.
I’d prefer counter-examples over damaged formulas. For example, consider the theorem that a continuous function on the interval [a,b] is bounded. The damaged formula might read “If f is continuous on (a,b), then it is bounded.” There is some merit in spotting that (a,b) doesn’t include the end point a. There is greater merit in noticing that if we leave off the end point we have f(x) = 1/(x-a) as a continuous function that is unbounded.
That leads on to the difference between what one might call syntactic memorisation and semantic memorisation. Confronted with the claim “If f is continuous on (a,b), then it is bounded.” one might know it is supposed to be [a,b] as a matter of rote memorisation, but such knowledge is only a stepping stone. One wants to press on to a deeper understanding so that even if one forgets whether it is supposed to be (a,b) or [a,b] one can quickly reconstruct the memory by running over a counter-example (a,b) and providing a proof for [a,b]
Those sorts of flash cards would be valuable, but I don’t know how to construct them in an automated way, whereas damaging formulas is comparatively easy.
Should there be a quiz context which is different from the memorization context?
I do this by hand for my programming flashcards. Take a correct example, make a couple broken versions of it, and review. Whenever I can’t distinguish between right and wrong, make a few more that hammer on the distinction. As long as there’s enough that I memorize the principle or syntactic rule or semantic concept, and not the specific examples themselves...
I really like that idea.
Thanks for the mention of Anki. I want to learn Spanish, partly to keep my brain healthy, and partly because the experience of communicating in another language and culture is very satisfying for me. (I already speak Indonesian fluently.)
Downloading AnkiDroid now—I’ll try putting all my new vocab straight into the app.
I love the idea of using something like this for NVC, too—I find NVC very useful, but I always struggle to remember the finer details. Will download those decks. Thank you!