Spaced Repetition Database for the Mysterious Answers to Mysterious Questions Sequence
I’m a big fan of spaced repetition software. There’s a lot I could say about how awesome I think it is and how much it has helped me, but the SuperMemo website covers the benefits better than I could. I will mention two things that surprised me. First, I had no idea how much fun it would be; I actually really enjoy doing the reviews every day. (For me this is hugely important, since it’s unlikely I would have kept up with it otherwise.) Second, it’s proven more useful than I had anticipated for maintaining coherence of beliefs across emotional states.
I’ve tried memorizing a variety types of things such as emacs commands, my favorite quotations, advice about how to communicate with children, and characters from books. One of my more recent projects has been making notecards of the lesswrong sequences. I tried to follow the rules for formulating knowledge from the SuperMemo website, but deciding which bits to encode and how is subjective. For reference, I asked my boyfriend to make a few too so we could compare, and his looked pretty different from mine.
So, with those caveats, I thought I might as well share what I’d come up with. As Paul Buchheit says, “‘Good enough’ is the enemy of ‘At all’”. If you download Anki, my favorite spaced repetition software (free and cross-platform) and go to Download > Shared Deck in the Menu, you should be able to search for and get my Less Wrong Sequences cards. I also put them up here, with the ones my boyfriend made of the first post for comparison.
I had read all the sequences before, but I have found that since I’ve started using the cards I’ve noticed the concepts coming up in my life more often, so I think my experiment has been useful.
Let me know what you think!
- Spaced Repetition Database for A Human’s Guide to Words by (10 Jan 2011 0:21 UTC; 48 points)
- A Challenge for LessWrong by (29 Jun 2010 23:47 UTC; 21 points)
- Thinking in Bayes: Light by (10 Oct 2011 4:08 UTC; 12 points)
- 's comment on Open Thread, August 2010-- part 2 by (10 Aug 2010 17:35 UTC; 3 points)
- 's comment on Learning and Internalizing the Lessons from the Sequences by (14 Sep 2016 17:11 UTC; 2 points)
- 's comment on Open Thread, September, 2010-- part 2 by (23 Nov 2010 1:06 UTC; 1 point)
- 's comment on Anki decks by LW users by (3 Apr 2013 10:14 UTC; 0 points)
I’m happy to see something about Anki here, but how are Sequences supposed to be SRS-able in any way? “Learning” them involves understanding, not memorization of text / facts.
Or is that the joke (“mysterious answers”) and I failed to see it?
That was my first reaction as well, until I noticed this phrase in the post which made me think:
Likewise, I’ve read all the sequences many times over, but the skills described there just don’t come up othen in real life. Sure, it might be because they aren’t very useful. But also it might be because I compartmentalized them and keep forgetting to apply them when opportunities arise. It might be worthwhile to try out spaced repetition to test that hypothesis.
On the other hand, Schelling’s ideas stuck with me after reading the book once and I immediately started applying them, seeing conflicts and precommitment moves everywhere. So maybe Eliezer’s findings really are useless for mortals :-)
del
I think your concern is a valid one, but that there’s also a solution. I think reviewing the sequences with the mindset of trying to guess a password would merely reinforce the misguided idea of verbal behavior having inherent truth value. And that’s why I wouldn’t even really use the word “memorization” to describe what I’m doing.
I think the way to “learn” the sequences is to practice applying the concepts all the time, which is more easily accomplished if you’re primed to have them pop into your mind at the right moment. And my experience has been that SRS has helped enable that for me.
While using supermemo I found it useful to put the sequence posts into the system and use the built in process for progressively reading and extracting the most important elements of the text (key paragraphs and sentences) for more frequent exposure.
Reading complex material through multiple exposures is an effective way of understanding concepts that are multiple inferential steps away and the process of actively extracting the key messages ensures a more complete understanding. Having supermemo handle the process of prompting me with material from a large ‘to read’ list also does away with huge akrasia problems (by narrowing the bottleneck down to ‘use supermemo at all’).
The supermemo documentation supplies tips on how to go about absorbing large volumes of material, filtering them by priority, breaking them down into concepts worth learning and, when appropriate, breaking that down into individual facts or concepts that can be prompted. When they cannot be broken down further it can just be useful to have paragraphs pop up as a reminder. Just that much exposure will be enough reminder to keep the concept fresh in the brain.
When using supermemo to help learn material from a dense texbook that I happened to consider to be particularly worth memorizing I ended up creating diagrams of some of the concepts and the repetition questions consisted of partially redacted versions of the diagram that prompted recall of whatever bit was missing.
I note that Anki doesn’t necessarily support some of these applications. SuperMemo itself is abysmally ugly and a pain in the arse to learn but the feature set is clearly that of an application created by someone who wanted to personally optimise his learning over diverse set of situations. It will be extremely frustrating for me if I migrate to a more polished but more specialised system.
I’ve wanted to try incremental reading myself, but not enough to install Windows on my Mac. I’m glad to hear you find it useful though—that makes me more likely to make a greater effort to experiment with it at some point in the future.
Great post, thanks! In these primitive times without knowledge pills, discussion of methods to improve our learning are very welcome. Here are some I know;
I found the SuperMemo discussed in this article pretty good for things like learning foreign words (I’m trying to pick up some Finnish vocabulary); I’m not sure how it would work for knowledge that cannot be so easily represented as A=B. I haven’t fully explored the potential though.
My personal favorite are audiobooks; it’s a great way to pick up a lot of qualitative knowledge (say, history or psychology), or ‘reading’ a lot of 19th-century literature. Librivox and friends are excellent resources for that. The great thing about audio books is that I can do something useful with my idle time (shopping, running, commuting etc)
For language learning, the Pimsleur is an interesting way; it’s about repeating simple phrases again and again—completely ignoring writing or grammar, but still a relatively quick way to be able to converse with locals (worked for me with Greek at least to some extent)
Memorization of random lists of things, like the ones mentioned in Mind Performance Hacks
Remembering the things to do; GTD and OrgMode are my main tools here.
Still, there is a whole universe of knowledge that cannot be acquired using such tools—procedural knowledge, deeply technical knowledge. I suppose there is no alternative to sitting down and excercise those gray cells; still, I’d be very interested if anyone has some new approaches there.
My favorite fast learning method is to have an urgent reason for learning something. Not only does it improve your focus and provide you with immediate practice, but it cuts through extraneous bullshit like nothing else.
Org Mode is amazing. I use it for notes and todo lists all the time, and it works nicely. But it’s also thoroughly ridiculous in terms of power. Did you know that it has an ASCII-art spreadsheet that can do symbolic integration? It’s true!
:-) Indeed—I regularly take the numbers in a table, do some statistical analysis on them use R and generate some graphs, which are then included in the PDF-export.
More cards. You can think of of such knowledge as simply having more facts—this is the old code/data tradeoff. (For example, multiplication can be an algorithm—or, it’s an infinitely large category of facts, such as 1x1=1, 1x2=2, 1x3=3...)
In practice, right now you would have to generate a large set of cards, but in the future SRS software will support cards which are programs; then it would be much easier to learn multiplication, say, or anything you could program (like Go life and death problems); see my comment on Reddit or look through mnemosyne-proj-users for my musings on the subject.
Any ideas for things to do while listening to audio books on a laptop? I can’t concentrate on reading and listening at the same time.
Video games. My choice is roguelikes like Nethack.
For me the combination is extremely immersive and a big time sink.
Tell me you have played ADOM. Love that game. Still play it a couple of times a year. I’ve Nethack a few times.
Sorry. I always had the impression that ADOM requires too much planning and skill system study. These days I like to play Nethack forks and Crawl. Currently attempting a 10-conduct tourist in Sporkhack.
I find simple spacial games can make it easier to absorb information than when listening with no other activity. Speeding the audio up also helps.
Personally, when I’m having access to a laptop, I’d like to do stuff that I can only with a computer, such as reading (web or just pdfs) or watch some videos. Those kind of activities do not go well together with listening to audio books...
So, I’d say it’s best to fill the time where there is little need to use the brain for other intellectual activities, such as when working out or doing the dishes. I do turn it off now and then though; it’s nice to have all that knowledge being force-fed to you, but of course we also need some time to do some actual thinking without any such distractions. Personally, when I’m having access to a laptop, I’d like to do stuff that I can only with a computer, such as reading (web or just pdfs) or watch some videos. Those kind of activities do not go well together with listening to audio books…
So, I’d say it’s best to get some portable audio player, and fill the time where there is little need to use the brain for other intellectual activities, such as when working out or doing the dishes. I do turn it off now and then though; it’s nice to have all that knowledge being force-fed to you, but of course we also need some time to do some actual thinking without any such distractions.
For the narrow field of knowledge where it applies, audio books are as close as it gets to a (slowly-working) knowledge pill, I suppose...
Put the audio books on an mp3 player. I’m studying japanese whilst doing the dishes. If a lesson goes over too much then I’ll may do a bit of revision whilst on the train or treadmill in the gym.
This is fantastic given that I am actually studying Japanese and it turns out Anki has very good support for this language. Thank you for posting this.
SRS software is very efficient, but still makes you think how terribly inefficient the brain is at remembering things. Wouldn’t it be better to just read a word list once, decide “this is important” and just remember it, instead of spending half an hour every day to make the words stick?
Maybe we aren’t very good at deciding what’s important. For our brains, important memories are memories that are accessed often. How else would you decide?
yes
This is great; would it be possible to link back to the sequence in question from the Flashcard itself?
Done! I uploaded a new version with links to the posts.
Now, even more very awesome ;)
Browsing the downloads section I found several pre prepared decks on pharmacology. This will come in handy, thanks!
I also note that they have a version for the iphone, which will be perfect for learning unobtrusively whenever I’m bored.
Hmm… Does Anki allow me to review all of my decks of study material simultaneously? Specifically opening up an individual topic of study defeats at least half of the purpose that I use spaced repetition for!
Anki also allows to tag cards, so instead of splitting your data bases to different decks, you can split them to different tags on a single deck. This way you can review them all together, as well as review specific tags if the need rises.
Excellent! Thanks Mark. That’s a feature that may tip the balance. I’ll certainly consider Anki for the iPhone app at least.
It’s a pretty standard feature. I mean, Mnemosyne has always had categories*, as has SuperMemo. I’m hardpressed to think of any serious SRS which doesn’t include categories or tags.
* instead of tags; tags are part of Mnemosyne 2.x
It doesn’t allow you to review all your decks simultaneously, but you can merge decks by importing one deck into another. http://ichi2.net/anki/wiki/FrequentlyAskedQuestions#How_can_I_merge_or_split_decks.3F
You should be able to do a “custom review”, which allows you to specify which cards to review, and you can tell it to show you cards from multiple decks
Does SuperMemo make any attempt to distinguish between learning “facts,” “concepts,” “relationships,” and “skills?”
I would be pretty astonished if the same studying routine yielded optimal results for all four of those categories.
Good questions (and something I’d like to see more research on myself.)
SuperMemo focuses on facts not skills. The very format of SuperMemo (and, more or less, the brain itself) is such that it relies on relationships so all the research can be safely said to apply to ‘facts and relationships’. It does not attempt to cover concepts as divorced from facts and relationships and it does not make specific allowances for learning skills at all.
Are you still using SuperMemo?
I’ve been using Mnemosyne for about a week now. I don’t have much interesting to remember at the moment, so I’ve been working on memorizing the Periodic Table (both numbers-symbols and order, so I can write out the whole thing to impress cute nerds). I feel compelled to post because I just had a really odd experience where something was in memory that I didn’t expect to be there- I had already memorized the symbol-number pair (both ways) for H-Ar (1-18) and just added the relationships (H->? He. ->He? H). For a number of them, I blanked until I said “ok, Si is 14 and 15 is P, so Si->P.” What was odd was that it worked- for some reason I was surprised that I was able to recall that 15 was P and that actually was right.
So- this works. Use it. You’ll be surprised by what you can remember.
This is really useful; thanks! I’ve been using Anki for little over a year now, and I’ve found it very useful for classes and learning programming. I really like this application, and I’d love to see any more decks that you happen to make. I’ll definitely start my own next time I go back and read through the archives.
Maybe the cards could be made even more effective if they asked you to come up with an application on the spot?
Hmm. I’m interested, but I’m not exactly sure what you’re envisioning. Could you elaborate? I have another deck with SAT grammar (because I’m an SAT tutor) and I have cards that ask me to come up with example sentences for common grammar mistakes. I have specific answers on the back of the cards, but I’ll mark them correct if I come up with anything that correctly demonstrates the principle. So maybe something analogous to that?
You could do that, or for a greater challenge you could mark the card as incorrect if you came up with anything but a new and original application.
I used Anki to study German and found it decently effective. It’s also the only notecard software I’ve used that I found I could customize as much as I wanted (for instance, having three or four “sides” to a card).
I’ve thought about an alternative (or complimentary) answer for the question ” Sequences + Spaced Repetition = ? “: Instead of this approach (which is basically distilling the posts into flashcards as per Incremental reading), how about a deck of cards with one linked post per card with further cards for particularly interesting/important points. When a card comes up for review, you read the linked post and send it to re-reading in the future. This can also be scaled to books, though that may be pushing it.
The Pro is that its easier to implement, while still being better than just one read-through of the sequences. One big con is that there’s no active recall; no questions to answer, so it probably won’t be as much a part of you.
The point about this being a complementary solution is that you can slowly convert links to posts into proper Q&A flashcards, which basically makes this an intermediate stage between “read once” and “make an anki deck for a whole sequence at once”.
Now I didn’t actually implement this yet (I only found out about spaced repetition a few weeks ago), and I’d like a second opinion from someone who’s used SRS for a while on whether this idea will work or not.
This was golden! I had vaguely heard about the background theory behind this sort of memorization scheme before, but never had anyone point me to “the best easy thing” in this area. I love when people compress advice down to an actionable best-of-breed pointer :-)
I’ve downloaded Anki and played with it and it appears to be something that will make a substantial positive difference to my life. Thanks!
I hunted around and found flashcard sets for amino acids (none do exactly what I want and I may make my own “biohacker’s amino acids” set) and spanish vocabulary, but I bet there there are all kinds of amazing card sets out there. Any tips on optimal usage or cool card sets would be appreciated :-)
IIRC, Anki is supposed to be able to export in Mnemosyne’s .xml format. I’d like to take a look myself, and if it’s a good set of cards, I could also upload it to the Mnemosyne card collection as well.
I don’t see a way to export to xml, but if you want a tab delimited text file I could send you that. Interested?
Mnemosyne’s XML is nicer (and I’m surprised Anki can’t export as XML, it used to play nice with Mnemosyne) since it lets one specify metadata like grades, hardness, and category. But tab-delimited would work, yes.
http://divia.posterous.com/my-less-wrong-cards-as-a-tab-delimited-text-f
Alright, I’ve uploaded it at http://mnemosyne-proj.org/content/lesswrong-mysterious-answers-mysterious-questions-sequence
Incidentally, any chance of a copyedit? I noticed that there were quite a few typos.
Yeah, I’ve changed a few that I’ve noticed myself since I posted them, but if you want to email me with other changes I’d love that.
I’m not entirely sure how to contribute them back. Best would be to work against the master copy in Anki, but how would I get the changes and communicate them back to you?
I like it. So that was you at the London meetup then I take it? I’m installing the software now. Portability/open-ness is key—any kind of long-term thing must have data that’s easily exportable, right? And my current computer setup is really just chaff in the breeze, totally short-term.
I just had a look at Anki and the first screenshot put me off:
The repetition mechanism looks suspiciously ad-hoc to me. 6.5 months, 1.5 years or 3.5 years, a bunch of round figures that I choose myself by looking at those very numbers? That isn’t at all how I want my spaced repetition to behave. The supermemo software is rubbish but the algorithm it uses and makes public is quite good. Unless Anki has some similar algorithm that they use (and have done research on) I’ll be rather reluctant to switch!
You’re not supposed to choose by looking at the numbers, only decide if the fact was Hard / Good / Easy to remember.
Here is some information on the algorithm used by Anki, it’s modified from one of SuperMemo’s algorithms.
I’m looking forward to finding one of these applications that uses a later generation version of the supermemo algorithm. SM2 is ok but the research that was put in to the later ones wasn’t bad. The detailed manner in which it was able to infer your individual learning profile and adapt repetition accordingly was also rather elegant. Of course, I could probably program it in to the Anki software myself if I was really interested.
FWIW, the main developer of Mnemosyne) (the SRS I’ve used for the past 2-3 years) is skeptical that SM3 and up really add anything compared to SM2. (See “Principles” and various emails by Peter to the mnemosyne-proj-users ML.)
That makes sense to me, since early on the time granularity is ‘one day’, leaving little room for adjustment and after a couple reviews pushes items out to 100s of days, shifting forward or back a few days doesn’t make much of a difference.
I would certainly expect diminishing returns. The key seems to be spaced repetition itself and the environmental conditions that our learning (and forgetting) mechanisms are adapted for can hardly be considered to be exquisitely precise.
I would have to look more closely at the existing studies and most likely perform more myself before I could establish just how much scope there is for optimising the repetition schedule by either individual or type of knowledge.
If you are serious, you may find the Mnemosyne database torrent useful: http://www.reddit.com/r/cogsci/comments/9aufn/ever_wanted_to_analyze_860mb_of_spaced_repetition/
Wow. I am extremely tempted to download that and click ‘start’. I’ve been reading too much Harry Potter. What would make it worthwhile for me is if all the components maintained their deck structure and so could be easily removed in bulk if a couple of cards did not interest me.