Ultralearning Ch. 8 - Principle 5 - Retrieval

Test to Learn

We are given some background about the legendary mathematician Srinivasa Ramanujan.

The Testing Effect

Testing yourself — trying to retrieve the information without looking at the text — clearly outperformed all other conditions. The act of trying to summon up knowledge from memory is a powerful learning tool on its own, beyond its connection to direct practice or feedback. Without the answers at hand, Ramanujan was forced to invent his own solutions to the problems, retrieving information from his mind rather than reviewing it in a book.

The Paradox of Studying

Review or studying has low efficiency. Whether you are ready or not, retrieval practice works better. Especially if you combine retrieval with the ability to look up the answers, retrieval practice is a much better form of studying than the ones most students apply.

Is Difficulty Desirable?

More difficult retrieval leads to better learning, provided the act of retrieval is itself successful. Free recall test, in which students need to recall as much as they can remember without prompting, tend to result in better retention than cued recall tests, in which students need to recall as much as they can remember without prompting, tend to result in better retention than cued recall tests, in which students are given hints about what they. Need to remember. Cued recall tests, in turn, are better than recognition tests, in which students are given hints about what they need to remember. Cued recall tests, in turn, are better than recognition tests, such as multiple-choice answers, where the correct answer needs to be recognized but not generated.

Giving someone a test immediately after they learn something improves retention less than giving them a slight delay, long enough so that answers aren’t in mind when they need them. Difficulty, far from being an obstacle to making retrieval work, may be part of the reason it does so.

The idea of desirable difficulties in retrieval makes a potent case for the ultra learning strategy. Low-intensity learning strategies typically involve either less or easier retrieval. Pushing difficulty higher and opting for testing oneself well before you are “ready” is more efficient. Difficulty can become undesirable if it gets so hard that retrieval becomes impossible. Delaying the first test of a newly learned fact has some benefits over testing immediately. However, if you delay the test too long, the information may be forgotten entirely.

The idea, therefore, is to find the right midpoint: far enough away to make whatever is retrieved remembered deeply, not so far away that you’ve forgotten everything. Although waiting too long before you test yourself may have disadvantages, increasing difficulty by giving yourself fewer clues and prompts are likely helpful, provided that you can get some feedback on them later.

Should You Take the Final Exam before the Class even Begins?

The act of taking a test not only is a source of learning but results in more learning than a similar amount of time spent in review. However, this still fits within the conventional idea of knowledge being first acquired, and then strengthened or tested later.

An interesting observation from retrieval research, known as the forward-testing effect, shows that retrieval not only helps enhance what you’ve learned previously but can even help prepare you to learn better. Regular testing of previously studied information can make it easier to learn new information. This means that retrieval works to enhance future learning, even when there is nothing to retrieve yet!

The reality of the forward-testing effect implies that practicing retrieval might not only benefit from starting earlier than one is “ready” but even before you have the possibility of answering correctly (i.e. testing yourself on material you haven’t learned yet).

What should be retrieved?

The author discusses what should or shouldn’t be retrieved. This is an important practical question. Nobody has time to master everything. During the MIT Challenge, the author covered a lot of different ideas. Some were directly relevant to the kind of programming that he wanted to do when he was done, so making sure he retained those ideas was a priority. Others were interesting, but since he had no plans to use them immediately, he put more effort into practicing retrieving the underlying concepts than doing technical calculations.

One class he did, for instance, was Modal Logic. As he has no plans to be a logician, he can honestly say, eight years later, that he couldn’t prove theorems in modal logic today. However, he can tell you what modal logic is for and when it is used, so if a situation arises in which the techniques he learned in that class might be useful, he’d have a much better time spotting it.

There will always be some things you choose to master and others you satisfy yourself with knowing you can look up if you need to.

One way to answer this question is simply to do direct practice. Directness sidesteps this question by forcing you to retrieve the things that come up often in the course of using the skill. Things that are rarely used or that are easier to look up than to memorize won’t be retrieved. These tend to be the things that don’t matter so much.

The problem with relying on direct practice exclusively is that knowledge that isn’t in your head can’t be used to help you solve problems. For instance, a programmer may realize a need to use a certain function to solve a problem but forgets how to write it out. Needing to look up the syntax might slow her down, but she will still be able to solve the problem. However, if you don’t have enough knowledge stored to recognize when you can use a function to solve your problem, no looking up can help you.

Being able to look things up is certainly an advantage, but without a certain amount of knowledge inside your head, it doesn’t help you solve hard problems.

This makes sense, as a programmer. I’m glad I’m taking the time to store extra knowledge in my head to improve my chances of making the right lookups.

Direct practice alone can fail to encourage enough retrieval by omitting knowledge that can help you solve a problem but isn’t strictly necessary to do so. Consider our programmer who has two different ways to solve her problem, A and B. Option A is much more effective, but B will also get the job done. Now suppose that she knows only about option B. She’ll continue to use the way she knows to solve the problem, even though it is less effect.

Here, our fledgling programmer might read about option A on a blog somewhere. But since simply reading is much less effective than repeated retrieval practice, chances are that she’ll forget about it when it comes time to apply the technique. This may sound abstract, but I’d argue that this is quite common with programmers, and often the thing separating mediocre programmers from great ones isn’t the range of problems they can solve but that the latter often know dozens of ways to solve problems and can select the best one for each situation. This kind of breadth requires a certain amount of passive exposure, which in turn benefits from retrieval practice.

How to Practice Retrieval

Retrieval works, but it isn’t always easy. Here are some useful methods that can be used to apply retrieval to almost any subject.

Tactic 1: Flash Cards

Flash cards are an amazingly simple, yet effective, way to learn paired associations between questions and answers. The old way of creating paper flash cards to drill yourself is powerful, but it has largely been superseded by spaced-repetition systems, as I’ll discuss in Principle 7.

The major drawback of flash cards is that they work really well for a specific type of retrieval — when there’s a pairing between a specific cue and a particular response. When the situation in which you need to remember the information is highly variable, this kind of practice can have drawbacks. Programmers can memorize syntax via flash cards, but concepts that need to be applied in real programs often don’t fit the cue-response framework that flash cards demand.

Tactic 2: Free Recall

A simple tactic for applying retrieval is, after reading a section from a book or sitting through a lecture, to try to write down everything you can remember on a blank piece of paper. Free recall like this is often very difficult, and there will be many things missed, even if you just finished reading the text in question. However, this difficulty is also a good reason why this practice is helpful. By forcing yourself to recall the main points and arguments, you’ll be able to remember them better later. You can do this immediately, or after a break of 10-15 minutes, or the next day.

Tactic 3: The Question-Book Method

Another strategy for taking notes is to rephrase what you’ve recorded as questions to be answered later. Instead of writing that the Magna Carta was signed in 1215, you could instead write the question “When was the Magna Carta signed?” With a reference to where to find the answer in case you forget. By taking notes as questions instead of answers, you generate the material to practice retrieval on later.

One mistake the author has made in applying this technique is to focus on the wrong kinds of things to ask questions about. That wasn’t intentional but rather a by-product of lazily restating the factual content in the book as questions. What’s harder and more useful is to restate the big idea of a chapter or section as a question. Since this is often implicit, it requires some deeper thinking and not just adding a question mark to some notes you copied verbatim. One rule the author has found helpful for this is to restrict yourself to one question per section of a text, thus forcing yourself to acknowledge and rephrase the main point rather than zoom icon a detail that will be largely irrelevant later.

Cornell-style notes are a good example of this note-taking process.

Tactic 4: Self-Generated Challenges

The above tactics work best with retrieval of simple information, such as facts or summaries of broad ideas you might encounter in a book or lecture. However, fi you’re trying to practice a skill, not merely remember information, they might not be enough. For a programmer, it’s not enough to know what an algorithm means, but be able to write it in code. In this case, as you go through your passive material, you can create challenges for yourself to solve later. You may encounter a new technique and then write a note to demonstrate that technique in an actual example. Creating a list of such challenges can serve as a prompt for mastering that information later in practice and can expand your library of tools that you are able to actually apply.

Tactic 5: Closed-Book Learning

Nearly any learning activity can become an opportunity for retrieval if you cut off the ability to search for hints. Concept mapping, for example, could be helpful if you do this without looking at the book while generating your concept map. Any practice, whether direct or a drill, can be cut off from the ability to look things up. By preventing yourself from consulting the source, the information becomes knowledge stored inside your head instead of inside a reference manual.

Revisiting Ramanujan

Ramanujan’s genius was aided immeasurably by two hallmarks of the ultra learner’s tool kit: obsessive intensity and retrieval practice. Retrieval is not a sufficient tool to create genius, but it may be a necessary one.

Trying to produce the answer rather than merely reviewing it is only half of a bigger cycle, however. To make retrieval effective, it helps to know whether the answer you dredged up from your mind was correct. That’s where feedback comes in.