Instrumental Rationality 2: Planning 101

lifelonglearner6 Oct 2017 14:23 UTC
17 points
4 comments14 min readLW link

[Instrumental Rationality sequence 2⁄7.]

[This section goes over the planning fallacy, our cognitive bias of making overconfident predictions in our time estimates. It starts with an overview of the field and moves into some models of how human planning works. We’ll move into three techniques to plan better and end with some more practical suggestions.]

Introducing the Planning Fallacy:

[We go over the basics of how humans can make overconfident predictions in their planning. Some basic statistics to maybe scare you a little bit.]

Humans are often overconfident, and perhaps for good reason.

Back on the savannah, overconfidence might have been an effective strategy for success. If you gave off the impression of being more capable than you really were, then being overconfident and bluffing could frighten stronger opponents and avoid direct conflict.

When it comes to making plans in the modern day, however, overconfidence in planning can be unhelpful. You might be able to convince everyone (including yourself) that you’ll finish that report in three days, but, if you don’t put in the effort, it’ll still really take you a week.

There’s a lot of evidence that suggests that our thinking is often subject to the planning fallacy, a tendency to make unrealistic predictions and plans.

Below is a scattering of stats and examples from different fields to guide your intuitions at the phenomenon I’m talking about:

First, some students were asked to predict when they were 99% sure they’d finish a project. But when the researchers actually followed up with them, we found that only about 45%, less than half of the students, had actually finished by their own predicted times *1.

In a related study, students were asked to predict when they’d finish, “assuming everything went as poor as it possibly could.” Basically, their worst-case scenario. Yet, only about 30% of students finished by their own self-appointed estimate *2.

In fact, similar results were also found in Japanese and Canadian cultures, giving evidence that this is a human (and not US-culture-based) phenomenon. Students continued to make overly optimistic predictions, even when they knew the task had taken them longer last time *3.

I don’t mean to just pick on students, though. The planning fallacy is present in many, many sectors.

For example, an overview of public transportation projects found that most of them were, on average, 20-45% above the estimated cost. Research has shown that these poor predictions haven’t improved at all in the past 30 years *4.

And there’s no shortage of high-profile examples, from the Scottish Parliament Building, which cost 10 times more than expected, to the Denver International Airport, which, by some estimates, took 16 months longer and $2 billion more than initially anticipated.

Other fields like tech and finance have their own share of overconfident estimates which fail to come true, from the information technology sector’s roughly 33% success rate for on-time project completion, to CFOs making having grossly overconfident judgments *5 *6.

If you couple the above data with some of your own experiences with far-too optimistic predictions, this hopefully paints a picture of how reality consistently fails to meet our expectations.

My goal here is to drive home the point that you, just like everyone else, are also susceptible to these errors in planning.

++++

Modeling the Planning Fallacy:

[We go over several models of how to think about the brain and the estimates it makes. I think that the inside /​ outside view distinction is the most useful one, and it’ll be the one I focus on]

Here, we’ll go over some simplified models of what’s happening inside our heads when the planning fallacy occurs.

First, there are some fairly basic potential explanations for our overly optimistic estimates.

For one, we might just be misremembering the actual details of how things played out in the past. It seems that we may underestimate how long things take us, and this can even occur in our memories, which leads us to make messed-up estimates *7.

On another level, it seems that we just don’t expect things to go wrong. Studies have found that we’re biased towards not looking at pessimistic scenarios *8.

There are lots of unknown unknowns in any situation, things we just won’t see coming. Yet, even if we know that unexpected events will occur, this doesn’t move us to take extra precautions.

We often just assume the best-case scenario when making plans, and we might miss out on relevant outside information.

This leads us to the model of the inside view and the outside view, which is what I think to be one of the best models for explaining this bias *9.

The inside view is the information you have about your specific project, or information inside your head. It’s the information that only someone inside the project would have.

Here are some examples of inside view thinking:

  1. You are writing an essay. You consider your topic, how much you know about the topic, and perhaps how much ink you have left in your pen. Then you predict how long you think it’ll take to finish writing the essay.

  2. You are running in a marathon. Halfway through, you check your energy level, how sore your muscles are, and if you’re hydrated. Then you make a prediction about what place you’ll finish.

  3. You just finished a very difficult math quiz. You think about the specific problems you had trouble with, the problems you felt were easy, and then you try to predict what score you’ll get on the quiz.

The inside view tends to focus on you. It’s about your strengths, abilities, limits, and things within your control. However, as any person in a traffic jam can tell you, it’s often the things beyond our control that have the greatest impact.

In these cases, the best we can do is to adapt to them as they show up.

In general, we seem to use inside view thinking when we make plans, and this is the source of our overconfidence. We’re focused on how we can help our project go right, rather than the things that can make our project go wrong.

This intuitive planning strategy can miss out on the unknown unknowns that are out there in the real world, leading us to make plans which fail to meet reality’s standards.

This is where the outside view comes in.

The outside view is information about all projects similar to yours that have happened, or what someone on the outside might say about your project. It’s about how well you are doing, compared to others doing something similar.

Where the inside view focuses on your individual strengths, the outside view looks at averages, the environment, and the things around you.

Here are some examples of outside view thinking applied to the above examples:

  1. You are writing an essay. Instead of considering the specific topic, you look back to how long it took you to write essays in the past. On average, you finished within 40-50 minutes, so you predict that you’ll probably take about that long this time.

  2. You are running in a marathon. Even though you feel pretty good, you see that there are already 4 people ahead of you. Also, you finished 7th in the last few marathons. So, chances are, you’ll probably finish 6th or 7th place this time, despite what your body is signaling to you.

  3. You just finished a very difficult math quiz. The teacher said that most of the class got close to a 60%. Even though you felt like you knew most of the answers, you predict that you probably also got close to 60%, maybe a little higher.

With these examples, I hope you get the basic idea of the distinction between the two ways of making plans.

Relating it back to the typical System 1 and System 2 model, the inside view is closer to a S1 process, as it’s our intuitive, default planning strategy. The outside view requires some additional thought (and perhaps research), making it more of a S2-type process.

++++

3 Techniques to Improve Planning:

We’ll be covering three techniques to help with improving your plans: Murphyjitsu, Reference Class Forecasting (RCF), and Back-planning (aka “backchaining”).

  1. Murphyjitsu is a little like using your brain’s built-in system to analyze for potential problems.

  2. RCF is flat-out using past history to improve your predictions.

  3. Back-planning is like planning. Except it’s done backwards.

++++

Murphyjitsu:

[Murphyjitsu is about using your intuitions to check for whether or not a plan will work. It draws on your ability to implicitly store information in your gut feelings and is way less magical than it sounds.]

Research:

The name Murphyjitsu comes from the infamous Murphy’s Law: “Anything that can go wrong, will go wrong.”

Murphyjitsu is based off a strategy called a “premortem” or “prospective hindsight”, which basically means imagining the project has already failed and looking backwards to see what went wrong *10.

But that can get a little complicated (“So you imagine yourself...in the future...looking back at the past...to figure out how to act...in the present?).

So here’s an alternative fun way to think of Murphyjitsu:

Say you’re sitting at your desk, getting ready to write a report on intertemporal travel. You’re confident you can finish before the hour is over. What could go wrong? Closing Facebook, you begin to start typing.

Suddenly, you hear a loud CRACK!

A burst of light floods your room as a figure pops into existence, dark and silhouetted by the brightness behind it. The light recedes, and the figure crumples to the ground. Floating in the air is a whirring gizmo, filled with turning gears. Strangely enough, your attention is drawn from the gizmo to the person on the ground:

The figure has a familiar sort of shape. You approach, tentatively, and find the splitting image of yourself! The person stirs and speaks.

“I’m you from one week into the future,” your future self croaks. Your future self tries to tries to get up, but sinks down again.

“Oh,” you say.

“I came from the future to tell you…” your temporal clone says in a scratched voice.

“To tell me what?” you ask. Already, you can see the whispers of a scenario forming in your head…

Future You slowly says, “To tell you… that the report on intertemporal travel that you were going to write… won’t go as planned at all. It failed.”

“Oh no!” you say.

Somehow, though, you aren’t surprised…

At this point, what plausible reasons for your failure come to mind?

For example, if you’re trying to write a blog post, you could imagine that you woke up the next day and no words got written. When in that scenario, what is the plausible explanation which explains things?

Maybe it’s that you stayed out too late with friends, or that you forgot about the task, or that you left your power cord in the office.

Whatever explanation, that is likely to be one of the most likely things that will derail your blog post tonight. Now that you’ve done Murphyjitsu, you can try to patch the problem by perhaps telling your friends that you need to get home by a certain time.

It turns out that putting ourselves in the future and looking back can help identify more risks, or see where things can go wrong. Prospective hindsight has been shown to increase our predictive power so we can make adjustments to our plans—before they fail *11.

In short, you’re trying to see likely potential failure modes, by doing a “surprise check” with your gut by asking yourself the question “Would I be surprised if I learned that this plan failed?”

It might feel a little weird to rely on your intuition or gut feeling to look for potential failure modes. But remember that it’s not just “coming out of nowhere”!. Your System 1 does have information. You’ve got a rich history of experiences to rely on (just from being alive!), and the Murphyjitsu procedure is designed to try and take advantage of those intuitions.

++++

Technique:

Here are the basic steps of Murphyjitsu:

  1. Figure out your goal. This is the thing you want to make plans to do.
    EX: “First, let’s say I decide to exercise every day. That’ll be my goal.”

  2. Write down which specific things you need to get done to make the thing happen. (Make a list.)
    EX: “But I should also be more specific than that, so it’s easier to tell what ‘exercising’ means. Let’s decide that I want to go running on odd days for 30 minutes and do strength training on even days for 20 minutes. And I want to do them in the evenings.”

  3. Now imagine it’s one week (or month) later, and yet you somehow didn’t manage to get started on your goal. (The visualization part here is important.) Are you surprised?
    EX: “Now, let’s imagine that it’s now one week later, and I didn’t go exercising at all! What went wrong?”

  4. Why? (What went wrong that got in your way?)
    EX: “The first thing that comes to mind is that I forgot to remind myself, and it just slipped out of my mind”

  5. Now imagine you take steps to remove the obstacle from Step 4.
    EX: “Well, what if I set some phone /​ email reminders? Is that good enough?”

  6. Return to Step 3. Are you still surprised that you’d fail? If so, your plan is probably good enough. (Don’t fool yourself!)
    EX: “Once again, let’s imagine it’s one week later and I made a reminder. But let’s say I still didn’t got exercising. How surprising is this?”

  7. If failure still seems likely, go through Steps 3-6 a few more times until you “problem proof” your plan.
    EX: “Hmm, I can see myself getting sore and/​or putting other priorities before it…(Step 4). So maybe I’ll also set aside the same time every day, so I can’t easily weasel out (Step 5).
    How do I feel now?     (Back to Step 3) Well, if once again I imagine it’s one week later and I once again failed, I’d be pretty surprised. My plan has two levels of fail-safes and I do want to do exercise anyway. Looks like it’s good! (Done)”

++++

Plan-Bot: An Automated Murphyjitsu Tool:

If you want to try out a very simple web-app that walks you through the Murphyjitsu prompt, I wrote an interactive series of question prompts that can be found here.

++++

Reference Class Forecasting (RCF):

[Reference Class Forecasting is about using past information to inform future estimates. It’s based off the assumption that getting a general sense for how plans for tasks similar to yours will help give you a more unbiased estimate.]

Research:

Reference class forecasting (RCF) is all about using the outside view rather than our optimistic inside view.

It’s about using past history to inform our future estimates.

Often, we’ll see all the ways that things can go right, but none of the ways things can go wrong. By looking at past history—other people who have tried the same or similar thing as us—we can get a better idea of how long things will really take.

Why does RCF do better than our naive planning processes? Basically, RCF works by looking only at results.

This means that we can avoid any potential biases that might have cropped up if we were to think it through normally. We’re shortcutting right to the data, i.e. what actually happened.

The rest of it is basic statistics; most people are close to average. So if we have an idea of what the average looks like, we can be sure we’ll be pretty close to average as well *12 *13.

When you Google the average time or look at your own data, you’re forming a “reference class”, i.e. a group of related things that can give you info about how long similar projects tend to take. Hence, the name “reference class forecasting”.

For example, if it usually takes me about 3 hours to finish homework (I use the web app Toggl to track my time), then I’ll predict that it will take me 3 hours today, too.

It’s obvious that RCF is incredibly simple. It literally just tells you that how long something will take you this time will be very close to how long it took you last time. But that doesn’t mean it’s ineffective!

Often, the past is a good benchmark of future performance, and it’s far better than any naive prediction your brain might spit out.

++++

Technique:

Here are the steps for RCF:

  1. Figure out what you want to do.
    EX: Brienne wants to design a logo for a nonprofit.

  2. See your records how long it took you last time.
    EX: Last time, Brienne took about three hours to make the logo.

  3. That’s your new prediction.
    EX: Brienne expects that it’ll also take her three hours this time around.

  4. If you don’t have past information, look for about how long it takes, on average, to do our thing. (This usually looks like Googling “average time to do X”.)
    EX: Brienne Googles “average time to design a logo” and comes up with an average of about 4 hours.

++++

RCF + Murphyjitsu Example:

In my own experience, I’ve found that using a mixture of Reference Class Forecasting and Murphyjitsu to be helpful for reducing overconfidence in my plans.

When starting projects, I will often ask myself, “What were the reasons that I failed last time?”

I then make a list of the first three or four “failure-modes” that I can recall. I now make plans to preemptively avoid those past errors.

(This can also be helpful in reverse—asking yourself, “How did I solve a similar difficult problem last time?” when facing a hard problem.)

Here’s an example:

Say I’m writing a long essay (like this one) and I want to know how what might go wrong. I’ve done several of these sorts of primers before, so I have a “reference class” of data to draw from. So what were the major reasons I fell behind for those posts?

<Cue thinking>

“Hmm…it looks like I would either forget about the project, get distracted, or lose motivation. Sometimes I’d want to do something else instead, or I wouldn’t be very focused. That’s definitely happened in the past.

Okay, great. Now what are some ways that I might be able to “patch” those problems?

Well, I can definitely start by making a priority list of my action items. So I know which things I want to finish first. I can also do short 5-minute planning sessions to make sure I’m actually writing. And I can do some more introspection to try and see what’s up with my motivation.”

<End thinking>

So, yeah, that’s a sort of snapshot of what my thoughts might look like in this situation.

++++

Back-Planning:

[Back-planning is a novel way of planning that starts from the end result and works backwards, as the name suggests. It’s also known as “backchaining” in some fields.]

Research:

Back-planning involves, as you might expect, planning from the end. Instead of thinking about where we start and how to move forward, we imagine we’re already at our goal and go back to the start.

The experimental evidence for back-planning basically suggests that, when using this technique, people will end up with more pessimistic (and thus realistic) time estimates.

Why? We’re not 100% sure.

The general gist of these theories is that back-planning is a weird, counter-intuitive way to think about things, which means it disrupts a lot of mental processes that can lead to overconfidence *14.

This means that back-planning can make it harder to fall into the groove of the easy “super optimistic best-case” planning we default to. Instead, we need to actually look at where things might go wrong. Which is, of course, what we want.

In my own experience, I’ve found that going through a quick back-planning session can help my intuitions “warm up” to my prediction more.

For example, I’ll sometimes use RCF to get a more accurate time estimate, but it still feels “off”. Walking through the plan through back-planning can help all the parts of me understand that it really will probably take longer.

++++

Technique:

Here are the steps for back-planning:

  1. Figure out the task you want to get done.
    EX: “Right now, I want to host a talk at my school. I know that’s the end goal.”

  2. Imagine you’re at the end of your task.
    EX: “So the end goal is me actually finishing the talk and taking questions.”

  3. Now move backwards, step-by-step. What is the step right before you finish?
    EX: “What happens right before that? Well, people would need to actually be in the room. And I would have needed a room.

  4. Repeat Step 3 until you get to where you are now.
    EX: “Is that all? (Step 3). Also, for people to show up, I would have needed publicity. Probably also something on social media. I’d need to publicize at least a week in advance, or else it won’t be common knowledge.
    And what about the actual talk? I would have needed slides, maybe memorize my talk. Also, I’d need to figure out what my talk is actually going to be on.”

  5. Write down how long you think the task will now take you.
    EX: “Huh, thinking it through like this, I’d need something like 3 weeks to get it done. One week for the actual slides, one week for publicity (at least), and one week for everything else that might go wrong.
    That feels more ‘right’ than my initial estimate of ‘I can do this by next week’.”

  6. You now have a detailed plan as well as a better prediction!
    EX: “Hooray!”

++++

Conclusion:

We’ve gone over several ways in which planning can hopefully be improved with some careful thinking. Hopefully I’ve showed that both your intuitions and reasoned analysis can provide useful data.

I think many of the considerations I outlined above in planning are great to keep in mind, but it’s also quite unrealistic to expect you to remember all of them when actually making plans.

I wholeheartedly endorse sacrificing some of the detail for speed. Finding a way to run these planning checks at the 5-second level is really what’s probably going to be really practical.

Happy planning!

++++

References (*):

1. Buehler, Roger, Dale Griffin, and Michael Ross. “Exploring the Planning Fallacy: Why People Underestimate their Task Completion Times.” Journal of Personality and Social Psychology 67.3 (1994): 366.

2. Buehler, Roger, Dale Griffin, and Michael Ross. “It’s About Time: Optimistic Predictions in Work and Love.” European Review of Social Psychology Vol. 6, (1995): 1–32.

3. Buehler, Roger, Dale Griffin. “Planning, Personality, and Prediction: The Role of Future Focus in Optimistic Time Predictions.” Organizational Behavior and Human Decision Processes, 92, (2003) 80–90

4. Flyvbjerg, Bent. “From Nobel Prize to Project Management: Getting Risks Right.” Project Management Journal 37.3 (2006): 5-15. Social Science Research Network.

5. Ben-David, Itzhak, John R. Graham, and Campbell R. Harvey. “Managerial Miscalibration.” No. w16215. Quarterly Journal of Economics. (2010). Social Science Research Network.

6. The Standish Group. “The Standish Group CHAOS Report.” Project Smart (2014).

7. Roy, Michael M., Nicholas JS Christenfeld, and Craig RM McKenzie. “Underestimating the Duration of Future Events: Memory Incorrectly Used or Memory Bias?.” Psychological Bulletin 131.5 (2005): 738.

8. Newby-Clark, Ian R., et al. “People Focus on Optimistic Scenarios and Disregard Pessimistic Scenarios While Predicting Task Completion Times.” Journal of Experimental Psychology: Applied 6.3 (2000): 171.

9. Kahneman, Daniel, and Dan Lovallo. “Timid Choices and Bold Forecasts: A Cognitive Perspective on Risk Taking.” Management Science 39.1 (1993): 17-31.

10. Klein, Gary. “Performing a Project Premortem.” Harvard Business Review 85.9 (2007): 18-19.

11. Veinott, Beth. “Klein, and Sterling Wiggins,“Evaluating the Effectiveness of the Premortem Technique on Plan Confidence,”.” Proceedings of the 7th International ISCRAM Conference (May, 2010).

12. Flyvbjerg, Bent. “From Nobel Prize to Project Management: Getting Risks Right.” Project Management Journal 37.3 (2006): 5-15. Social Science Research Network.

13. Flyvbjerg, Bent. “Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice.” European Planning Studies 16.1 (2008): 3-21.

14. Wiese, Jessica, Roger Buehler, and Dale Griffin. “Backward Planning: Effects of Planning Direction on Predictions of Task Completion Time.” Judgment and Decision Making 11.2 (2016): 147.

Next essay

lifelonglearner6 Oct 2017 14:23 UTC
17 points
4 comments14 min readLW link