Part one: what is this book and should you read it?
Predicting the Next President: The Keys to the White House (henceforth rendered “The Keys”) is an ambitious book. Penned by historian Allan Lichtman, The Keys explains his system for forecasting the outcome of US presidential elections. This system is also called “The Keys”. So, to help readers figure out whether I’m talking about the book or the system, I’m only going to italicize The Keys when I’m explicitly referring to the book. The Keys is a 13-strong checklist of “true-or-false” statements that measure how well the incumbent party has performed during its term in office. If six or more statements are judged to be “false” then the election goes to the challenger. Five or fewer, and the incumbent takes it.
According to Lichtman, The Keys reflect a number of axiomatic truths about the democratic system in the US. And although he’s keen to caution against superposing these truths non-reflectively on democratic systems in other countries, he also believes The Keys speak to political processes in liberal democracies more generally.
In his introductory chapter, Lichtman describes a “pragmatic” American electorate that chooses a president “according to the performance of the party holding the White House” and theorises: “If candidates and the media could come to understand that governing, not campaigning, counts in presidential elections, we could have a new kind of presidential politics.” (Lichtman, 2016, pp. 8). This new kind of politics would dump the attack ads and go deep on policy promises, outlining substantial agendas of what candidates actually planned to do with their desired four years in office.
I should be clear from the outset that I think Lichtman’s vision is an appealing one. I also think The Keys is well written, and The Keys is well conceived. Assuming you’re in the target audience, you will probably enjoy reading this book, and might come away from it with something useful. What is the target audience? Probably people who enjoy reading long copy about predictive methodology and political history (given where you’re reading this, I don’t think it’s a stretch to assume that includes you, dear reader).
With all that out of the way, let me tell you what’s coming next. In part two, I’ll give a brief overview of what’s in this book and how I’m going to address it. In the third section, I’m going to break down each of the 13 keys and explain how you can use them to create your own electoral forecast. Then, in part four, I’ll look at some of the criticisms levelled against The Keys and give my own brief critique. Finally, in part five, I’ll offer my opinion on the both the book and the system.
If you don’t want to wade through all of that, here’s the “TLDR” version: I like The Keys, but I think it’s ultimately a flawed predictive methodology that doesn’t live up to its own hype. And I like The Keys but I take issue with some of the claims Lichtman makes and (what I think is) a sometimes deceptively rhetorical style of writing. However, these issues don’t quite manage to make the book unenjoyable or the system it describes unworkable. So, if it sounds like your cup of tea, I suggest you take the plunge and start reading.
Part two: mapping the structure of this review to the structure of The Keys
One of the best things about The Keys is how straightforward it is: it’s just 13 true or false statements. Five or fewer false answers, and the incumbent keeps the White House, six or more and they lose it. The Keys is also straightforward, giving you a full rundown of all 13 keys in chapter two, which I’ll summarise in the next section. But before I get to that, let me briefly sketch the rest of the book’s structure out, and explain how much attention I give to each bit.
Chapter one deals with most of the theory (we’ll discuss this theory in parts three, four, and five of this review).
Chapters three to 11 are Professor Lichtman’s reading of American political history in relation to The Keys, which I’ll only mention in passing. I understand that these chapters are going to be a draw for some readers, but I found them a slog. To me – who came here for the theory and the method – these sections read as filler. Let me be clear, this is not fair to Professor Lichtman, who writes well and knows his subject, hence the “to me” qualifier in the previous sentence. But, from an operational perspective, there is nothing you need to read in them – all they offer is historic example after historic example of The Keys aligning with presidential election outcomes (and, of course, some insight into Lichtman’s own view of American political history).
Chapter 12 is Professor Lichtman’s prediction for the 2016 US presidential election, which I’ll talk about in part four of this review (remember, this is the 2016 edition, so it was published before Mr. Trump took the White House).
Finally, chapter 13 outlines what Professor Lichtman thinks the “lessons” of the 13 Keys are for political candidates. Despite its relative brevity, this is one of the most important chapters in the book, and it will feature pretty heavily in part five of this review.
Part three: the 13 Keys to the White House
Without further ado, here are the 13 Keys. This is a condensed and only-slightly paraphrased list (I will never in good conscience use the word “listicle”) taken from chapter two of The Keys. Each key is titled as-per Professor Lichtman’s original, and contains a statement which you have to decide is true or false. I’ve addressed scoring The Keys above, but let me reiterate it here: once you’ve assessed the veracity of each statement, all you need to do is count up how many “false” results you’ve got. Six or more, and the challenger wins the White House. Five or fewer, and the incumbent does.
The Incumbent-Party Mandate: The incumbent party holds more seats in the House of Representatives after the mid-term elections than it did after the previous mid-term elections.
Nomination Contest: The sitting president doesn’t face any serious competition for his party’s nomination.
Incumbency: The sitting president is the incumbent party’s candidate.
Third Party: The race is between “the big two” with no serious third-party campaign.
Short-term Economy: There’s no recession during the election campaign.
Long-term Economy: Real annual per capita GDP growth during the current term equals or exceeds mean growth in the previous two terms (for the economically challenged, this basically means the US economy has been growing as fast, or faster, during the term in which the election takes place as the two terms that directly precede it).
Policy Change: The incumbent makes major changes in national policy (Lichtman defines what a “major” change would look like: “redirecting the course of government or […] innovating new policies and programs that have broad effects on the nation’s commerce, welfare, or outlook.”) (Lichtman, 2016, pp. 53).
Social Unrest: There is no “sustained social unrest” during the incumbent’s term in office (Lichtman, 2016, pp. 55).
Scandal: The incumbent administration hasn’t had to contend with a major scandal (what counts as “major”? “The wrongdoing has to bring discredit upon the president himself, calling into question his personal integrity, or at least his faithfulness in upholding the law.”) (Lichtman, 2016, pp. 57).
Foreign or Military Failures: The incumbent administration hasn’t suffered a major foreign policy or military failure (there’s that word “major” again. This time it means: “a major disaster that appears to undermine America’s national interests or seriously diminish its standing in the world.”) (Lichtman, 2016, pp. 60).
Foreign or Military Successes: The incumbent administration has made a major foreign policy or military success (major: “perceived as dramatically improving the nation’s interests or prestige.”) (Lichtman, 2016, pp. 62 – 63).
Incumbent Charisma: The incumbent-party candidate is either immensely charismatic or a national hero (“an extraordinarily persuasive or dynamic candidate, or one who has attained heroic status through achievements prior to his nomination as a presidential candidate.”) (Lichtman, 2016, pp. 65).
Challenger Charisma: The challenging-party candidate is neither immensely charismatic nor a national hero (see Key 12’s description in parenthesis, but remember that Key 13 is claiming the challenger is not charismatic or a hero, whereas Key 12 is claiming the incumbent is one or both of these things).
It’s worth going back and re-reading these carefully, because some of the phrasing is a little awkward. This is an artifact of Lichtman’s decision to render The Keys as a series of true or false statements that are sometimes opposed to one another, and you can’t really write around it.
In Chapter two of The Keys Lichtman gives us pages of explanation for each key. It’s worth reading, and it’s interesting reading… but it’s not essential reading. You can use his predictive methodology with nothing more than the list above, provided you are able to figure out whether each statement given is true or false.
Part four: critiquing The Keys
On August 31, 2011, renowned political forecaster and number cruncher Nate Silver penned a post on FiveThirtyEight with the headline: Despite Keys, Obama Is No Lock. Although titularly concerned with then-incumbent president Barack Obama’s chances in the 2012 presidential election, the post is more noteworthy today for its blistering critique of The Keys: “Superficially quite impressive. But ‘superficial’ is the crucial term here.” (Silver, 2011, online).
What is Silver talking about specifically? Professor Lichtman’s claim that The Keys “called the winner” of every election since 1860 (Silver again: “That’s 38 elections in a row!”). Nate runs through “several problems with this model” noting first “the nature of the keys […] several of which are quite subjective”. Silver admits, for example, that “candidate quality” is a thing, but notes that it can be a pretty amorphous one. Why, he asks, was Mr Obama scored as charismatic in 2008 but not in 2012? Shouldn’t John McCain have qualified as a war hero in 2008? Why doesn’t Afghanistan count as a foreign policy failure, given that it’s unpopular with Americans?
Professor Lichtman responded to Silver a week later, claiming “his critique of the keys system cannot withstand scrutiny” (Lichtman, 2011, online) According to Lichtman, his “published definition” of the challenger charisma/hero key “rules out counting McCain as a national hero since he did not lead the nation through war, however admirable.” He continues to claim “in [the] first edition of the Keys to the White House published many years before the 2008 election, I wrote that candidates attain heroic stature only ‘through vital leadership in war like Ulysses Grant and Dwight Eisenhower.’” (Lichtman, 2011, online).
This is probably true – I don’t have the first edition to hand. However, I do have the 2016 edition, and I’m going to quote the passages from both key 12 and key 13 that apply here. To be clear, these are the only passages in the description of these keys that deal with the subject (heroism) under discussion. I apologise for the length of these quotes – I’m including them because I think it’s really important to illustrate a wider criticism I want to make:
“There have been only two clearly heroic nominees, both of whom attained this stature through vital leadership in war: Ulysses Grant and Dwight Eisenhower. Many other candidates, including William McKinley, George McGovern, and George H. W. Bush have had impressive military records but have fallen far short of the heroic status attained by Grant and Eisenhower. Of all presidential candidates since 1860, only Theodore Roosevelt, the leader of the “Rough Riders” during the Spanish-American War, combined personal charisma with near-heroic accomplishment.” (Lichtman, 2016, pp. 65).
“Heroic stature is easier to identify. To meet the threshold, a candidate’s achievement must be deemed critical to the nation’s success in an important endeavour, and probably should be of relatively recent vintage. New Jersey senator Bill Bradley’s career as a highly acclaimed professional basketball player, while appealing to a large segment of the population, does not make him a national hero. Ohio senator John Glenn’s achievement as the first American astronaut to orbit the earth might have conferred that status early on, but probably would no longer have secured the key had he been nominated in 1984.” (Lichtman, 2016, pp. 67).
Now, dear reader, I am going to make a prediction of my own that relies on nothing but my faith in your reading comprehension skills: you’re wondering “where, in the above quotes, does Professor Lichtman explicitly state that vital leadership in war is required to turn the ‘hero’ key?” Well, let me answer you: nowhere. He explicitly states it nowhere.
Now, you might also be thinking it’s unfair of me to quote the 2016 version of The Keys, given that this criticism and response dates back to 2011. But that leads us on to the following question: why is Professor Lichtman quoting himself from 1996, when the most recent edition of his work (at the time) was the 2008 edition? Thanks to the magic of Google Books, I can take a pretty good guess: because the text in the 2008 version is exactly the same as the text in the 2016 version.
In other words, Silver’s criticism is correct – it is not clear whether McCain (who won the Silver Star and Purple Heart, among other wartime baubles) should have turned the ‘hero’ key or not. And Lichtman’s response (in my opinion) is disingenuous. He quoted an out-of-date version of his own work to create the impression that turning key 12 was less subjective than it actually was at the time of writing (2011). Or, if the text was the same back in 1996, he is simply being obfuscatory and quoting himself out of context to make it appear as though leading the nation through war was always a clearly defined prerequisite for turning key 12, when in fact it was not.
It’s hard to be charitable here. If you re-read that second lengthy quotes from Lichtman, you’ll see that he is explicit about there being loads of ways to turn the hero key other than providing “vital leadership in war”. And let’s be clear, it’s already a stretch to equate the provision of “vital leadership in war” with “lead[ing] the nation through war”. The latter seems like a specific (and much more selective) example of the former. Unfortunately, as we’ll see in a moment, this kind of equivocation is not a one off.
Turning back to Silver’s critique, there is some slightly misguided discussion about The Keys’ inability to determine the margin of victory in any given election (in his retort, Lichtman quite reasonably responds that “the keys were not designed to predict vote percentages. They were designed to forecast whether the incumbent or challenging party will prevail in the popular vote, regardless of the margin of victory”). Then Nate hits upon something important, which I will quote in full:
“If there are, say, 25 keys that could defensibly be included in the model, and you can pick any set of 13 of them, that is a total of 5,200,300 possible combinations. It’s not hard to get a perfect score when you have that large a menu to pick from! […] It’s less that he has discovered the right set of keys than that he’s a locksmith and can keep minting new keys until he happens to open all 38 doors.” (Silver, 2011, online).
Why is this important? It is important because it describes the way in which Professor Lichtman mis-represents the input he used to create The Keys (results from historic US presidential elections) as the output of his system. Here’s an example: he writes “retrospectively, the keys account for the results of every presidential election from 1860 through 1980, much longer than any other prediction system” (Lichtman, 2016, pp. 9). This is not a lie, but it is deceptively worded. What does it even mean for a system to “retrospectively” account for the very data it was built upon? It would be a bloody poor system if it didn’t! To put it another way, we all know that The Keys is not going to call the 1932 election for Herbert Hoover, and we all intuitively know why: The Keys does not predict the result of this election (or even “account” for it) but is in fact predicated on it. As Silver notes above, there is nothing fantastically impressive in designing a model that can regurgitate the data it is modelled upon – hindsight, as they say, is 20-20.
Let’s move on from hindsight to what we’re really interested in here: foresight. Since The Keys’ creation, Professor Lichtman has called all ten US presidential elections (the first being 1984). What’s his actual predictive track record? Frankly, it’s great. In all that time, he has only called the race incorrectly once – the 2000 election, when George Bush Jr. beat then-Vice President Al Gore in the Electoral College despite losing the popular vote. Ah, but hang on – The Keys is designed to predict popular vote victory, not Electoral College victory. So, over nearly 40 years, The Keys has managed an incredible success rate of 100%! Let’s turn now to the penultimate chapter of The Keys, where Professor Lichtman forecasts the result of the 2016 presidential election… his prediction? Uncertain. Luckily, Professor Lichtman didn’t leave it there.
In a September 23, 2016 interview with The Washington Post (sorry about the paywall), Professor Lichtman responds to the question of who, with seven weeks left before voting day, will be president by the end of the year: “Based on the 13 keys, it would predict a Donald Trump victory. Remember, six keys and you’re out, and right now the Democrats are out – for sure – five keys.” (Stevenson, 2016, online).
After the election, when Professor Lichtman had been proved right, unlike the vast majority of pundits and pollsters, the media was rightly impressed (and here, and here, for example). Or was it right to be impressed? Just a paragraph ago, I wrote that The Keys predict the winner of the popular vote, not the electoral college. A few paragraphs before that, I quoted Professor Lichtman’s retort to Nate Silver from 2011 that makes this fact explicit. And don’t forget, in my 2016 edition of The Keys, it’s stated in the very first chapter (Lichtman, 2016, pp. 9). President Trump, of course, did not win the popular vote – Hillary Clinton won 65,853,514 votes to Trump’s 62,984,828. Now, if you’re like me, you like your academics honest, and probably hope to read that Professor Lichtman quickly corrected this error and owned up to his mistake. Sadly… well, you can probably guess how it really went.
When I was adding the finishing touches to this review (October 18, 2021), I googled “Lichtman calls 2016 for Trump” (without the quotation marks). One of the articles that came up was an American University (where Lichtman is Distinguished Professor of History) article titled Historian’s Prediction: Donald J. Trump to Win 2016 Election. This article was published on September 26, 2016. The very first paragraph on that page is not part of the article itself. It reads: “Editor’s Note: This story has been updated with a correction. It has been corrected to read that Prof. Lichtman’s 13 Keys system predicts the winner of the presidential race, not the outcome of the popular vote.” Confusingly, it later claims that The Keys also predicted “Al Gore’s popular vote victory in 2000.” (Basu, 2016, online).
The pertinent question here is which election Lichtman admits to having called wrong: 2000, when he got the Electoral College vote wrong, or 2016 when he got the popular vote wrong? The answer is: neither.
My copy of The Keys was published in 2016, and it claims The Keys predict the outcome of the popular vote. But by the end of the same year, Lichtman had decided it predicts the winner of the presidential race instead. When did he change his mind? Not being Professor X, I can’t tell you for certain. However, what I lack in telepathic powers, I make up for with access to the Wayback Machine. So I can tell you without a shadow of a doubt that on November 12, 2016, just four days after the relevant election results came in, that editor’s note quoted in the previous paragraph didn’t exist.
Again, it’s hard to be charitable here… I’m not saying that Lichtman waited until the results came in, saw he was wrong, and decided to retroactively change what he claimed to be predicting rather than admit to his mistake. I’m just saying it looks like he did.
Part five: a final judgement
I think I’ve made my problems with The Keys pretty clear: it’s an engaging, interesting, and informative read, but a lot of it feels dishonest to me. Maybe that’s too harsh – Professor Lichtman might just be overly enthusiastic. I would probably be less sceptical if I hadn’t had to pay for the book, and if Professor Lichtman didn’t publish new editions so regularly.
Criticisms aside, though, there are plenty of reasons I talked the book up in part one of this review that go beyond “it’s an engaging read”. For one thing, Lichtman does have a genuinely positive political vision, and as someone with an interest in political theory (my master’s degree was in philosophy), it’s hard not to be attracted to it. Here are the “lessons” Lichtman thinks politicians should take away from The Keys, as presented in its final chapter:
Fire the hucksters (media consultants, pollsters, advertising and media strategists, etc.).
Concentrate on substance (focus on governing rather than rhetoric).
Don’t play it safe (propose bold and innovative policy initiatives).
Don’t hide from ideology (which can be the driving force behind an administration).
Take the high road (don’t repudiate your opponent unnecessarily – build yourself up rather than tear them down).
Pick the best candidate for the number-two slot (forget courting fringe votes, pick the best leader in the party to be your running mate).
Get off the merry-go-round (tone down all that campaigning, slash the media appearances, and ditch the photo opportunities).
To me, this sounds like a much better type of politics.
(It’s important to remember, though, that there’s a big difference between the suggestion that ads are less persuasive than we’re told they are, and the suggestion that people don’t frequently operate under the sway of persuasion. The former is pretty well documented, and vitally important to remember in a world where the media stirs up fears about online advertising re-routing democracy. The latter is, frankly, ludicrous, as study after study after study demonstrates).
Another thing that really stands out to me as a positive about The Keys is that it doesn’t pretend to be more scientific than it is. Lichtman is clear that The Keys include some subjective judgements. Ironically, I think this actually makes it better able to engage with objective reality. People are confusing, messy, subjective things, and when you put millions of them together in a complex system defined by inscrutable non-linear dynamics, predictions become decidedly fuzzy. This fuzziness is built into The Keys not because it’s complex, but because it doesn’t try to hide people’s inherent complexity from themselves.
I suspect most of us think that it matters whether or not a presidential candidate is extremely charismatic, but how do we quantify something that’s fundamentally qualitative? Step one has to be factoring it into our methodology and taking it seriously. As anyone familiar with Bayes’ Theorem already knows, evaluating our priors and adjusting expectations to incorporate new information is a fundamental aspect of making accurate predictions. In The Keys, we’re given a framework within which to think about the impact of charisma on an electorate, and the quantification is built into the system automatically: it’s one of thirteen equally weighted factors.
My final assessment of The Keys is that it’s a fun and interesting read – although you should expect to be bored by the middle chapters, unless you’re a political history enthusiast. And my assessment of The Keys is that it’s a valuable addition to a political forecaster’s arsenal. If you’re already adept at forecasting, I wouldn’t suggest swapping Bayes for Lichtman – but The Keys could help you think about political issues in a wider, more useful, way (I would probably say it had this effect on me, despite already having a solid grounding in political theory).
Neither The Keys nor The Keys are, however, perfect.
The Keys suffers from a lack of objectivity when assessing The Keys’ ability to produce accurate forecasts, and in places this bleeds into (what I have to conclude is) outright deception. And The Keys, despite its incredible track record and intuitive nature, has nothing like the success rate Lichtman claims.
Lichtman has made ten predictions with this system and gotten nine of them right. The subjective nature of The Keys itself makes this success rate hard to evaluate… is Lichtman particularly good at using The Keys because he’s a quantitative historian who specialises in American political history? Would he make good predictions no matter what framework he used? A fairer way to assess The Keys’ performance would be for Lichtman to teach a team of undergrads his system and track their results over time. Given Professor Lichtman’s apparent dedication to preserving the illusion of omniscience, I doubt we’ll see this kind of rigour employed any time soon.
All in all, The Keys is deserving of a space on the bookshelf if you’re into political forecasting. But, until Professor Lichtman addresses some of the issues raised in this review, I don’t think it’s deserving of a great deal more than that.
Book Review: ‘Predicting the Next President: The Keys to the White House’
Part one: what is this book and should you read it?
Predicting the Next President: The Keys to the White House (henceforth rendered “The Keys”) is an ambitious book. Penned by historian Allan Lichtman, The Keys explains his system for forecasting the outcome of US presidential elections. This system is also called “The Keys”. So, to help readers figure out whether I’m talking about the book or the system, I’m only going to italicize The Keys when I’m explicitly referring to the book. The Keys is a 13-strong checklist of “true-or-false” statements that measure how well the incumbent party has performed during its term in office. If six or more statements are judged to be “false” then the election goes to the challenger. Five or fewer, and the incumbent takes it.
According to Lichtman, The Keys reflect a number of axiomatic truths about the democratic system in the US. And although he’s keen to caution against superposing these truths non-reflectively on democratic systems in other countries, he also believes The Keys speak to political processes in liberal democracies more generally.
In his introductory chapter, Lichtman describes a “pragmatic” American electorate that chooses a president “according to the performance of the party holding the White House” and theorises: “If candidates and the media could come to understand that governing, not campaigning, counts in presidential elections, we could have a new kind of presidential politics.” (Lichtman, 2016, pp. 8). This new kind of politics would dump the attack ads and go deep on policy promises, outlining substantial agendas of what candidates actually planned to do with their desired four years in office.
I should be clear from the outset that I think Lichtman’s vision is an appealing one. I also think The Keys is well written, and The Keys is well conceived. Assuming you’re in the target audience, you will probably enjoy reading this book, and might come away from it with something useful. What is the target audience? Probably people who enjoy reading long copy about predictive methodology and political history (given where you’re reading this, I don’t think it’s a stretch to assume that includes you, dear reader).
With all that out of the way, let me tell you what’s coming next. In part two, I’ll give a brief overview of what’s in this book and how I’m going to address it. In the third section, I’m going to break down each of the 13 keys and explain how you can use them to create your own electoral forecast. Then, in part four, I’ll look at some of the criticisms levelled against The Keys and give my own brief critique. Finally, in part five, I’ll offer my opinion on the both the book and the system.
If you don’t want to wade through all of that, here’s the “TLDR” version: I like The Keys, but I think it’s ultimately a flawed predictive methodology that doesn’t live up to its own hype. And I like The Keys but I take issue with some of the claims Lichtman makes and (what I think is) a sometimes deceptively rhetorical style of writing. However, these issues don’t quite manage to make the book unenjoyable or the system it describes unworkable. So, if it sounds like your cup of tea, I suggest you take the plunge and start reading.
Part two: mapping the structure of this review to the structure of The Keys
One of the best things about The Keys is how straightforward it is: it’s just 13 true or false statements. Five or fewer false answers, and the incumbent keeps the White House, six or more and they lose it. The Keys is also straightforward, giving you a full rundown of all 13 keys in chapter two, which I’ll summarise in the next section. But before I get to that, let me briefly sketch the rest of the book’s structure out, and explain how much attention I give to each bit.
Chapter one deals with most of the theory (we’ll discuss this theory in parts three, four, and five of this review).
Chapters three to 11 are Professor Lichtman’s reading of American political history in relation to The Keys, which I’ll only mention in passing. I understand that these chapters are going to be a draw for some readers, but I found them a slog. To me – who came here for the theory and the method – these sections read as filler. Let me be clear, this is not fair to Professor Lichtman, who writes well and knows his subject, hence the “to me” qualifier in the previous sentence. But, from an operational perspective, there is nothing you need to read in them – all they offer is historic example after historic example of The Keys aligning with presidential election outcomes (and, of course, some insight into Lichtman’s own view of American political history).
Chapter 12 is Professor Lichtman’s prediction for the 2016 US presidential election, which I’ll talk about in part four of this review (remember, this is the 2016 edition, so it was published before Mr. Trump took the White House).
Finally, chapter 13 outlines what Professor Lichtman thinks the “lessons” of the 13 Keys are for political candidates. Despite its relative brevity, this is one of the most important chapters in the book, and it will feature pretty heavily in part five of this review.
Part three: the 13 Keys to the White House
Without further ado, here are the 13 Keys. This is a condensed and only-slightly paraphrased list (I will never in good conscience use the word “listicle”) taken from chapter two of The Keys. Each key is titled as-per Professor Lichtman’s original, and contains a statement which you have to decide is true or false. I’ve addressed scoring The Keys above, but let me reiterate it here: once you’ve assessed the veracity of each statement, all you need to do is count up how many “false” results you’ve got. Six or more, and the challenger wins the White House. Five or fewer, and the incumbent does.
The Incumbent-Party Mandate: The incumbent party holds more seats in the House of Representatives after the mid-term elections than it did after the previous mid-term elections.
Nomination Contest: The sitting president doesn’t face any serious competition for his party’s nomination.
Incumbency: The sitting president is the incumbent party’s candidate.
Third Party: The race is between “the big two” with no serious third-party campaign.
Short-term Economy: There’s no recession during the election campaign.
Long-term Economy: Real annual per capita GDP growth during the current term equals or exceeds mean growth in the previous two terms (for the economically challenged, this basically means the US economy has been growing as fast, or faster, during the term in which the election takes place as the two terms that directly precede it).
Policy Change: The incumbent makes major changes in national policy (Lichtman defines what a “major” change would look like: “redirecting the course of government or […] innovating new policies and programs that have broad effects on the nation’s commerce, welfare, or outlook.”) (Lichtman, 2016, pp. 53).
Social Unrest: There is no “sustained social unrest” during the incumbent’s term in office (Lichtman, 2016, pp. 55).
Scandal: The incumbent administration hasn’t had to contend with a major scandal (what counts as “major”? “The wrongdoing has to bring discredit upon the president himself, calling into question his personal integrity, or at least his faithfulness in upholding the law.”) (Lichtman, 2016, pp. 57).
Foreign or Military Failures: The incumbent administration hasn’t suffered a major foreign policy or military failure (there’s that word “major” again. This time it means: “a major disaster that appears to undermine America’s national interests or seriously diminish its standing in the world.”) (Lichtman, 2016, pp. 60).
Foreign or Military Successes: The incumbent administration has made a major foreign policy or military success (major: “perceived as dramatically improving the nation’s interests or prestige.”) (Lichtman, 2016, pp. 62 – 63).
Incumbent Charisma: The incumbent-party candidate is either immensely charismatic or a national hero (“an extraordinarily persuasive or dynamic candidate, or one who has attained heroic status through achievements prior to his nomination as a presidential candidate.”) (Lichtman, 2016, pp. 65).
Challenger Charisma: The challenging-party candidate is neither immensely charismatic nor a national hero (see Key 12’s description in parenthesis, but remember that Key 13 is claiming the challenger is not charismatic or a hero, whereas Key 12 is claiming the incumbent is one or both of these things).
It’s worth going back and re-reading these carefully, because some of the phrasing is a little awkward. This is an artifact of Lichtman’s decision to render The Keys as a series of true or false statements that are sometimes opposed to one another, and you can’t really write around it.
In Chapter two of The Keys Lichtman gives us pages of explanation for each key. It’s worth reading, and it’s interesting reading… but it’s not essential reading. You can use his predictive methodology with nothing more than the list above, provided you are able to figure out whether each statement given is true or false.
Part four: critiquing The Keys
On August 31, 2011, renowned political forecaster and number cruncher Nate Silver penned a post on FiveThirtyEight with the headline: Despite Keys, Obama Is No Lock. Although titularly concerned with then-incumbent president Barack Obama’s chances in the 2012 presidential election, the post is more noteworthy today for its blistering critique of The Keys: “Superficially quite impressive. But ‘superficial’ is the crucial term here.” (Silver, 2011, online).
What is Silver talking about specifically? Professor Lichtman’s claim that The Keys “called the winner” of every election since 1860 (Silver again: “That’s 38 elections in a row!”). Nate runs through “several problems with this model” noting first “the nature of the keys […] several of which are quite subjective”. Silver admits, for example, that “candidate quality” is a thing, but notes that it can be a pretty amorphous one. Why, he asks, was Mr Obama scored as charismatic in 2008 but not in 2012? Shouldn’t John McCain have qualified as a war hero in 2008? Why doesn’t Afghanistan count as a foreign policy failure, given that it’s unpopular with Americans?
Professor Lichtman responded to Silver a week later, claiming “his critique of the keys system cannot withstand scrutiny” (Lichtman, 2011, online) According to Lichtman, his “published definition” of the challenger charisma/hero key “rules out counting McCain as a national hero since he did not lead the nation through war, however admirable.” He continues to claim “in [the] first edition of the Keys to the White House published many years before the 2008 election, I wrote that candidates attain heroic stature only ‘through vital leadership in war like Ulysses Grant and Dwight Eisenhower.’” (Lichtman, 2011, online).
This is probably true – I don’t have the first edition to hand. However, I do have the 2016 edition, and I’m going to quote the passages from both key 12 and key 13 that apply here. To be clear, these are the only passages in the description of these keys that deal with the subject (heroism) under discussion. I apologise for the length of these quotes – I’m including them because I think it’s really important to illustrate a wider criticism I want to make:
“There have been only two clearly heroic nominees, both of whom attained this stature through vital leadership in war: Ulysses Grant and Dwight Eisenhower. Many other candidates, including William McKinley, George McGovern, and George H. W. Bush have had impressive military records but have fallen far short of the heroic status attained by Grant and Eisenhower. Of all presidential candidates since 1860, only Theodore Roosevelt, the leader of the “Rough Riders” during the Spanish-American War, combined personal charisma with near-heroic accomplishment.” (Lichtman, 2016, pp. 65).
“Heroic stature is easier to identify. To meet the threshold, a candidate’s achievement must be deemed critical to the nation’s success in an important endeavour, and probably should be of relatively recent vintage. New Jersey senator Bill Bradley’s career as a highly acclaimed professional basketball player, while appealing to a large segment of the population, does not make him a national hero. Ohio senator John Glenn’s achievement as the first American astronaut to orbit the earth might have conferred that status early on, but probably would no longer have secured the key had he been nominated in 1984.” (Lichtman, 2016, pp. 67).
Now, dear reader, I am going to make a prediction of my own that relies on nothing but my faith in your reading comprehension skills: you’re wondering “where, in the above quotes, does Professor Lichtman explicitly state that vital leadership in war is required to turn the ‘hero’ key?” Well, let me answer you: nowhere. He explicitly states it nowhere.
Now, you might also be thinking it’s unfair of me to quote the 2016 version of The Keys, given that this criticism and response dates back to 2011. But that leads us on to the following question: why is Professor Lichtman quoting himself from 1996, when the most recent edition of his work (at the time) was the 2008 edition? Thanks to the magic of Google Books, I can take a pretty good guess: because the text in the 2008 version is exactly the same as the text in the 2016 version.
In other words, Silver’s criticism is correct – it is not clear whether McCain (who won the Silver Star and Purple Heart, among other wartime baubles) should have turned the ‘hero’ key or not. And Lichtman’s response (in my opinion) is disingenuous. He quoted an out-of-date version of his own work to create the impression that turning key 12 was less subjective than it actually was at the time of writing (2011). Or, if the text was the same back in 1996, he is simply being obfuscatory and quoting himself out of context to make it appear as though leading the nation through war was always a clearly defined prerequisite for turning key 12, when in fact it was not.
It’s hard to be charitable here. If you re-read that second lengthy quotes from Lichtman, you’ll see that he is explicit about there being loads of ways to turn the hero key other than providing “vital leadership in war”. And let’s be clear, it’s already a stretch to equate the provision of “vital leadership in war” with “lead[ing] the nation through war”. The latter seems like a specific (and much more selective) example of the former. Unfortunately, as we’ll see in a moment, this kind of equivocation is not a one off.
Turning back to Silver’s critique, there is some slightly misguided discussion about The Keys’ inability to determine the margin of victory in any given election (in his retort, Lichtman quite reasonably responds that “the keys were not designed to predict vote percentages. They were designed to forecast whether the incumbent or challenging party will prevail in the popular vote, regardless of the margin of victory”). Then Nate hits upon something important, which I will quote in full:
“If there are, say, 25 keys that could defensibly be included in the model, and you can pick any set of 13 of them, that is a total of 5,200,300 possible combinations. It’s not hard to get a perfect score when you have that large a menu to pick from! […] It’s less that he has discovered the right set of keys than that he’s a locksmith and can keep minting new keys until he happens to open all 38 doors.” (Silver, 2011, online).
Why is this important? It is important because it describes the way in which Professor Lichtman mis-represents the input he used to create The Keys (results from historic US presidential elections) as the output of his system. Here’s an example: he writes “retrospectively, the keys account for the results of every presidential election from 1860 through 1980, much longer than any other prediction system” (Lichtman, 2016, pp. 9). This is not a lie, but it is deceptively worded. What does it even mean for a system to “retrospectively” account for the very data it was built upon? It would be a bloody poor system if it didn’t! To put it another way, we all know that The Keys is not going to call the 1932 election for Herbert Hoover, and we all intuitively know why: The Keys does not predict the result of this election (or even “account” for it) but is in fact predicated on it. As Silver notes above, there is nothing fantastically impressive in designing a model that can regurgitate the data it is modelled upon – hindsight, as they say, is 20-20.
Let’s move on from hindsight to what we’re really interested in here: foresight. Since The Keys’ creation, Professor Lichtman has called all ten US presidential elections (the first being 1984). What’s his actual predictive track record? Frankly, it’s great. In all that time, he has only called the race incorrectly once – the 2000 election, when George Bush Jr. beat then-Vice President Al Gore in the Electoral College despite losing the popular vote. Ah, but hang on – The Keys is designed to predict popular vote victory, not Electoral College victory. So, over nearly 40 years, The Keys has managed an incredible success rate of 100%! Let’s turn now to the penultimate chapter of The Keys, where Professor Lichtman forecasts the result of the 2016 presidential election… his prediction? Uncertain. Luckily, Professor Lichtman didn’t leave it there.
In a September 23, 2016 interview with The Washington Post (sorry about the paywall), Professor Lichtman responds to the question of who, with seven weeks left before voting day, will be president by the end of the year: “Based on the 13 keys, it would predict a Donald Trump victory. Remember, six keys and you’re out, and right now the Democrats are out – for sure – five keys.” (Stevenson, 2016, online).
After the election, when Professor Lichtman had been proved right, unlike the vast majority of pundits and pollsters, the media was rightly impressed (and here, and here, for example). Or was it right to be impressed? Just a paragraph ago, I wrote that The Keys predict the winner of the popular vote, not the electoral college. A few paragraphs before that, I quoted Professor Lichtman’s retort to Nate Silver from 2011 that makes this fact explicit. And don’t forget, in my 2016 edition of The Keys, it’s stated in the very first chapter (Lichtman, 2016, pp. 9). President Trump, of course, did not win the popular vote – Hillary Clinton won 65,853,514 votes to Trump’s 62,984,828. Now, if you’re like me, you like your academics honest, and probably hope to read that Professor Lichtman quickly corrected this error and owned up to his mistake. Sadly… well, you can probably guess how it really went.
When I was adding the finishing touches to this review (October 18, 2021), I googled “Lichtman calls 2016 for Trump” (without the quotation marks). One of the articles that came up was an American University (where Lichtman is Distinguished Professor of History) article titled Historian’s Prediction: Donald J. Trump to Win 2016 Election. This article was published on September 26, 2016. The very first paragraph on that page is not part of the article itself. It reads: “Editor’s Note: This story has been updated with a correction. It has been corrected to read that Prof. Lichtman’s 13 Keys system predicts the winner of the presidential race, not the outcome of the popular vote.” Confusingly, it later claims that The Keys also predicted “Al Gore’s popular vote victory in 2000.” (Basu, 2016, online).
The pertinent question here is which election Lichtman admits to having called wrong: 2000, when he got the Electoral College vote wrong, or 2016 when he got the popular vote wrong? The answer is: neither.
My copy of The Keys was published in 2016, and it claims The Keys predict the outcome of the popular vote. But by the end of the same year, Lichtman had decided it predicts the winner of the presidential race instead. When did he change his mind? Not being Professor X, I can’t tell you for certain. However, what I lack in telepathic powers, I make up for with access to the Wayback Machine. So I can tell you without a shadow of a doubt that on November 12, 2016, just four days after the relevant election results came in, that editor’s note quoted in the previous paragraph didn’t exist.
Again, it’s hard to be charitable here… I’m not saying that Lichtman waited until the results came in, saw he was wrong, and decided to retroactively change what he claimed to be predicting rather than admit to his mistake. I’m just saying it looks like he did.
Part five: a final judgement
I think I’ve made my problems with The Keys pretty clear: it’s an engaging, interesting, and informative read, but a lot of it feels dishonest to me. Maybe that’s too harsh – Professor Lichtman might just be overly enthusiastic. I would probably be less sceptical if I hadn’t had to pay for the book, and if Professor Lichtman didn’t publish new editions so regularly.
Criticisms aside, though, there are plenty of reasons I talked the book up in part one of this review that go beyond “it’s an engaging read”. For one thing, Lichtman does have a genuinely positive political vision, and as someone with an interest in political theory (my master’s degree was in philosophy), it’s hard not to be attracted to it. Here are the “lessons” Lichtman thinks politicians should take away from The Keys, as presented in its final chapter:
Fire the hucksters (media consultants, pollsters, advertising and media strategists, etc.).
Concentrate on substance (focus on governing rather than rhetoric).
Don’t play it safe (propose bold and innovative policy initiatives).
Don’t hide from ideology (which can be the driving force behind an administration).
Take the high road (don’t repudiate your opponent unnecessarily – build yourself up rather than tear them down).
Pick the best candidate for the number-two slot (forget courting fringe votes, pick the best leader in the party to be your running mate).
Get off the merry-go-round (tone down all that campaigning, slash the media appearances, and ditch the photo opportunities).
To me, this sounds like a much better type of politics.
Particularly, I would like to say a few words about jettisoning the performative aspects of politics and focussing on the substantive, which is a major theme in Lichtman’s list of takeaways. There is a not-insignificant body of research that suggests advertising is far less effective than the people who create it would have you believe. I have worked in the advertising industry for over a decade, and this rings true to me. So, I think there’s something to Lichtman’s assertion that ads, rallies, and related tools are less impactful than presidential candidates hope.
(It’s important to remember, though, that there’s a big difference between the suggestion that ads are less persuasive than we’re told they are, and the suggestion that people don’t frequently operate under the sway of persuasion. The former is pretty well documented, and vitally important to remember in a world where the media stirs up fears about online advertising re-routing democracy. The latter is, frankly, ludicrous, as study after study after study demonstrates).
Another thing that really stands out to me as a positive about The Keys is that it doesn’t pretend to be more scientific than it is. Lichtman is clear that The Keys include some subjective judgements. Ironically, I think this actually makes it better able to engage with objective reality. People are confusing, messy, subjective things, and when you put millions of them together in a complex system defined by inscrutable non-linear dynamics, predictions become decidedly fuzzy. This fuzziness is built into The Keys not because it’s complex, but because it doesn’t try to hide people’s inherent complexity from themselves.
I suspect most of us think that it matters whether or not a presidential candidate is extremely charismatic, but how do we quantify something that’s fundamentally qualitative? Step one has to be factoring it into our methodology and taking it seriously. As anyone familiar with Bayes’ Theorem already knows, evaluating our priors and adjusting expectations to incorporate new information is a fundamental aspect of making accurate predictions. In The Keys, we’re given a framework within which to think about the impact of charisma on an electorate, and the quantification is built into the system automatically: it’s one of thirteen equally weighted factors.
My final assessment of The Keys is that it’s a fun and interesting read – although you should expect to be bored by the middle chapters, unless you’re a political history enthusiast. And my assessment of The Keys is that it’s a valuable addition to a political forecaster’s arsenal. If you’re already adept at forecasting, I wouldn’t suggest swapping Bayes for Lichtman – but The Keys could help you think about political issues in a wider, more useful, way (I would probably say it had this effect on me, despite already having a solid grounding in political theory).
Neither The Keys nor The Keys are, however, perfect.
The Keys suffers from a lack of objectivity when assessing The Keys’ ability to produce accurate forecasts, and in places this bleeds into (what I have to conclude is) outright deception. And The Keys, despite its incredible track record and intuitive nature, has nothing like the success rate Lichtman claims.
Lichtman has made ten predictions with this system and gotten nine of them right. The subjective nature of The Keys itself makes this success rate hard to evaluate… is Lichtman particularly good at using The Keys because he’s a quantitative historian who specialises in American political history? Would he make good predictions no matter what framework he used? A fairer way to assess The Keys’ performance would be for Lichtman to teach a team of undergrads his system and track their results over time. Given Professor Lichtman’s apparent dedication to preserving the illusion of omniscience, I doubt we’ll see this kind of rigour employed any time soon.
All in all, The Keys is deserving of a space on the bookshelf if you’re into political forecasting. But, until Professor Lichtman addresses some of the issues raised in this review, I don’t think it’s deserving of a great deal more than that.