Assessing Kurzweil: the gory details

Stuart_Armstrong15 Jan 2013 14:29 UTC
21 points
6 comments3 min readLW linkArchive

This post goes along with this one, which was merely summarising the results of the volunteer assessment. Here we present the further details of the methodology and results.

Kurzweil’s predictions were decomposed into 172 separate statements, taken from the book “The Age of Spiritual Machines” (published in 1999). Volunteers were requested on Less Wrong and on reddit.com/​r/​futurology. 18 people initially volunteered to do varying amounts of assessment of Kurzweil’s predictions; 9 ultimately did so.

Each volunteer was given a separate randomised list of the numbers 1 to 172, with instructions to go through the statements in the order given by the list and give their assessment of the correctness of the prediction (the exact instructions are at the end of this post). They were to assess the predictions on the following five point scale:

  • 1=True, 2=Weakly True, 3=Cannot decide, 4=Weakly False, 5=False

They assessed a varying amount of predictions, giving 531 assessments in total, for an average of 59 assessments per volunteer (the maximum attempted was all 172 predictions, the minimum was 10). They generally followed the randomised order correctly—there were three out of order assessments (assessing prediction 36 instead of 38, 162 instead of a 172, and missing out 75). Since the number of errors was very low, and seemed accidental, I decided that this would not affect the randomisation and kept those answers in.

The assessments (anonymised) can be found here.

In parallel, volunteers on Youtopia were also given the task of assessing the predictions. They were given the same instructions (minus the 5th and 7th clause), except that they were free to work on whichever predictions they wanted to, with the proviso that they didn’t overwrite someone else’s assessments. Instead, they could post a second opinion (not necessarily different from the first) in a separate column.

For some reason, prediction number 20 (“LUIs are frequently combined with animated personalities”) was left out of the Youtopia assessment. In total, 204 assessments were made (171 primary assessments, 33 second opinions).

The results of that exercise, with identifying information removed, can be found here.

Instructions

The instructions given to the assessors were as follows:

1) The timeline of Kurzweil’s prediction is up to 2011, and the location (unless specified otherwise) is the United States.

I’ve given Kurzweil a two-year grace period (he said they would all be true by 2009). This is because I think he forced a lot of predictions into the “true by ten years from now (1999)” format. Also, this makes it a bit easier for you, as you don’t need to go as far back into history.

2) A prediction is something that allows you to make a profit.

This is the true test of a prediction: if you’re in 1999, and you believe one of Kurzweil’s predictions, could you make your life better than someone who didn’t believe the prediction? If Kurzweil made a brilliant, correct prediction but nobody at the time would have realised what it meant, then it doesn’t count as a correct prediction. A prediction needs to make sense ahead of time, in a way you can take advantage of.

3) Resolving unclear terms is maybe the most important part of your job.

Some of Kurzweil’s predictions are ambiguous, including terms like “many”, “most”, “routinely”. Figure out what these terms mean for you. Predictions are acts of communication; they are only valid if the reader understands them correctly. The truth of “people will routinely use mobile phone” depends entirely on what meaning you give to “routinely”. In 1999, how you would have imagined a future in which people “routinely” use mobile phones?

4) No gain from ambiguity, no “benefit of the doubt”.

Do not interpret an ambiguous statement as true, simply to give the predictor the benefit of the doubt. With hindsight, some things will seem a lot more inevitable than they actually were, and some predictions will seem as if they “must refer to X”. For instance, “Two mighty towers will fall” seems like it refers to the Sep 11 terrorist attacks—but there are many ways of interpreting that figuratively or literally (two institutions will be undermined, Tolkein’s “the two towers” will be made into a film, etc...). Again the question is whether people in 1999 could have foreseen something like that outcome, based on that prediction.

5) Answer the prediction you’re working on, not the nearby ones.

The predictions just before and after are useful to give some context to the prediction you’re currently working on, to explain some terms and clarify what Kurzweil is talking about. But you should only answer the exact prediction you’re working on (don’t worry, those other prediction will have someone else working on them!). Thus the second prediction in “There will be a terrorist attack on the 9th of September, 2011. 4006 people will be killed in it” is false.

6) No penalty for triviality.

In hindsight, many prediction may seem trivial or obvious. This doesn’t mean they were trivial at the time. But in any case, your job is not to estimate how useful or hard the predictions were, but how accurate. “Computers will get faster” is a true prediction.

7) Follow the order in the text file included.

You are welcome and encouraged to answer more predictions than you promised to—but stick to the predictions given in the randomized text file! This will make the experiment statistically significant… Unless of course you intend to answer all the predictions!

Stuart_Armstrong15 Jan 2013 14:29 UTC
21 points
6 comments3 min readLW linkArchive