PredictionBook: A Short Note
New Updates
PredictionBook has been updated and the speed improvements are massive. I was considering abandoning it because navigating the site was so slow (especially on my Android smartphone), but now I’m here to stay. Since there is no news feed on the website itself, it seemed appropriate to announce the update here. Also, the layout has changed slightly (for the better IMO) and the probability assignments have been made more intuitive (which many of the newcomers were having trouble with). The updates, along with the (slowly) growing user base, make now a better time then ever to join. Thanks TrikeApps and Ivan Kozik!
My Experience So Far
I was one of those that was convinced to give PredictionBook a shot based on gwern’s article. It has only been a month, so I am using up my willpower preventing myself from drawing any firm conclusions about my web of belief or sanity from such a small sample size. Ultimately, my goal is to know what 5% feels like and I already believe I have made a small step towards doing so. Even if PredictionBook turns out to be a failure for improving my calibration to the extent that I can intuitively feel what my degree of belief in a proposition is in percentage terms, it would still be worth it for the simple reason that I now have an inkling about how poorly calibrated I am. Finally, on at least three occasions, I have found myself trying to cook up some kind of rationalization in order to put off doing some task that I probably should be doing, but have felt additional pressure not to give in to akrasia because “my prediction will be judged false”.
Admonition/Guilt Trip
Honestly, PredictionBook is a rationalist’s dream tool. How would you even know if you are actually becoming less wrong without tracking your predictions? Eliezer Yudkowsky, Robin Hanson, Lukeprog, Yvain, Alicorn, Phil Goetz, wedrifid, Anna Salamon, Wei Dai, and cousin_it where the hell are you? Do you doubt the potential of PredictionBook to improve your calibration and diminish akrasia or is rationality really about “affiliating with an ideology or signaling one’s authority” and not about having the map reflect the territory (yes, I realize this is a false dichotomy, but it makes for better rhetoric)?
The Future of PredictionBook
I would like to see a few changes in the way PredictionBook works and a few new features. Sometimes when you are entering dates into the date field it shows an incorrect number of minutes/days/years (but seems to fix itself once the prediction is submitted). This seems like a minor nitpick that should be easily fixable. Another minor change would be to make it more intuitive how the judging buttons work, since some of the newer members try pushing them instead of assigning a probability to the statement in the prediction.
It would be nice to have a measure of the user’s calibration that can be compared across time, so that the user can easily determine if their calibration is getting better or worse. This could be something like a measure of how closely an OLS line of the user’s predictions matches the line-of-perfect-calibration in the graph on the user’s page. Also, I would like to see some of the community conventions written down in an FAQ-like document (specifically how conditional predictions should be judged), perhaps after they have been discussed.
I should add that the source code is now available on github, but it is written in Ruby (which I don’t have any experience with yet). If there are any Ruby programmers on the Less Wrong Public Goods Team, this might be something worth considering for a project.
Note: Now that I called some people out, my probability estimates concerning who will make predictions on PredictionBook will have to be adjusted.
EDIT: There already exists a measure of calibration somewhat like the one I hinted at above; it’s called a Brier score.
EDIT 2: lukeprog has joined the PredictionBook community.
gwern told me about PB in January 2011, and given my heavy use, I’ve spent about two days tweaking it and fixing bugs. If you like the changes, feel free to upvote this comment (I could use more than 0 karma). Note that TrikeApps is responsible for making the database faster, not me.
Also, if you see old predictions with bizarre estimates (e.g. 99% instead of 1%), it’s because almost everyone made mistakes due the old copy displaying “99% against” instead of “1%” for 1% - the new version doesn’t do this.
I edited the article to give you some credit.
Me, two months ago: “PB is mostly useless. Sure, things like elections can be judged, but I believe plenty of things that would be really hard for anyone to confirm. How would I ever turn them into predictions?”
Me, two weeks ago: *click* “That’s the whole point, isn’t it? Time to get active.”
Started a beeminder about making a number of predictions a week, and made a prediction about whether I’ll keep on top of the beeminder. Which counts as a prediction for this week, so I immediately added it to the beeminder.
I am … not sure I am using these tools in the intended manner.
I’ve decided to sign up. I don’t know why I dismissed this several times the first few times I saw it on here, it looks really interesting...
Edit: Why are you allowed to see what others predicted before you make a prediction yourself? That seems like it would wholly bias people...
This is a good point. Depending on what exactly you are using it for, you might or might not want to see others’ predictions before you make your own. It would be a useful feature to be able to toggle that functionality on/off in the user settings.
One should be able to usefully incorporate other people’s opinions, if only to adjust in the opposite direction!
As well, it’d be something of a UI/usability nightmare. If you really want to avoid contamination/additional information, make your own prediction and set it private or something.
I contemplated a strategy of always taking the average of the listed predictions. But that would be boring.
A less boring strategy: make two accounts, one assigning probabilities based on your intuition alone; the other based on the average of the listed predictions. Compare their calibrations.
You convinced me, I signed up and made some predictions.
Welcome to PredictionBook!
I don’t think in numbers, so I would expect to do really really badly for a really really long time, and I’m shy about things I realize I’m bad at. If I were going to embark on a long-term project to log lots of predictions for my own calibration and learning-what-percentages-feel-like, a text file on my own computer would do, wouldn’t it? I’d know if I tampered with it.
On occasions when I’m confident about a specific prediction and don’t feel shy about it, I prefer to just take some money from a nearby disagreer. I suppose if I wanted to make a longer-term bet, PredictionBook might be an okay place to stash the commitment if it’s set up for that? Is it? But I feel like I’d be likely to forget about a long term bet so I’d rather avoid making them.
I’m probably more numerically inclined than you are but one thing I’ve found to be really helpful is that PredictionBook has given me much more of a feel for what things like “60%” or “95%” mean in the real world. How confident does that really translate into to be accurate? The effort involved is minimal and the payoff can be subtle but high.
But how does PredictionBook help with that over a little text file where I would write things like “60%: That fleebs are spruckled by 2012; that thus-and-such experiment confirms that glox is a form of spolk; that abritsens are publicly repudiated by Mr. Blafwem before such time as he resigns” or whatever things I might actually be disposed to register predictions about?
Graphing is pretty nice to have, as it fixes a rather more-than-trivial inconvenience; I’m not sure even I care enough to figure out how to munge a raw text file into a proper calibration graph (generated using… what, GraphViz? I don’t even know where to start) to look at and go ‘oh, I’m really underconfident in the 10/90% range, why is that?’
It doesn’t help that aspect that directly at all.
Although I think having the ability to see what other people have predicted helps prevent lies to oneself especially of the form “well, that was just a black swan”. When someone else predicted the other direction or simply had much reduced confidence one isn’t able to say that.
Some important information about the world is currently communicated in numbers (for example, statistics on likely causes of death). PredictionBook may be a good way of learning to feel those numbers?
You can make your predictions private.
In which case the advantage of putting them on PredictionBook is what?
PredictionBook supports “When will you know?” and “Email me when I should know the outcome”, so you can make a long term prediction and set a reminder date.
Not forgetting about them, maybe? If I’m going to write something down for myself that I’m going to want to remember months or years later, I’d rather do it online somewhere where I can be sure I won’t simply lose track of it.
TrikeApps doesn’t get all of the credit here—we pushed in the speed improvements, but the layout and copy changes were contributed by Ivan Kozik. Kudos to you, Ivan.
I fixed the article to better share credit.
My first PredictionBook prediction.
Welcome Luke! I wagered against you joining, but I’m glad you’re here.
I lost my password for the site, and there doesn’t seem to be any way to get a replacement password emailed to me. Does anyone know how to arrange that?
Poke me with a PM convincing me that you own the account. A Forgot Password feature is one of the obvious features we didn’t get around to. Sorry about that.
On a related note it looks like you may be getting an influx of David Brin fans to PB. http://davidbrin.blogspot.com/2011/11/world-of-wonders-and-worries.html
Yeah, I told Trike about that when Brin linked it on Google+ (before the blog post); they moved PB to a bigger server to cope and apparently that’s when they decided to optimize the queries to be pure SQL or something.
There’s an issue I’ve been having with PredictionBook with the amount of “personal” predictions, like if a person will finish a task by a certain time. Could there be a separate category for this type of prediction?
In general, there’s not a good way of categorizing predictions right now. A few people have tried when they do predictions to make tags by hand by just commenting in the form “[personal]” or “[politics]” or something similar. But this just assists in searching for specific types of predictions. Being able to really categorize and to do boolean searches of categories would be nice addition.
PredictionBook now has a basic tagging functionality. Props to CFAR and Bellroy for supporting me in getting the feature added.
Upvoted, we very much need a way to tag predictions and a way to see our calibration on particular topics. I can’t believe I forgot to include this in my article.
By the way is anyone else getting an error message when they try to go directly from the email links to the updated predictions when someone has commented? I keep getting an error message on those.
Is this a good spot to make requests for additional functionality? If someone has the time, two things I’d really like to see is an ability to link efficiently in comments to different predictions using their numbers. The ideal syntax would be something like [3847] going to prediction number 3847.
This post and the comments inspired me to take a quick look at my history with PB.
joined: 2009-10-14, the day it was announced on LW
entered estimates on 5 “easy” predictions from others, one non-trivial, and created 3 private predictions
entered 2 private predictions and 2 estimates the next month, november 2009
briefly back in february 2010, 1 private prediction and one estimate
briefly back in september 2010, for 5 private predictions of professional interest
started using it again more regularly september 2011, with most of my activity occurring since then (25 transactions)
My participation in the Good Judgment project was the trigger for renewed interest in predictions and PB.com.
Sounds tongue in cheek, but hopefully it doesn’t actually make for better rhetoric here… right?
I was just being a little bit silly.