I did all my predictions last night immediately after the email showed up, so that meant I got to place a lot of bets at 50⁄50 odds :)
(Then I recorded everything privately in PredictionBook. No point in leaving my predictions trapped on their site.)
Interface-wise, I don’t like it at all. I’m still not sure what exactly I am betting at or with, compared to PB with straight probabilities or Intrade with share prices.
Did you take the “training refresher”? That includes a general-knowledge test at the end which scores you on both calibration and resolution. My results were pretty poor (but not abysmal):
You got 63% of the items correct, and your average confidence rating over all of the items was 74.33%. (...) In this exercise, your calibration is 11.00 (average confidence minus percent correct). (...) Your confidence when you were correct was 75.26%, and your confidence when you were incorrect was 72.73%. The difference is 2.53%.
I’d be curious to compare with yours if you’d care to share.
Without actually going through the whole refresher, it seems to be the same; when I did the training, I don’t remember that calibration/resolution test. Perhaps that is one of the experimental differences.
I didn’t remember that test from earlier, either. Worth checking out? I don’t mind accidentally unblinding a little if it is an experimental/control difference—curious folks will be curious.
I just went through the whole thing again; there was no test of that kind at the end. (What there was was the previous multiple-choice quiz about some example forecasts and how they went wrong.) Looks like this is an experimental/control difference. I’d rather not discuss that bit further—this isn’t about possibly life-or-death drugs, after all, and I already know where I can find calibration tests like that.
I’ve been wondering if it’s not the other way round, the Good Judgement project copying Inkling Market’s questions? What info do you have that leads you to think the copying was in the direction you assume?
My evidence for the other way round is that the Brent question has a starred footnote which is present on IM but not on GJ, while the star is in the text of the GJ question.
First, for those of you who logged in before September 6, please be aware that the tournament’s sponsor issued 10 new questions today, which we have posted. As the About page notes, new questions usually will be distributed on Mondays (but not every Monday). These questions arrived on Tuesday because of the Labor Day holiday.
My understanding was that the sponsor was IARPA. And googling, I don’t see any listed connections between Inkling and Good Judgement Project.
Stray asterisks are very suspicious. I see one in the Inkling question, but I don’t see the footnote itself. It has a “background information” section, but all their questions do. Is “last price” a technical term? If the usual term is “settlement price,” and GJ doesn’t make that clear, then it is quite suspicious.
Here aretwo two more Inkling questions with asterisks. One has an explicit footnote. The other is a change to the question.
Have you entered any comments on your predictions at the GJ site? (You’re supposed to enter a minimum number of comments over one year, and also a minimum number of responses to others’ comments. My understanding is that this will in time be run as a team game, with team play conventions.)
From my first experiences, I’m assuming the scoring will be pretty much as with PB.com—based on probability. Their model seems to be calibration/resolution rather than the visual “slope” representation.
Comments? I don’t see any relevant fields for that, checking right now, nor does my ‘About’ include the substring “comment”. Another experimental difference, I guess...
I did all my predictions last night immediately after the email showed up, so that meant I got to place a lot of bets at 50⁄50 odds :)
(Then I recorded everything privately in PredictionBook. No point in leaving my predictions trapped on their site.)
Interface-wise, I don’t like it at all. I’m still not sure what exactly I am betting at or with, compared to PB with straight probabilities or Intrade with share prices.
Did you take the “training refresher”? That includes a general-knowledge test at the end which scores you on both calibration and resolution. My results were pretty poor (but not abysmal):
I’d be curious to compare with yours if you’d care to share.
Without actually going through the whole refresher, it seems to be the same; when I did the training, I don’t remember that calibration/resolution test. Perhaps that is one of the experimental differences.
I didn’t remember that test from earlier, either. Worth checking out? I don’t mind accidentally unblinding a little if it is an experimental/control difference—curious folks will be curious.
I just went through the whole thing again; there was no test of that kind at the end. (What there was was the previous multiple-choice quiz about some example forecasts and how they went wrong.) Looks like this is an experimental/control difference. I’d rather not discuss that bit further—this isn’t about possibly life-or-death drugs, after all, and I already know where I can find calibration tests like that.
Fine with me. :)
BTW, look what I found. Did you know about this one?
Looks like someone is being very naughty. I’ve asked him on Twitter.
I’ve been wondering if it’s not the other way round, the Good Judgement project copying Inkling Market’s questions? What info do you have that leads you to think the copying was in the direction you assume?
My evidence for the other way round is that the Brent question has a starred footnote which is present on IM but not on GJ, while the star is in the text of the GJ question.
The description, from the latest email, is
My understanding was that the sponsor was IARPA. And googling, I don’t see any listed connections between Inkling and Good Judgement Project.
Stray asterisks are very suspicious. I see one in the Inkling question, but I don’t see the footnote itself. It has a “background information” section, but all their questions do. Is “last price” a technical term? If the usual term is “settlement price,” and GJ doesn’t make that clear, then it is quite suspicious.
Here are two two more Inkling questions with asterisks. One has an explicit footnote. The other is a change to the question.
Have you entered any comments on your predictions at the GJ site? (You’re supposed to enter a minimum number of comments over one year, and also a minimum number of responses to others’ comments. My understanding is that this will in time be run as a team game, with team play conventions.)
From my first experiences, I’m assuming the scoring will be pretty much as with PB.com—based on probability. Their model seems to be calibration/resolution rather than the visual “slope” representation.
Comments? I don’t see any relevant fields for that, checking right now, nor does my ‘About’ include the substring “comment”. Another experimental difference, I guess...
The “Why did you answer the way you did” field. I’ve been assuming we’re both using the same underlying app, i.e. Crowdcast. But perhaps we’re not...