I’d be interested in hearing more about this; I’m unfamiliar with the SciCast interface.
I notice you put user friendly in quotes—do you think it was not in fact user friendly, or was it misleading in an important dimension?
For comparison, I don’t think the Metaculus interface is simple enough to do the job. I agree it would be a fraught process, but my starting assumption is that developing an intuitive and correct UI is always a fraught process.
I realized after my initial comment that it was a bit too terse to be productive. Sorry.
(pre-note: Metaculus is not a market, and I don’t consider it interesting. I’m addressing and interested only in the case of an actual market where binary securities on events are being traded, with other market participants directly, or all through a market maker like SciCast.)
SciCast had two (major & relevant) modes for placing a trade, I remember them as “easy” and “normal”. Normal presented you with a slider from 0% to 100%, you could move the slider back and forth, and as you moved it, you were presented with a continuously updating quote for the “cost” of making the trade. The cost could be negative (equivalent to selling a previously held long position, or closing an existing short position), or positive (going long, or covering a short, whatever). After you made a trade, the interface immediately reflected the percentage that you’d chosen as the new value of the market. While you could go terribly wrong with this interface (and a lot of people did), it was at least possible to do reasonable things.
The Very Bad interface was the “easy” one. Instead of a slider, and instead of being given any feedback about the costs of your trade, you were presented simply with five choices, probably something like <20%, 20-40%, 40-60%, 60-80% and >80%. I don’t remember the exact equations that were used at this point, but it caused you to submit a trade that moved the market from wherever it was to as close as it get to the midpoint(?) of the region, without spending more than 1⁄2 (1/4? something) of your remaining points.
You weren’t shown how many points the trade was about to cost you. And neither interface showed you how many points you already had invested in the positions you were adjusting!
About the worst thing (I remember) that happened to naive users (through no fault of their own, IMO) was to come along to a question where the current state of the market was maybe 5%, and then do an easy mode trade for “<20%”. The system would spend their points to move the market from 5% to 10%.
The worst thing (which the bad interface encouraged), was users who thought that they should be trading until the market forecast matched their forecast. So if the easy mode trade didn’t move the market to where they wanted (or I’d moved the market back in the seconds following their trade), they’d do another easy mode trade. Over and over, with no idea of the position they were building up on a single question.
The short version of the problem, in real-world stock market terms, is: just because you think APPL is only going to go up in the long run doesn’t mean that you dump the entirety of your savings into it via market orders.
Think about kelly betting for a minute, just because the user has given you what they believe the odds to be isn’t enough to compute the correct bet for them. You also need to know their bankroll. And if you’re letting them trade on lots of questions simultaneously, you should be taking in a lot of probability estimates from the user, comparing them all to the state of the market, and making the trades that are profitable: ones where 1) the user believes the market is very wrong and 2) where corrections can be made inexpensively.
(For bonus points the user gives you the conditional probabilities between all those things, too, so that you can avoid betting too much on correlated trades.)
thats, uh, probably enough for now. i’ll keep an eye out for any questions.
Interesting! We designed “safe mode” to be safe against the common newbie failure of losing all cash in a few big trades. It achieved that much at least. Each trade spent max of about *1%* of your cash, and moved it at most halfway towards your goal. A dedicated newbie could still burn cash, but it took effort.
Still, from your report, that may have happened more than I thought. If so, it may not even have achieved its main design goal. :-(
“The worst thing” was indeed atrocious. We fixed that fairly quickly, but … ugh.
Thanks for the thoughtful critique. We’re about to reuse the engine. I’m giving this some thought.
i’m not sure if it would fix easy mode, but i feel that the points that go into a trade should be some function of the points that have already gone into that question, such that a user’s total loss on a given question is capped.
That is brutally bad. What was the training information like? Was it even possible for a naive user to become less naive without losing their ass a bunch of times?
I remember the tutorials as focusing on easy mode, and treating it as the thing to start with. With Charles refreshing my memory about how easy mode behaved (thank you), I think there was plenty of time for a slow and cautious user to come in and perhaps question the nature of easy mode before they’d done any damage to their account. Certainly I used it a bit at the start, and still recovered nicely.
Hoping that your users will be slow and/or cautious isn’t an ideal plan, though.
I’d be interested in hearing more about this; I’m unfamiliar with the SciCast interface.
I notice you put user friendly in quotes—do you think it was not in fact user friendly, or was it misleading in an important dimension?
For comparison, I don’t think the Metaculus interface is simple enough to do the job. I agree it would be a fraught process, but my starting assumption is that developing an intuitive and correct UI is always a fraught process.
I realized after my initial comment that it was a bit too terse to be productive. Sorry.
(pre-note: Metaculus is not a market, and I don’t consider it interesting. I’m addressing and interested only in the case of an actual market where binary securities on events are being traded, with other market participants directly, or all through a market maker like SciCast.)
SciCast had two (major & relevant) modes for placing a trade, I remember them as “easy” and “normal”. Normal presented you with a slider from 0% to 100%, you could move the slider back and forth, and as you moved it, you were presented with a continuously updating quote for the “cost” of making the trade. The cost could be negative (equivalent to selling a previously held long position, or closing an existing short position), or positive (going long, or covering a short, whatever). After you made a trade, the interface immediately reflected the percentage that you’d chosen as the new value of the market. While you could go terribly wrong with this interface (and a lot of people did), it was at least possible to do reasonable things.
The Very Bad interface was the “easy” one. Instead of a slider, and instead of being given any feedback about the costs of your trade, you were presented simply with five choices, probably something like <20%, 20-40%, 40-60%, 60-80% and >80%. I don’t remember the exact equations that were used at this point, but it caused you to submit a trade that moved the market from wherever it was to as close as it get to the midpoint(?) of the region, without spending more than 1⁄2 (1/4? something) of your remaining points.
You weren’t shown how many points the trade was about to cost you. And neither interface showed you how many points you already had invested in the positions you were adjusting!
About the worst thing (I remember) that happened to naive users (through no fault of their own, IMO) was to come along to a question where the current state of the market was maybe 5%, and then do an easy mode trade for “<20%”. The system would spend their points to move the market from 5% to 10%.
The worst thing (which the bad interface encouraged), was users who thought that they should be trading until the market forecast matched their forecast. So if the easy mode trade didn’t move the market to where they wanted (or I’d moved the market back in the seconds following their trade), they’d do another easy mode trade. Over and over, with no idea of the position they were building up on a single question.
The short version of the problem, in real-world stock market terms, is: just because you think APPL is only going to go up in the long run doesn’t mean that you dump the entirety of your savings into it via market orders.
Think about kelly betting for a minute, just because the user has given you what they believe the odds to be isn’t enough to compute the correct bet for them. You also need to know their bankroll. And if you’re letting them trade on lots of questions simultaneously, you should be taking in a lot of probability estimates from the user, comparing them all to the state of the market, and making the trades that are profitable: ones where 1) the user believes the market is very wrong and 2) where corrections can be made inexpensively.
(For bonus points the user gives you the conditional probabilities between all those things, too, so that you can avoid betting too much on correlated trades.)
thats, uh, probably enough for now. i’ll keep an eye out for any questions.
Your critique is plausible. I was never a fan of these supposedly simple interfaces.
Interesting! We designed “safe mode” to be safe against the common newbie failure of losing all cash in a few big trades. It achieved that much at least. Each trade spent max of about *1%* of your cash, and moved it at most halfway towards your goal. A dedicated newbie could still burn cash, but it took effort.
Still, from your report, that may have happened more than I thought. If so, it may not even have achieved its main design goal. :-(
“The worst thing” was indeed atrocious. We fixed that fairly quickly, but … ugh.
Thanks for the thoughtful critique. We’re about to reuse the engine. I’m giving this some thought.
i’m not sure if it would fix easy mode, but i feel that the points that go into a trade should be some function of the points that have already gone into that question, such that a user’s total loss on a given question is capped.
That is brutally bad. What was the training information like? Was it even possible for a naive user to become less naive without losing their ass a bunch of times?
I remember the tutorials as focusing on easy mode, and treating it as the thing to start with. With Charles refreshing my memory about how easy mode behaved (thank you), I think there was plenty of time for a slow and cautious user to come in and perhaps question the nature of easy mode before they’d done any damage to their account. Certainly I used it a bit at the start, and still recovered nicely.
Hoping that your users will be slow and/or cautious isn’t an ideal plan, though.