How much should the performance of the market change our opinion about
the viability of using prediction platforms to predict RCTs, and thus be
plausibly useful in selecting experiments to run and actions to perform?
We can define the maximum entropy distribution
(our prior on how good causal Futarchy markets should
be) over possible
log scores as having the mean of the log score of random forecasts,
namely −0.6931…
The exponential distribution is defined by one parameter, which is
λ=1μ (the mean of the distribution), in this case
λ=10.6931≈1.4427 (for convenience flipping
the distribution to be defined over positive reals). The logscore
observed for the Pomodoro method market was 0.3258, so the posterior
distribution is Exponential(λ+1/x): λn=1.4427+1/0.326≈4.5102.
To calculate the bits of evidence we got from
running the market, we calculate the information
gain, the bits of
evidence are calculated by log₂(posterior odds / prior odds).
For continuous distributions, we use probability densities, for the
exponential distribution:
I don’t really have a comparison point which to compare this result to,
but ≈0.2 bits of evidence seems fairly small to me. I guess I’ll have
to run some more experiments for further evidence (two currently in progress).
Acknowledgements
Many thanks to clippy
(twitter) for M500,
and Tetraspace
(twitter) for M1000, which I
used to subsidize the markets. Also many thanks to the manifold admin
Genzy for subsidizing each market
with M450.
Your funding of the sciences is greatly appreciated.
My gratitude also goes out to all the traders on the markets. You help
me prioritize, you help us gain knowledge.
0.202 Bits of Evidence In Favor of Futarchy
So, I put up some prediction markets on the results of quantified self RCTs. I ran one of the experiments, and scored one market on the results.
How much should the performance of the market change our opinion about the viability of using prediction platforms to predict RCTs, and thus be plausibly useful in selecting experiments to run and actions to perform?
We can define the maximum entropy distribution (our prior on how good causal Futarchy markets should be) over possible log scores as having the mean of the log score of random forecasts, namely −0.6931…
The maximum entropy distribution for a given mean on the positive reals is the exponential distribution.
The exponential distribution is defined by one parameter, which is λ=1μ (the mean of the distribution), in this case λ=10.6931≈1.4427 (for convenience flipping the distribution to be defined over positive reals). The logscore observed for the Pomodoro method market was 0.3258, so the posterior distribution is Exponential(λ+1/x): λn=1.4427+1/0.326≈4.5102.
To calculate the bits of evidence we got from running the market, we calculate the information gain, the bits of evidence are calculated by log₂(posterior odds / prior odds).
For continuous distributions, we use probability densities, for the exponential distribution:
log2(4.5102⋅exp(−4.5102⋅0.326))(1.4427⋅exp(−1.4427⋅0.326))≈log2(1.0367/0.9014)≈0.20176
I don’t really have a comparison point which to compare this result to, but ≈0.2 bits of evidence seems fairly small to me. I guess I’ll have to run some more experiments for further evidence (two currently in progress).
Acknowledgements
Many thanks to clippy (twitter) for M500, and Tetraspace (twitter) for M1000, which I used to subsidize the markets. Also many thanks to the manifold admin Genzy for subsidizing each market with M450.
Your funding of the sciences is greatly appreciated.
My gratitude also goes out to all the traders on the markets. You help me prioritize, you help us gain knowledge.
See Also
Replication Markets, Prediction markets for Science, and Social Science Prediction Platform
What I learned gathering thousands of nootropic ratings (troof, 2022)
[Part 1] Amplifying generalist research via forecasting – Models of impact and challenges (jacobjacob/ozziegooen/Elizabeth/NunoSempere/bgold, 2019)
[Part 2] Amplifying generalist research via forecasting – results from a preliminary exploration (jacobjacob/ozziegooen/Elizabeth/NunoSempere/bgold, 2019)