Thanks for writing such a thorough article! I’d be interested in seeing how LVPMs work in practice, but I must admit I’m coming from a position of extreme skepticism: Given how complicated real-world situations like the Russia/Ukraine war are, I’m skeptical a latent variable model can provide any marginal price efficiency over a simple set of conditional and unconditional markets.
My suspicion is that if a LVPM were created for a question like “Will China invade Taiwan by 2030?” that most of the predictive power would come from people betting directly on the latent variable rather than from any model-provided updates as a result of people betting on indicator variables. The number, type, and conditional dependency graph of indicator variables is too complicated to capture in a simple model and would function worse than human intuition, imho.
Other thoughts:
For LVPM to be useful, you’d probably want to add/remove indicator variables in real-time. What would be the process for doing this be? What is the payout of someone who bet before certain indicator variables were added? (This is sort of similar to the problems Manifold has faced in determining payouts for free-response markets.)
Price efficiency will suffer if you can’t sell your position before resolution or can’t provide liquidity in the form of a limit order or AMM liquidity provision. I think all of these problems are solvable in theory, but may require a good deal more mathematical cleverness.
Making a compelling UI for a LVPM is a hard problem.
I can’t make any guarantees, but the first step to getting a prototype of this up and running on Manifold (or elsewhere) would be creating a typescript npm package with the market logic. You might be able to convince me to work on this at our next hackathon...
Given how complicated real-world situations like the Russia/Ukraine war are, I’m skeptical a latent variable model can provide any marginal price efficiency over a simple set of conditional and unconditional markets.
When you say “a simple set of conditional and unconditional markets”, what do you have in mind?
Edit: I mean for a specific case such as the Russia/Ukraine war, what would be some conditional and unconditional markets that could be informative? (I understand how conditional and unconditional markets work in principle, I’m more interested in how it would practically compete with LVPMs.)
My suspicion is that if a LVPM were created for a question like “Will China invade Taiwan by 2030?” that most of the predictive power would come from people betting directly on the latent variable rather than from any model-provided updates as a result of people betting on indicator variables. The number, type, and conditional dependency graph of indicator variables is too complicated to capture in a simple model and would function worse than human intuition, imho.
So I am not 100% sure what you are referring to here. When you are talking about “model-provided updates as a result of people betting on indicator variables”, I can think of two things you might be referring to:
If someone bets directly on a market Xi, i.e. changing P(Xi) to a new value, then (using some math) we can treat this as evidence to update P(Y), using the fact that we know P(Xi|Y).
If someone bets on a conditional market P(Xi|Y), then the parameters of this conditional market may change, which in a sense constitutes an update as a result of people betting on indicator variables.
(Possibly something else?)
Your language makes me think you are referring to 1, and I agree that this will plausibly be of limited value. I’m most excited about 2 because it allows people to better aggregate information about the plausible outcomes, while taking correlations into account.
For LVPM to be useful, you’d probably want to add/remove indicator variables in real-time. What would be the process for doing this be? What is the payout of someone who bet before certain indicator variables were added? (This is sort of similar to the problems Manifold has faced in determining payouts for free-response markets.)
You can treat the addition or removal of an indicator as a trade performed by the market owner. The payout for a person for a trade t1 does not depend on any trades t2 that happen after the person’s t1. (This follows from the fact that we describe the payoffs as a function of the probabilities just before and just after t1.) So the owner of the market can add and remove however many indicator variables they want without any trouble.
Price efficiency will suffer if you can’t sell your position before resolution or can’t provide liquidity in the form of a limit order or AMM liquidity provision. I think all of these problems are solvable in theory, but may require a good deal more mathematical cleverness.
I think the logP(→X|θ)−logP(→X|ϕ) part of my scoring counts as an AMM liquidity provision? I might be misunderstanding what AMM liquidity provision means.
To scale up the liquidity, one can multiply this expression by the logP(→X|θ)−logP(→X|ϕ) expression by some constant so there is a deeper pool of liquidity.
Of course, a consequence of this is that the person creating the latent variable market has to provide enough liquidity to cover this, i.e. to create a market with a starting state ϕ, they would have to pay max→X,θlogP(→X|θ)−logP(→X|ϕ). In practice I would assume maxθlogP(→X|θ) is like 1, so it reduces to −logP(→X|ϕ), which for e.g. a ϕ predicting uniform distribution over P(→X) would be O(n) where n is the number of indicators.
Making a compelling UI for a LVPM is a hard problem.
At first glance this seems true; my UI seems to have confused people. However, I am not a designer.
I can’t make any guarantees, but the first step to getting a prototype of this up and running on Manifold (or elsewhere) would be creating a typescript npm package with the market logic. You might be able to convince me to work on this at our next hackathon...
Hm, this sounds like a task that would be relatively isolated from the rest of your codebase, and therefore something I could do independently without learning much of Manifold Market’s code? I would be up for programming this. Do you have any examples of similar modules in your codebase right now that I can look at, to see what interface it should provide?
When you say “a simple set of conditional and unconditional markets”, what do you have in mind?
Unconditional: “Will China invade Taiwan by 2030?”; Conditional: “If China experiences a recession any time before 2030, will China invade Taiwan by 2030?”
Your language makes me think you are referring to 1, and I agree that this will plausibly be of limited value.
Isn’t estimating P(Y) from P(Xi) most of the value of LVPMs?!
I think the logP(→X|θ)−logP(→X|ϕ) part of my scoring counts as an AMM liquidity provision?
Yes, that’s correct; if the market creator is willing to issue these payouts, then they are playing the role of AMM.
The question is how to add third-party liquidity provision to this system, i.e. where users can inject and remove liquidity from the market to increase payouts for traders (ideally while being compensated for their efforts).
Hm, this sounds like a task that would be relatively isolated from the rest of your codebase, and therefore something I could do independently without learning much of Manifold Market’s code?
That’s the idea. You can see some example code for our Uniswap-style AMM here, but honestly, any well-designed api would be fine. What I’d like is: 1. A typescript interface that defines the current state of a LVPM at any given point, 2. Betting function: Given a bet on some variable and the current market state, return the new market state and user position, 3. Resolution function: given the current market state, the final outcome of the market, and a list of user positions, return a list of all the user payouts.
Unconditional: “Will China invade Taiwan by 2030?”; Conditional: “If China experiences a recession any time before 2030, will China invade Taiwan by 2030?”
...
Isn’t estimating P(Y) from P(Xi) most of the value of LVPMs?!
Kind of. The point is to estimate P(Y). However, the value is not supposed to come from automatically estimating P(Y), but rather from providing an objective way of scoring manual predictions about P(Y) when we don’t have an objective definition of Y.
I agree that if the goal was to automatically predict P(Y) from P(Xi), structures of conditional prediction markets seems better. See also the “Latent variable markets vs combinatorial markets” section of my post; there are some cases where e.g.P(Y|Xi) is more interesting.
So basically, yes, the value of LVPMs comes from estimating P(Y), but the way we estimate P(Y) is by having people make predictions on P(Y). All the elaborate P(Xi|Y) has the primary purpose of providing a way to objectively score predictions on Y, and a secondary purpose of providing information/common knowledge about the domain of applicability of the Y concept.
Yes, that’s correct; if the market creator is willing to issue these payouts, then they are playing the role of AMM.
One thing I guess I should note is that the market maker only has to pay the first person. To clarify how this works, suppose the market maker creates a market with initial state θ0, and a first trader then updates this state to θ1, and a second trader then updates the state to θ2. Assuming for simplicity that the initial distribution P(→X|θ0) is uniform over →X, when the market resolves to a specific →x, we can then read the payoffs here:
The second trader gets a payout of logP(→x|θ2)−logP(→x|θ1).
The first traders gets a payout of logP(→x|θ1)−logP(→x|θ0).
The market maker paid −logP(→x|θ0) for making the market, which we can think of them as getting a payout of logP(→x|θ0) to make the signs of the expressions all aligned.
If we add all of these payouts up, we get (logP(→x|θ2)−logP(→x|θ1))+(logP(→x|θ1)−logP(→x|θ0))+(logP(→x|θ0))=logP(→x|θ2), which in general is a nonpositive number[1], and so the market will never give a greater payout than what is put in. Essentially what happens is that each trader pays off the next trader, guaranteeing that it all adds up. (This won’t require the individual traders to pay any huge amount of money, because they are always paying relative to logP(X|θ), which accounts for most of the pool.)
The question is how to add third-party liquidity provision to this system, i.e. where users can inject and remove liquidity from the market to increase payouts for traders (ideally while being compensated for their efforts).
I think if people add some additional liquidity, one can just find a new scale factor on the logP(→x|θ)−logP(→x|ϕ) payoffs that continues to make everything add up?
That’s the idea. You can see some example code for our Uniswap-style AMM here, but honestly, any well-designed api would be fine. What I’d like is: 1. A typescript interface that defines the current state of a LVPM at any given point, 2. Betting function: Given a bet on some variable and the current market state, return the new market state and user position, 3. Resolution function: given the current market state, the final outcome of the market, and a list of user positions, return a list of all the user payouts.
Thanks! I’ll take a look.
I’m thinking I should maybe implement LVPMs based on latent class analysis rather than item response theory because it is easier to design an interface for and get an intuition for (since LCA literally just enumerates the set of possible outcomes).
though presumably often logP(→x|θ2)≈0 rather than just logP(→x|θ2)<0, because as the close date nears, \vec x is going to be better and better known, and so the probaility is going to move towards 1
Thanks for writing such a thorough article! I’d be interested in seeing how LVPMs work in practice, but I must admit I’m coming from a position of extreme skepticism: Given how complicated real-world situations like the Russia/Ukraine war are, I’m skeptical a latent variable model can provide any marginal price efficiency over a simple set of conditional and unconditional markets.
My suspicion is that if a LVPM were created for a question like “Will China invade Taiwan by 2030?” that most of the predictive power would come from people betting directly on the latent variable rather than from any model-provided updates as a result of people betting on indicator variables. The number, type, and conditional dependency graph of indicator variables is too complicated to capture in a simple model and would function worse than human intuition, imho.
Other thoughts:
For LVPM to be useful, you’d probably want to add/remove indicator variables in real-time. What would be the process for doing this be? What is the payout of someone who bet before certain indicator variables were added? (This is sort of similar to the problems Manifold has faced in determining payouts for free-response markets.)
Price efficiency will suffer if you can’t sell your position before resolution or can’t provide liquidity in the form of a limit order or AMM liquidity provision. I think all of these problems are solvable in theory, but may require a good deal more mathematical cleverness.
Making a compelling UI for a LVPM is a hard problem.
I can’t make any guarantees, but the first step to getting a prototype of this up and running on Manifold (or elsewhere) would be creating a typescript npm package with the market logic. You might be able to convince me to work on this at our next hackathon...
Thanks for the reply!
When you say “a simple set of conditional and unconditional markets”, what do you have in mind?
Edit: I mean for a specific case such as the Russia/Ukraine war, what would be some conditional and unconditional markets that could be informative? (I understand how conditional and unconditional markets work in principle, I’m more interested in how it would practically compete with LVPMs.)
So I am not 100% sure what you are referring to here. When you are talking about “model-provided updates as a result of people betting on indicator variables”, I can think of two things you might be referring to:
If someone bets directly on a market Xi, i.e. changing P(Xi) to a new value, then (using some math) we can treat this as evidence to update P(Y), using the fact that we know P(Xi|Y).
If someone bets on a conditional market P(Xi|Y), then the parameters of this conditional market may change, which in a sense constitutes an update as a result of people betting on indicator variables.
(Possibly something else?)
Your language makes me think you are referring to 1, and I agree that this will plausibly be of limited value. I’m most excited about 2 because it allows people to better aggregate information about the plausible outcomes, while taking correlations into account.
You can treat the addition or removal of an indicator as a trade performed by the market owner. The payout for a person for a trade t1 does not depend on any trades t2 that happen after the person’s t1. (This follows from the fact that we describe the payoffs as a function of the probabilities just before and just after t1.) So the owner of the market can add and remove however many indicator variables they want without any trouble.
I think the logP(→X|θ)−logP(→X|ϕ) part of my scoring counts as an AMM liquidity provision? I might be misunderstanding what AMM liquidity provision means.
To scale up the liquidity, one can multiply this expression by the logP(→X|θ)−logP(→X|ϕ) expression by some constant so there is a deeper pool of liquidity.
Of course, a consequence of this is that the person creating the latent variable market has to provide enough liquidity to cover this, i.e. to create a market with a starting state ϕ, they would have to pay max→X,θlogP(→X|θ)−logP(→X|ϕ). In practice I would assume maxθlogP(→X|θ) is like 1, so it reduces to −logP(→X|ϕ), which for e.g. a ϕ predicting uniform distribution over P(→X) would be O(n) where n is the number of indicators.
At first glance this seems true; my UI seems to have confused people. However, I am not a designer.
I should probably repeat my request for a designer to work with.
Hm, this sounds like a task that would be relatively isolated from the rest of your codebase, and therefore something I could do independently without learning much of Manifold Market’s code? I would be up for programming this. Do you have any examples of similar modules in your codebase right now that I can look at, to see what interface it should provide?
Unconditional: “Will China invade Taiwan by 2030?”; Conditional: “If China experiences a recession any time before 2030, will China invade Taiwan by 2030?”
Isn’t estimating P(Y) from P(Xi) most of the value of LVPMs?!
Yes, that’s correct; if the market creator is willing to issue these payouts, then they are playing the role of AMM.
The question is how to add third-party liquidity provision to this system, i.e. where users can inject and remove liquidity from the market to increase payouts for traders (ideally while being compensated for their efforts).
That’s the idea. You can see some example code for our Uniswap-style AMM here, but honestly, any well-designed api would be fine. What I’d like is: 1. A typescript interface that defines the current state of a LVPM at any given point, 2. Betting function: Given a bet on some variable and the current market state, return the new market state and user position, 3. Resolution function: given the current market state, the final outcome of the market, and a list of user positions, return a list of all the user payouts.
Kind of. The point is to estimate P(Y). However, the value is not supposed to come from automatically estimating P(Y), but rather from providing an objective way of scoring manual predictions about P(Y) when we don’t have an objective definition of Y.
I agree that if the goal was to automatically predict P(Y) from P(Xi), structures of conditional prediction markets seems better. See also the “Latent variable markets vs combinatorial markets” section of my post; there are some cases where e.g.P(Y|Xi) is more interesting.
So basically, yes, the value of LVPMs comes from estimating P(Y), but the way we estimate P(Y) is by having people make predictions on P(Y). All the elaborate P(Xi|Y) has the primary purpose of providing a way to objectively score predictions on Y, and a secondary purpose of providing information/common knowledge about the domain of applicability of the Y concept.
One thing I guess I should note is that the market maker only has to pay the first person. To clarify how this works, suppose the market maker creates a market with initial state θ0, and a first trader then updates this state to θ1, and a second trader then updates the state to θ2. Assuming for simplicity that the initial distribution P(→X|θ0) is uniform over →X, when the market resolves to a specific →x, we can then read the payoffs here:
The second trader gets a payout of logP(→x|θ2)−logP(→x|θ1).
The first traders gets a payout of logP(→x|θ1)−logP(→x|θ0).
The market maker paid −logP(→x|θ0) for making the market, which we can think of them as getting a payout of logP(→x|θ0) to make the signs of the expressions all aligned.
If we add all of these payouts up, we get (logP(→x|θ2)−logP(→x|θ1))+(logP(→x|θ1)−logP(→x|θ0))+(logP(→x|θ0))=logP(→x|θ2), which in general is a nonpositive number[1], and so the market will never give a greater payout than what is put in. Essentially what happens is that each trader pays off the next trader, guaranteeing that it all adds up. (This won’t require the individual traders to pay any huge amount of money, because they are always paying relative to logP(X|θ), which accounts for most of the pool.)
I think if people add some additional liquidity, one can just find a new scale factor on the logP(→x|θ)−logP(→x|ϕ) payoffs that continues to make everything add up?
Thanks! I’ll take a look.
I’m thinking I should maybe implement LVPMs based on latent class analysis rather than item response theory because it is easier to design an interface for and get an intuition for (since LCA literally just enumerates the set of possible outcomes).
though presumably often logP(→x|θ2)≈0 rather than just logP(→x|θ2)<0, because as the close date nears, \vec x is going to be better and better known, and so the probaility is going to move towards 1
I unfortunately have to go to sleep now but I will respond to this tomorrow.