You get a hypothetical bayesian judge to take each person’s opinion as evidence.
Uh. How are you going to express options as probabilities?
This method works for forecasters because it’s typically done for mutually exclusive events. Either it rains, or it doesn’t, and your Bayesian judge can integrate different weathermen together into one superior weatherman (especially if the Bayesian judge also has access to the info the weathermen are using to make predictions with).
But when people are deciding what path the corporation should take, even if they assign probabilities of success to each possible action, only one action gets taken. And someone who is good at predicting when actions will fail may be bad at predicting when they will succeed (or vice versa).
I’m not sure they have to be probabilities. The probabilities can be internal to the calculation (what’s the probability of this person having an opinion like this if such and such is the best course of action).
It might be helpful to get people to assign probabilities, but you might end up confusing yourself.
I’m not sure I understand what you mean with the other issues, sounds like something that could be handled with calibration tests and looking at previous records.
What? Evidence is almost never a probability. Probability is an internal detail of cognitive engines. How does a bayesian spam filter work? The words it processes are not probabilities.
You observe events and assign them probabilities. The opinions of the group are events.
I… don’t think we’re using the same Bayesianism, or we’re having serious communication problems.
Suppose my spam filter sees the word “movie.” It thinks P(“movie”|legit)=.005, and P(“movie”|spam)=.001. That gives a likelihood ratio of 5 that the email is legitimate because it saw the word “movie”. That likelihood ratio isn’t a probability- but it’s a ratio of probabilities.
The problem in question, though, isn’t analogous to determining whether emails are spam or legit (logistic regression). It’s the problem of how to write emails. And so if one person wants to end the email with “Sincerely,” another with “Warmly,” and a third with “Yours,” how do you turn that into input for your Bayesian judge? I suppose you could elicit the probabilities that the person emailed will buy from you (how much do they buy?) given the word chosen, but it’s not obvious that would be helpful, especially when you’re doing this for everything at once rather than targeted A/B testing. (Also, what do you do when it’s clear a fourth word should have been considered?)
The word “movie” is not a probability itself, the same way the opinions people express are events and not probabilities.
With people’s opinions of what to do, there’s no reason you have to constrain what they say to things like “I, Bob, assign 50% probability that plan A is the best”. Even if you did that, you still have to consider that as evidence, it’s not like you can use another agent’s probability estimate directly in some way that you can’t use other types of statements, because it’s not your estimate. Bob might not even know probability theory.
If Bob says plan A is best, Linda and Alice say plan B is best, but Bob scored better on calibration assessments (including past decisions), and they have a poor record, you would integrate all the evidence with factors like P(Bob=A|Best=A) = 0.8, P(Alice=B|best=A) = .4, and so on to estimate P(Best=A|Bob=A, Alice=B,Linda=B).
Do you see what I mean? Bayes doesn’t become useless just because the environment is not composed of agents making explicit probability estimates.
But you never measure Best=A; you just measure how A performed, and you possibly compare that with your expectations of how A would perform. The system as just described runs into all of the problems that plurality voting has- irrelevant alternatives aren’t independent, etc.
Bayes doesn’t become useless just because the environment is not composed of agents making explicit probability estimates.
Bayes has a specific use: to maneuver through conditional probabilities. When you have moved outside the domain of a tool, it should be used with caution.
Do you see E = H anywhere? I don’t. E is the evidence, like, say, “bob thinks plan A is best”. H is the hidden variable that we are trying to infer. In this case H is “Plan A is best”.
The system as just described runs into all of the problems that plurality voting has- irrelevant alternatives aren’t independent, etc.
Like I said, I have not gone thru the math. I am not proposing a concrete formulation, I am trying to explain the concept that you don’t need to actually observe probability estimates from other agents to reason about the world.
Bayes has a specific use: to maneuver through conditional probabilities. When you have moved outside the domain of a tool, it should be used with caution.
Conditional probabilities like P(Bob thinks A is best | A is best).
Do you see E = H anywhere? I don’t. E is the evidence, like, say, “bob thinks plan A is best”. H is the hidden variable that we are trying to infer. In this case H is “Plan A is best”.
But to determine how much weight to give Bob as a judge for the second decision, you need to know whether or not Plan A was best for the first decision.
I am trying to explain the concept that you don’t need to actually observe probability estimates from other agents to reason about the world.
I agree that you don’t need to actually observe probability estimates from other agents to reason about the world. What I believe is that a Bayesian Judge is a tool that operates on probability estimates from other agents- and so if you want to reason this way, then you need this data.
But to determine how much weight to give Bob as a judge for the second decision, you need to know whether or not Plan A was best for the first decision.
You don’t need certainty. And you don’t necessarily need that particular evidence. It would still work using calibration tests to weight them.
The only evidence you really have access to from last time is who voted for what, and whether everyone thinks it was a good idea in hindsight. I think that would be enough.
What I believe is that a Bayesian Judge is a tool that operates on probability estimates from other agents- and so if you want to reason this way, then you need this data.
Ok. we are talking about different things. I’m talking about using bayesian methods to integrate evidence like votes, voting records, and hindsight estimations of optimality to determine the best distribution of probability over which plan is best (or some other output).
I have no idea how this “Bayesian Judge” thing that uses probability estimates directly would even work.
I have no idea how this “Bayesian Judge” thing that uses probability estimates directly would even work.
Here’s an article on Bayesian aggregation of forecasts. Essentially, you look at past forecasts to get P(Bob: “rain”|rain) and P(Bob: “rain”|~rain). (You can just elicit those expert likelihoods, but if you want this to be a formula rather than a person, you need them to be the data you’re looking for instead of just suggestive of the data you’re looking for.) From just that, you could calibrate Bob to find out what P(rain|Bob: “rain”) and P(rain|Bob: “~rain”) are. When you also have data on past predictions from Alice, Charlie, and David, you can combine them and get a more sophisticated estimate than any individual expert. It’s generally able to notice things like “when Alice and Bob agree, they’re both wrong,” which you couldn’t find by just computing individual calibrations.
That is, this thing you’ve been talking about is a procedure that’s already been worked out and that I’ve personally performed. It’s typically only done for forecasters of mutually exclusive possibilities and is inappropriate for decision-makers for reasons I’ve already mentioned.
The judge is a mathematical construction. I just mean that you use bayes rule in a well-defined way that everyone can agree on to take opinions and such as evidence.
Not sure how it would turn out, haven’t thought much. Should be able to do at least as well as prediction markets, tho.
That really sounds like a mysterious answer to me. Is there anything more to this than ‘Bayesian agent which inputs opinions and outputs awesome decisions’?
There is no internal detail yet, but this is not a mysterious answer.
What I said was basically “there exists a mathematical procedure for this problem that you would get if you went thru a derivation starting with bayes rule.” I don’t know how it works because I have not done the math. Optimality and uniqueness theorems suggest it should be at least as good as a prediction market (because at worst you just pull the bayes structure out of a prediction market), you could probably do much better, tho.
If that is a mysterious answer, the term means very little.
If you want detail, imagine naive bayes over people’s opinions of various proposals. Or something. That’s the 30 seconds of thought version, you could probably do a lot better in 5 minutes.
Well I came up with an interesting idea a few weeks ago for this very problem.
You get a hypothetical bayesian judge to take each person’s opinion as evidence.
I don’t know the details, but you might make people do some calibration tests to adjust for overconfidence, stupidity, etc.
Uh. How are you going to express options as probabilities?
This method works for forecasters because it’s typically done for mutually exclusive events. Either it rains, or it doesn’t, and your Bayesian judge can integrate different weathermen together into one superior weatherman (especially if the Bayesian judge also has access to the info the weathermen are using to make predictions with).
But when people are deciding what path the corporation should take, even if they assign probabilities of success to each possible action, only one action gets taken. And someone who is good at predicting when actions will fail may be bad at predicting when they will succeed (or vice versa).
I’m not sure they have to be probabilities. The probabilities can be internal to the calculation (what’s the probability of this person having an opinion like this if such and such is the best course of action).
It might be helpful to get people to assign probabilities, but you might end up confusing yourself.
I’m not sure I understand what you mean with the other issues, sounds like something that could be handled with calibration tests and looking at previous records.
If they’re not probabilities, then what will you use Bayes’ Rule for?
What? Evidence is almost never a probability. Probability is an internal detail of cognitive engines. How does a bayesian spam filter work? The words it processes are not probabilities.
You observe events and assign them probabilities. The opinions of the group are events.
I… don’t think we’re using the same Bayesianism, or we’re having serious communication problems.
Suppose my spam filter sees the word “movie.” It thinks P(“movie”|legit)=.005, and P(“movie”|spam)=.001. That gives a likelihood ratio of 5 that the email is legitimate because it saw the word “movie”. That likelihood ratio isn’t a probability- but it’s a ratio of probabilities.
The problem in question, though, isn’t analogous to determining whether emails are spam or legit (logistic regression). It’s the problem of how to write emails. And so if one person wants to end the email with “Sincerely,” another with “Warmly,” and a third with “Yours,” how do you turn that into input for your Bayesian judge? I suppose you could elicit the probabilities that the person emailed will buy from you (how much do they buy?) given the word chosen, but it’s not obvious that would be helpful, especially when you’re doing this for everything at once rather than targeted A/B testing. (Also, what do you do when it’s clear a fourth word should have been considered?)
The word “movie” is not a probability itself, the same way the opinions people express are events and not probabilities.
With people’s opinions of what to do, there’s no reason you have to constrain what they say to things like “I, Bob, assign 50% probability that plan A is the best”. Even if you did that, you still have to consider that as evidence, it’s not like you can use another agent’s probability estimate directly in some way that you can’t use other types of statements, because it’s not your estimate. Bob might not even know probability theory.
If Bob says plan A is best, Linda and Alice say plan B is best, but Bob scored better on calibration assessments (including past decisions), and they have a poor record, you would integrate all the evidence with factors like P(Bob=A|Best=A) = 0.8, P(Alice=B|best=A) = .4, and so on to estimate P(Best=A|Bob=A, Alice=B,Linda=B).
Do you see what I mean? Bayes doesn’t become useless just because the environment is not composed of agents making explicit probability estimates.
But you never measure Best=A; you just measure how A performed, and you possibly compare that with your expectations of how A would perform. The system as just described runs into all of the problems that plurality voting has- irrelevant alternatives aren’t independent, etc.
Bayes has a specific use: to maneuver through conditional probabilities. When you have moved outside the domain of a tool, it should be used with caution.
Of course you don’t; that’s the hypothesis.
P(H|E) = P(E|H)*P(H)/P(E)
Do you see E = H anywhere? I don’t. E is the evidence, like, say, “bob thinks plan A is best”. H is the hidden variable that we are trying to infer. In this case H is “Plan A is best”.
Like I said, I have not gone thru the math. I am not proposing a concrete formulation, I am trying to explain the concept that you don’t need to actually observe probability estimates from other agents to reason about the world.
Conditional probabilities like P(Bob thinks A is best | A is best).
I’m done.
But to determine how much weight to give Bob as a judge for the second decision, you need to know whether or not Plan A was best for the first decision.
I agree that you don’t need to actually observe probability estimates from other agents to reason about the world. What I believe is that a Bayesian Judge is a tool that operates on probability estimates from other agents- and so if you want to reason this way, then you need this data.
You don’t need certainty. And you don’t necessarily need that particular evidence. It would still work using calibration tests to weight them.
The only evidence you really have access to from last time is who voted for what, and whether everyone thinks it was a good idea in hindsight. I think that would be enough.
Ok. we are talking about different things. I’m talking about using bayesian methods to integrate evidence like votes, voting records, and hindsight estimations of optimality to determine the best distribution of probability over which plan is best (or some other output).
I have no idea how this “Bayesian Judge” thing that uses probability estimates directly would even work.
Here’s an article on Bayesian aggregation of forecasts. Essentially, you look at past forecasts to get P(Bob: “rain”|rain) and P(Bob: “rain”|~rain). (You can just elicit those expert likelihoods, but if you want this to be a formula rather than a person, you need them to be the data you’re looking for instead of just suggestive of the data you’re looking for.) From just that, you could calibrate Bob to find out what P(rain|Bob: “rain”) and P(rain|Bob: “~rain”) are. When you also have data on past predictions from Alice, Charlie, and David, you can combine them and get a more sophisticated estimate than any individual expert. It’s generally able to notice things like “when Alice and Bob agree, they’re both wrong,” which you couldn’t find by just computing individual calibrations.
That is, this thing you’ve been talking about is a procedure that’s already been worked out and that I’ve personally performed. It’s typically only done for forecasters of mutually exclusive possibilities and is inappropriate for decision-makers for reasons I’ve already mentioned.
neat!
Needs more specification. Is the judge a member of the group? Selected in advance (how?)?
The judge is a mathematical construction. I just mean that you use bayes rule in a well-defined way that everyone can agree on to take opinions and such as evidence.
Not sure how it would turn out, haven’t thought much. Should be able to do at least as well as prediction markets, tho.
That really sounds like a mysterious answer to me. Is there anything more to this than ‘Bayesian agent which inputs opinions and outputs awesome decisions’?
There is no internal detail yet, but this is not a mysterious answer.
What I said was basically “there exists a mathematical procedure for this problem that you would get if you went thru a derivation starting with bayes rule.” I don’t know how it works because I have not done the math. Optimality and uniqueness theorems suggest it should be at least as good as a prediction market (because at worst you just pull the bayes structure out of a prediction market), you could probably do much better, tho.
If that is a mysterious answer, the term means very little.
If you want detail, imagine naive bayes over people’s opinions of various proposals. Or something. That’s the 30 seconds of thought version, you could probably do a lot better in 5 minutes.
Didn’t have any of that background information, sorry.
You’re right, though, this isn’t a mysterious answer.
The judge is selected by another hypothetical Bayesian agent, who is selected by yet another hypothetical Bayesian agent, who is selected by....