I think that prediction markets can help us select better proxies, but the initial set up (at least) will require people pretty clever with ontologies.
For example, say a group comes up with 20 proposals for specific ways of answering the question, “How much value has this organization created?”. A prediction market could predict the outcome of the effectiveness of each proposal.
I’d hope that over time people would put together lists of “best” techniques to formalize questions like this, so doing it for many new situations would be quite straightforward.
Another related idea we played around with, but which didn’t make it into the final whitepaper:
What if we just assumed that Brier score was also predictive of good judgement. Then, people, could create a distribution over several measures of “how good will this organization do” and we could use standard probability theory and aggregation tools to create an aggregated final measure.
The way we handled this with Verity was to pick a series of values, like “good judgement”, “integrity,” “consistency” etc. Then the community would select exemplars who they thought represented those values the best.
As people voted on which proposals they liked best, we would weight their votes by:
1. How much other people (weighted by their own score on that value) thought they had that value.
2. How similarly they voted to the examplars.
This sort of “value judgement” allows for fuzzy representation of high level judgement, and is a great supplement to more objective metrics like Brier score which can only measure well defined questions.
Eigentrust++ is a great algorithm that has the properties needed for this judgement-based reputation. The Verity Whitepaper goes more into depth as to how this would be used in practice.
I think that prediction markets can help us select better proxies, but the initial set up (at least) will require people pretty clever with ontologies.
For example, say a group comes up with 20 proposals for specific ways of answering the question, “How much value has this organization created?”. A prediction market could predict the outcome of the effectiveness of each proposal.
I’d hope that over time people would put together lists of “best” techniques to formalize questions like this, so doing it for many new situations would be quite straightforward.
Another related idea we played around with, but which didn’t make it into the final whitepaper:
What if we just assumed that Brier score was also predictive of good judgement. Then, people, could create a distribution over several measures of “how good will this organization do” and we could use standard probability theory and aggregation tools to create an aggregated final measure.
The way we handled this with Verity was to pick a series of values, like “good judgement”, “integrity,” “consistency” etc. Then the community would select exemplars who they thought represented those values the best.
As people voted on which proposals they liked best, we would weight their votes by:
1. How much other people (weighted by their own score on that value) thought they had that value.
2. How similarly they voted to the examplars.
This sort of “value judgement” allows for fuzzy representation of high level judgement, and is a great supplement to more objective metrics like Brier score which can only measure well defined questions.
Eigentrust++ is a great algorithm that has the properties needed for this judgement-based reputation. The Verity Whitepaper goes more into depth as to how this would be used in practice.
Deference networks seem underrated.