GiveWell already uses expert advice for expedient impact assessments. Albeit on a small scale, without using academic- know how and with suboptimal choice and choice architecture of their experts. Hope you can improve on it :)
You’ve picked the wrong problem domain for the scoring rules. Briar comes from probability assessment, there are already more sophisticated approaches to this problem several levels removed from the mathematical theory and synthesising several theoreums.
The most proximate implementations of what you are suggesting are either delphi groups (risk analysis) or prediction markets (rationalist subculture mainly, but also academic). You probably already know how prediction markets work and you can look up ‘expert elicitation’ or ‘eliciting expert judgement’ and similar terms if you’re interested. Happy to answer any tougher questions you can’t get answered.
There are structured approaches to delphi groups which incorporate bayes rules and insights around the psychology of eliciting and structuring expert judgement that you could mimic. There is at least one major corporate consultancy focused on this already, however. AFAIK there are no implementations of this kind in the blockchain. Whether that is a worthwhile competitive advantage is another question.
You have a strategic mindset, I like it. If I’ve interpreted your question accurately, the reason other’s in the know may not have responded is the xy problem.
There are structured approaches to delphi groups which incorporate bayes rules and insights around the psychology of eliciting and structuring expert judgement that you could mimic.
Yes, the technology I’m using (prediction polls) are essentially this. It’s Delphi groups weighted by Brier scores. The paper I link to above compares them to a prediction market with the same questions—with proper extremizing algorithms, the prediction poll actually does better (especially early on).
The reason I came up with this solution is that I wanted to use prediction markets for a specific class of impact assesments, but they weren’t suited for the task. Prediction markets require either a group of interested suckers to take the bad bets, or a market maker who is sufficiently interested in the outcome to be willing to take the bad side on ALL the sucker bets. My solution complements prediction markets by being much better in those cases by avoiding the zero sum game, and instead just directly paying experts for their expertise.
GiveWell already uses expert advice for expedient impact assessments. Albeit on a small scale, without using academic- know how and with suboptimal choice and choice architecture of their experts. Hope you can improve on it :)
You’ve picked the wrong problem domain for the scoring rules. Briar comes from probability assessment, there are already more sophisticated approaches to this problem several levels removed from the mathematical theory and synthesising several theoreums.
The most proximate implementations of what you are suggesting are either delphi groups (risk analysis) or prediction markets (rationalist subculture mainly, but also academic). You probably already know how prediction markets work and you can look up ‘expert elicitation’ or ‘eliciting expert judgement’ and similar terms if you’re interested. Happy to answer any tougher questions you can’t get answered.
There are structured approaches to delphi groups which incorporate bayes rules and insights around the psychology of eliciting and structuring expert judgement that you could mimic. There is at least one major corporate consultancy focused on this already, however. AFAIK there are no implementations of this kind in the blockchain. Whether that is a worthwhile competitive advantage is another question.
You have a strategic mindset, I like it. If I’ve interpreted your question accurately, the reason other’s in the know may not have responded is the xy problem.
Yes, the technology I’m using (prediction polls) are essentially this. It’s Delphi groups weighted by Brier scores. The paper I link to above compares them to a prediction market with the same questions—with proper extremizing algorithms, the prediction poll actually does better (especially early on).
The reason I came up with this solution is that I wanted to use prediction markets for a specific class of impact assesments, but they weren’t suited for the task. Prediction markets require either a group of interested suckers to take the bad bets, or a market maker who is sufficiently interested in the outcome to be willing to take the bad side on ALL the sucker bets. My solution complements prediction markets by being much better in those cases by avoiding the zero sum game, and instead just directly paying experts for their expertise.