In general, I wouldn’t make it too complicated and accept some arbitrariness. There is a predetermined panel of e.g. 5 experts and e.g. 3 categories (feasibility, effectiveness, everything else). All submissions first get scored by 2 experts with a shallow judgment (e.g., 5-10 minutes). Maybe there is some “saving” mechanism if an overeager expert wants to read plans that weren’t assigned to them. Everything in the top N% then gets scored by all experts with a more detailed review. Then, there is a final ranking.
I’d hope that the time spent per expert is only 5-10 hours in total. I’d be fine with missing a bunch of posts that contain good ideas that are badly communicated or otherwise easy to miss on the shallow review.
My main goal with the contest would be that writing a good plan and communicating it clearly is incentivized.
At BlueDot we’ve been thinking about this a fair bit recently, and might be able to help here too. We have also thought a bit about criteria for good plans and the hurdles a plan needs to overcome, as well as have reviewed a lot of the existing literature on plans.
How would you operationalize a contest for short-timeline plans?
Something like the OpenPhil AI worldview contest: https://www.openphilanthropy.org/research/announcing-the-winners-of-the-2023-open-philanthropy-ai-worldviews-contest/
Or the ARC ELK prize: https://www.alignment.org/blog/prizes-for-elk-proposals/
In general, I wouldn’t make it too complicated and accept some arbitrariness. There is a predetermined panel of e.g. 5 experts and e.g. 3 categories (feasibility, effectiveness, everything else). All submissions first get scored by 2 experts with a shallow judgment (e.g., 5-10 minutes). Maybe there is some “saving” mechanism if an overeager expert wants to read plans that weren’t assigned to them. Everything in the top N% then gets scored by all experts with a more detailed review. Then, there is a final ranking.
I’d hope that the time spent per expert is only 5-10 hours in total. I’d be fine with missing a bunch of posts that contain good ideas that are badly communicated or otherwise easy to miss on the shallow review.
My main goal with the contest would be that writing a good plan and communicating it clearly is incentivized.
I’m tempted to set this up with Manifund money. Could be a weekend project.
Go for it. I have some names in mind for potential experts. DM if you’re interested.
At BlueDot we’ve been thinking about this a fair bit recently, and might be able to help here too. We have also thought a bit about criteria for good plans and the hurdles a plan needs to overcome, as well as have reviewed a lot of the existing literature on plans.
I’ve messaged you on Slack.