The following is Metaculus’s year-in-review CEO letter written by Gaia Dempsey.
2022 was a year of progress, growth, and change at Metaculus.
We matured and grew as an organization.
Mission Update: In the spring, we expanded the scope of our mission, creating a solid foundation for our work. Specifically, our mission is to build epistemic infrastructure that enables the global community to model, understand, predict, and navigate the world’s most important and complex challenges. We also clearly defined the three primary ways we enact our mission, namely: providing forecasts as a public service, fostering a global forecasting community, and supporting forecasting research.
Pro Forecaster Program: In response to a sharp increase in global uncertainty with the outbreak of the war in Ukraine, we designed new operational procedures including an infohazard review process, and recruited over 30 candidates into an exceptionally talented Pro Forecasting team.
Becoming a Public Benefit Corporation: Shortly after, we committed to providing three specific public benefits in our charter, reflecting our mission statement, as part of the process of becoming a Public Benefit Corporation.
Growing Our Team: With a mandate to expand our platform, programs, and operations, we set about upgrading our hiring processes thanks to great resource recommendations from Jim Savage (a), and Aaron Hamlin (b), and got laser focused on recruiting. Over the course of the year we doubled our overall headcount, building four strong new teams: leadership, AI forecasting, engineering, and design (some engineering start dates are in early 2023).
We launched over a dozen forecasting tournaments and generated over 1000 aggregate forecasts.
Expansion of Forecasting Programs: Over the last year, we expanded our existing partnerships and began collaborations with a number of fantastic new partners. In total, we launched 13 new forecasting tournaments offering a total of $77,500 in prize money, with a number of programs delivering policy-relevant predictions on public health, biosecurity, and nuclear-risk. Outside of these core themes (each of which I’ll touch on later), Forecasting Our World In Data was a standout launch: a project that “probes the long-term future, delivering predictions on topics like global investment in AI, world life expectancy, CO2 emissions, and more on time horizons from one to 100 years.” Another favorite was our final launch of the year, the Sagan Tournament, focused on all things space-related — from technology, to scientific discovery, to governance.
Ukraine War Response: After accurately predicting the war in Ukraine, the Metaculus community continued to closely monitor the conflict, responding rapidly to new developments in a forecasting tournament launched just 48 hours after the invasion. With nuclear security expert Peter Scoblic, we deployed the Red Lines in Ukraine project as an early-warning system gauging the likelihood Russia would make use of nuclear weapons.
Biosecurity & Public Health: 2022 saw us increasing our impact in public health and biosecurity, executing the $25,000 Biosecurity Tournament with the Institute for Progress and Guarding Against Pandemics, as well as the Real-Time Pandemic Decision Making tournament with UVA’s Biocomplexity Institute to aid COVID computational modeling efforts. In May when clusters of Mpox virus infections were observed across multiple countries, our community responded immediately through our Mpox Series, contributing ensemble predictions and providing a crucial health tool for assessing widespread community transmission when data were sparse.
We grew our forecasting community and connected with a wider network.
News Embedding & Media Highlights: In an especially turbulent year, we collaborated with physicist Max Tegmark and his team at Improve the News to bring needed context to fast-moving news stories using timely forecasts by embedding forecasts in over 800 articles. We also had cameo appearances in a number of mainstream outlets, including The Atlantic, the Washington Post, BBC News, the Financial Times, Fast Company, the leading Spanish paper El Pais, and major Dutch outlet De Telegraaf, and were featured in the new, comprehensive CEA article that provides an introduction to Effective Altruism, as well as theoretical physicist Sean Carroll’s podcast Mindscape.
A Million Predictions, and a Hackathon: Metaculus achieved a fantastic milestone, surpassing 1,000,000 predictions over more than 7,000 questions. To celebrate, we hosted our first-ever data science hackathon, giving talented teams and individuals across the globe access to a rich trove of forecasting data. Unsurprisingly, the forecasting community displayed admirable creativity and rigor in their projects, and selecting the winners was no easy task. We also hosted a Week of Talks on forecasting, spanning a diverse range of topics from theory to practice.
We upgraded our infrastructure and shipped new features.
Upgraded Tech Stack: In 2022, we rewrote nearly the entire Metaculus application, modernizing the Metaculus tech stack to support our 2023 product roadmap. We have a number of enhancements and new features in store, big and small, and we can’t wait to share them with the community.
Question Groups & Fan Graphs: We released question groups and fan graphs, enabling the grouping of related questions and the visualization of their forecasts in series.
Private Forecasting Spaces & Language Localization: To support partner projects, including internationally, we developed private forecasting spaces that enable confidential forecasting for a group or organization of any size, as well as the ability to translate the Metaculus interface into any language.
Tournament Scoring & Leaderboards: We updated our tournament scoring and leaderboard systems to bring greater rigor and clarity to how we reward and incentivize forecasting skill.
We collaborated on research and published reports on the biggest potential risks of 2022.
Nuclear Escalation in Ukraine: After the invasion of Ukraine, policymakers and the public became increasingly concerned about the prospect of nuclear escalation. We recruited a team of Metaculus Pro Forecasters to make their judgments on key questions and provide their rationales, all of which were drawn up in a full nuclear risk report.
Predicting the Omicron BA.1 Wave: In partnership with the University of Virginia’s Biocomplexity Institute and the Virginia Department of Health, we co-authored a paper demonstrating the accuracy and robustness of using Metaculus’s COVID-19 Omicron variant forecasting ensembles in combination with computational models by providing valuable real-time forecasts.
Mpox Rapid Information Aggregation: When an unexpected number of Mpox cases were reported in early May 2022, we organized a rapid forecasting response to gauge the potential scope and impact of the outbreak. Working with our research partner Tom McAndrew at Lehigh University, we co-authored a paper on the efficacy of rapid human judgment forecasting, which was published (in record time) in The Lancet Digital Health.
Forecasting the US-China AI & Nuclear Landscape: With our partners at the Institute for Security and Technology, we launched an initiative evaluating intervention points in the US-China nuclear relationship, with a special focus on the integration of AI into nuclear command, control, and communications systems. The resulting report combines insights from nuclear and policy subject matter experts and Metaculus Pro Forecasters.
I’m extremely proud of what we’ve accomplished as a team, and I’m deeply grateful for the support of our partner organizations and the forecasting community. If you’re excited by what we’re doing and would like to get in touch, please do feel free to grab some time with me, or shoot our team a note.
Metaculus Year in Review: 2022
The following is Metaculus’s year-in-review CEO letter written by Gaia Dempsey.
2022 was a year of progress, growth, and change at Metaculus.
We matured and grew as an organization.
Mission Update: In the spring, we expanded the scope of our mission, creating a solid foundation for our work. Specifically, our mission is to build epistemic infrastructure that enables the global community to model, understand, predict, and navigate the world’s most important and complex challenges. We also clearly defined the three primary ways we enact our mission, namely: providing forecasts as a public service, fostering a global forecasting community, and supporting forecasting research.
Pro Forecaster Program: In response to a sharp increase in global uncertainty with the outbreak of the war in Ukraine, we designed new operational procedures including an infohazard review process, and recruited over 30 candidates into an exceptionally talented Pro Forecasting team.
Core Funding: In the summer, we secured a $5.5M core funding grant from Open Philanthropy to expand our work.
Becoming a Public Benefit Corporation: Shortly after, we committed to providing three specific public benefits in our charter, reflecting our mission statement, as part of the process of becoming a Public Benefit Corporation.
Growing Our Team: With a mandate to expand our platform, programs, and operations, we set about upgrading our hiring processes thanks to great resource recommendations from Jim Savage (a), and Aaron Hamlin (b), and got laser focused on recruiting. Over the course of the year we doubled our overall headcount, building four strong new teams: leadership, AI forecasting, engineering, and design (some engineering start dates are in early 2023).
We launched over a dozen forecasting tournaments and generated over 1000 aggregate forecasts.
Expansion of Forecasting Programs: Over the last year, we expanded our existing partnerships and began collaborations with a number of fantastic new partners. In total, we launched 13 new forecasting tournaments offering a total of $77,500 in prize money, with a number of programs delivering policy-relevant predictions on public health, biosecurity, and nuclear-risk. Outside of these core themes (each of which I’ll touch on later), Forecasting Our World In Data was a standout launch: a project that “probes the long-term future, delivering predictions on topics like global investment in AI, world life expectancy, CO2 emissions, and more on time horizons from one to 100 years.” Another favorite was our final launch of the year, the Sagan Tournament, focused on all things space-related — from technology, to scientific discovery, to governance.
Ukraine War Response: After accurately predicting the war in Ukraine, the Metaculus community continued to closely monitor the conflict, responding rapidly to new developments in a forecasting tournament launched just 48 hours after the invasion. With nuclear security expert Peter Scoblic, we deployed the Red Lines in Ukraine project as an early-warning system gauging the likelihood Russia would make use of nuclear weapons.
Biosecurity & Public Health: 2022 saw us increasing our impact in public health and biosecurity, executing the $25,000 Biosecurity Tournament with the Institute for Progress and Guarding Against Pandemics, as well as the Real-Time Pandemic Decision Making tournament with UVA’s Biocomplexity Institute to aid COVID computational modeling efforts. In May when clusters of Mpox virus infections were observed across multiple countries, our community responded immediately through our Mpox Series, contributing ensemble predictions and providing a crucial health tool for assessing widespread community transmission when data were sparse.
We grew our forecasting community and connected with a wider network.
80% YOY Forecaster Growth: We grew our monthly forecaster base by 80% relative to 2021, and launched over 1000 new forecasting questions.
Forecasting-Infused Reading & Listening: We published 40 excellent essays in the Metaculus Journal, each fortified by falsifiable predictions, on topics of immediate import, such as the Endgame of the War in Ukraine and Forecasting for Situational Awareness in an Emerging Public Health Crisis, and the longer-term, such as Computability and Complexity, and Life on the Icy Moons and Ocean Worlds of the Outer Solar System. Many of our Journal articles are also available in audio format, on our podcast, also launched in 2022.
News Embedding & Media Highlights: In an especially turbulent year, we collaborated with physicist Max Tegmark and his team at Improve the News to bring needed context to fast-moving news stories using timely forecasts by embedding forecasts in over 800 articles. We also had cameo appearances in a number of mainstream outlets, including The Atlantic, the Washington Post, BBC News, the Financial Times, Fast Company, the leading Spanish paper El Pais, and major Dutch outlet De Telegraaf, and were featured in the new, comprehensive CEA article that provides an introduction to Effective Altruism, as well as theoretical physicist Sean Carroll’s podcast Mindscape.
A Million Predictions, and a Hackathon: Metaculus achieved a fantastic milestone, surpassing 1,000,000 predictions over more than 7,000 questions. To celebrate, we hosted our first-ever data science hackathon, giving talented teams and individuals across the globe access to a rich trove of forecasting data. Unsurprisingly, the forecasting community displayed admirable creativity and rigor in their projects, and selecting the winners was no easy task. We also hosted a Week of Talks on forecasting, spanning a diverse range of topics from theory to practice.
We upgraded our infrastructure and shipped new features.
Upgraded Tech Stack: In 2022, we rewrote nearly the entire Metaculus application, modernizing the Metaculus tech stack to support our 2023 product roadmap. We have a number of enhancements and new features in store, big and small, and we can’t wait to share them with the community.
Question Groups & Fan Graphs: We released question groups and fan graphs, enabling the grouping of related questions and the visualization of their forecasts in series.
Private Forecasting Spaces & Language Localization: To support partner projects, including internationally, we developed private forecasting spaces that enable confidential forecasting for a group or organization of any size, as well as the ability to translate the Metaculus interface into any language.
Tournament Scoring & Leaderboards: We updated our tournament scoring and leaderboard systems to bring greater rigor and clarity to how we reward and incentivize forecasting skill.
We collaborated on research and published reports on the biggest potential risks of 2022.
Nuclear Escalation in Ukraine: After the invasion of Ukraine, policymakers and the public became increasingly concerned about the prospect of nuclear escalation. We recruited a team of Metaculus Pro Forecasters to make their judgments on key questions and provide their rationales, all of which were drawn up in a full nuclear risk report.
Predicting the Omicron BA.1 Wave: In partnership with the University of Virginia’s Biocomplexity Institute and the Virginia Department of Health, we co-authored a paper demonstrating the accuracy and robustness of using Metaculus’s COVID-19 Omicron variant forecasting ensembles in combination with computational models by providing valuable real-time forecasts.
Mpox Rapid Information Aggregation: When an unexpected number of Mpox cases were reported in early May 2022, we organized a rapid forecasting response to gauge the potential scope and impact of the outbreak. Working with our research partner Tom McAndrew at Lehigh University, we co-authored a paper on the efficacy of rapid human judgment forecasting, which was published (in record time) in The Lancet Digital Health.
Forecasting the US-China AI & Nuclear Landscape: With our partners at the Institute for Security and Technology, we launched an initiative evaluating intervention points in the US-China nuclear relationship, with a special focus on the integration of AI into nuclear command, control, and communications systems. The resulting report combines insights from nuclear and policy subject matter experts and Metaculus Pro Forecasters.
I’m extremely proud of what we’ve accomplished as a team, and I’m deeply grateful for the support of our partner organizations and the forecasting community. If you’re excited by what we’re doing and would like to get in touch, please do feel free to grab some time with me, or shoot our team a note.
Onward,
Gaia Dempsey
CEO, Metaculus