Would it be helpful to think about something like “what Brier score will a person in the reference class of “people-similar-to-Eliezer_2022-in-all-relevant-ways” have after making a bunch of predictions on Metaculus?” Perhaps we should set up this sort of question on Metaculus or Manifold? Though I would probably refrain from explicitly mentioning Eliezer in it.
That might be possible, but it would take a lot of effort to make resolvable. Who is “similar to Eliezer”, and how do we define that in advance? Which forecasting questions are we going to check their Brier score on? Which not-like-Eliezer forecasters are we going to compare them to (since scores are much more informative for ranking forecasters, than in isolation)? Etc. I’d rather people like Eliezer just placed their forecasts properly!
Would it be helpful to think about something like “what Brier score will a person in the reference class of “people-similar-to-Eliezer_2022-in-all-relevant-ways” have after making a bunch of predictions on Metaculus?” Perhaps we should set up this sort of question on Metaculus or Manifold? Though I would probably refrain from explicitly mentioning Eliezer in it.
That might be possible, but it would take a lot of effort to make resolvable. Who is “similar to Eliezer”, and how do we define that in advance? Which forecasting questions are we going to check their Brier score on? Which not-like-Eliezer forecasters are we going to compare them to (since scores are much more informative for ranking forecasters, than in isolation)? Etc. I’d rather people like Eliezer just placed their forecasts properly!