For the cost-effectiveness modelling, we estimated the time per claim as a lognormal with 90% confidence interval 10 to 70 minutes and mean of 30 minutes. This was based on survey data from participants.
I think that what’s important is 1) the opportunity cost of the time, rather than the actual number of minutes, and 2) the fact that Elizabeth’s work can be outsourced/parallelised at all, even if it takes others a bit longer than her.
It’s unclear to me whether I should think of the forecasters as more replaceable than Elizabeth. If they’re all generalist researchers, having “a bunch of generalist researchers do generalist research for the same amount of time as the original researcher” doesn’t seem obviously scalable.
(That said, my current belief is that this work was pretty interesting and important overall)
The forecasters were only quite loosely selected for “some forecasting experience”. Some of them I know are very able forecasters, others are people much less experienced, and who I don’t think are affiliated that much with the rationality or effective altruism communities.
The forecasters had a few weeks to estimate all the claims. There was no limit within that time.
That said, the incentives were capped, so I think the forecasters spent time in relative proportion to their expected payouts. You can see a bit more detail in the Cost-effectiveness section.
Perhaps I missed this, but how long were the forecasters expected to spend per claim?
For the cost-effectiveness modelling, we estimated the time per claim as a lognormal with 90% confidence interval 10 to 70 minutes and mean of 30 minutes. This was based on survey data from participants.
Okay, so in quite a few cases the forecasters spent more time on a question than Elizabeth did? That seems like an important point to mention.
Yes.
Curious why you think it’s important?
I think that what’s important is 1) the opportunity cost of the time, rather than the actual number of minutes, and 2) the fact that Elizabeth’s work can be outsourced/parallelised at all, even if it takes others a bit longer than her.
It’s unclear to me whether I should think of the forecasters as more replaceable than Elizabeth. If they’re all generalist researchers, having “a bunch of generalist researchers do generalist research for the same amount of time as the original researcher” doesn’t seem obviously scalable.
(That said, my current belief is that this work was pretty interesting and important overall)
The forecasters were only quite loosely selected for “some forecasting experience”. Some of them I know are very able forecasters, others are people much less experienced, and who I don’t think are affiliated that much with the rationality or effective altruism communities.
The forecasters had a few weeks to estimate all the claims. There was no limit within that time.
That said, the incentives were capped, so I think the forecasters spent time in relative proportion to their expected payouts. You can see a bit more detail in the Cost-effectiveness section.