I somewhat agree with the sentiment. We found it a bit hard to scope the idea correctly. Defining subcategories as you suggest and then diving into each of them is definitely on the list of things that I think are necessary to make progress on them.
I’m not sure the post would have been better if we used a more narrow title, e.g. “We need a science of capability evaluations” because the natural question then would be “But why not for propensity tests or for this other type of eval. I think the broader point of “when we do evals, we need some reason to be confident in the results no matter which kind of eval” seems to be true across all of them.
I somewhat agree with the sentiment. We found it a bit hard to scope the idea correctly. Defining subcategories as you suggest and then diving into each of them is definitely on the list of things that I think are necessary to make progress on them.
I’m not sure the post would have been better if we used a more narrow title, e.g. “We need a science of capability evaluations” because the natural question then would be “But why not for propensity tests or for this other type of eval. I think the broader point of “when we do evals, we need some reason to be confident in the results no matter which kind of eval” seems to be true across all of them.