One category is novel epistemic infrastructure that doesn’t really exist in general and would benefit all communities—over the longer term those seem like the most important missing things (but we won’t be able to build them straightforwardly / over the short term and they won’t be built for the alignment community in particular, they are just things that are missing and important and will eventually be filled in). The most salient instances are better ways of dividing up the work of evaluating arguments and prioritizing things to look at, driven by reputation or implicit predictions about what someone will believe or find useful.
In general for this kind of innovation I think that almost all of the upside comes from people copying the small fraction of successful instances (each of which likely involves more work and a longer journey than could be justified for any small group).
The other category is stuff that could be set up more quickly / has more of a reference class. I don’t really have a useful answer for that, though I’m excited for eventually developing something a bit more like academic workshops that serve a community with a shared sense of the problem and who actually face similar day-to-day difficulties. I think this hasn’t really been the case for attempts at literal academic workshops; I expect it to probably grow out of coordination between alignment efforts at ML labs.
What sort of epistemic infrastructure do you think is importantly missing for the alignment research community?
One category is novel epistemic infrastructure that doesn’t really exist in general and would benefit all communities—over the longer term those seem like the most important missing things (but we won’t be able to build them straightforwardly / over the short term and they won’t be built for the alignment community in particular, they are just things that are missing and important and will eventually be filled in). The most salient instances are better ways of dividing up the work of evaluating arguments and prioritizing things to look at, driven by reputation or implicit predictions about what someone will believe or find useful.
In general for this kind of innovation I think that almost all of the upside comes from people copying the small fraction of successful instances (each of which likely involves more work and a longer journey than could be justified for any small group).
The other category is stuff that could be set up more quickly / has more of a reference class. I don’t really have a useful answer for that, though I’m excited for eventually developing something a bit more like academic workshops that serve a community with a shared sense of the problem and who actually face similar day-to-day difficulties. I think this hasn’t really been the case for attempts at literal academic workshops; I expect it to probably grow out of coordination between alignment efforts at ML labs.