I think we worry way too much about reputation concerns. These seem hypothetical to me, and if we just fund a lot of work some of it will be great and rise to the top, the rest will be mediocre and forgotten or ignored.
I think you’re overconfident that mediocre work will be “forgotten or ignored”. We don’t seem to have reliable metrics for measuring the goodness of alignment work. We have things like post karma and what high-status researchers are willing to say publicly about the work, but IMO these are not reliable metrics for the purpose of detecting mediocre work. (Partially due to Goodhart’s law; people who seek funding for alignment research probably tend to optimize for their posts getting high karma and their work getting endorsements from high-status researchers). FWIW I don’t think the reputation concerns are merely hypothetical at this point.
I guess, but I think of this as weighing two options:
worry about reputation and err towards false negatives in funding
don’t worry about reputation and err towards false positives in funding
I want to increase the chance that AI safety is addressed, and allowing more false positives in funding (since I assess most false positives to produce neutral outcomes relative to achieving AI safety) seems the better trade-off, so all else equal I prefer to worry less about reputation.
I think there are some arguments that reputation matters long term, but it’s not clear we have long enough for that to matter or that funding more stuff would actually hurt reputation, so lacking better arguments I remain convinced we should just fund more stuff more freely.
I think you’re overconfident that mediocre work will be “forgotten or ignored”. We don’t seem to have reliable metrics for measuring the goodness of alignment work. We have things like post karma and what high-status researchers are willing to say publicly about the work, but IMO these are not reliable metrics for the purpose of detecting mediocre work. (Partially due to Goodhart’s law; people who seek funding for alignment research probably tend to optimize for their posts getting high karma and their work getting endorsements from high-status researchers). FWIW I don’t think the reputation concerns are merely hypothetical at this point.
I guess, but I think of this as weighing two options:
worry about reputation and err towards false negatives in funding
don’t worry about reputation and err towards false positives in funding
I want to increase the chance that AI safety is addressed, and allowing more false positives in funding (since I assess most false positives to produce neutral outcomes relative to achieving AI safety) seems the better trade-off, so all else equal I prefer to worry less about reputation.
I think there are some arguments that reputation matters long term, but it’s not clear we have long enough for that to matter or that funding more stuff would actually hurt reputation, so lacking better arguments I remain convinced we should just fund more stuff more freely.