I agree we might end up in a world like that, where it proves impossible to make a decent safety case.
I don’t think that thing I said is consistent with “impossible to make a safety case good enough to make TAI worth building”? I think you can probably make a safety case which gets to around 1-5% risk while having AIs that are basically black boxs and which are very powerful, but not arbitrarily powerful. (Such a safety case might require decently expensive measures.) (1-5% risk can be consistent with this being worth it—e.g. if there is an impending hot war. That said, it is still a horrifying level of risk that demands vastly more investment.)
See e.g. control for what one part of this safety case could look like. I think that control can go quite far.
Other parts could look like:
A huge array of model organisms experiments which convince us that P(serious misalignment) is low (or low given XYZ adjustments to training).
coup probes or other simple runtime detection/monitoring techiques.
ELK/honesty techniques which seem to work in a wide variety of cases.
Ensuring AIs closely imitate humans and generalize similarly to humans in the cases we checked.
Ensuring that AIs are very unlikely to be schemers via making sure most of their reasoning happens in CoT and their forward passes are quite weak. (I often think about this from the perspective of control, but it can also be considered separately.)
It’s possible you don’t think any of this stuff gets off the ground. Fair enough if so.
I don’t think that thing I said is consistent with “impossible to make a safety case good enough to make TAI worth building”? I think you can probably make a safety case which gets to around 1-5% risk while having AIs that are basically black boxs and which are very powerful, but not arbitrarily powerful. (Such a safety case might require decently expensive measures.) (1-5% risk can be consistent with this being worth it—e.g. if there is an impending hot war. That said, it is still a horrifying level of risk that demands vastly more investment.)
See e.g. control for what one part of this safety case could look like. I think that control can go quite far.
Other parts could look like:
A huge array of model organisms experiments which convince us that P(serious misalignment) is low (or low given XYZ adjustments to training).
coup probes or other simple runtime detection/monitoring techiques.
ELK/honesty techniques which seem to work in a wide variety of cases.
Ensuring AIs closely imitate humans and generalize similarly to humans in the cases we checked.
Ensuring that AIs are very unlikely to be schemers via making sure most of their reasoning happens in CoT and their forward passes are quite weak. (I often think about this from the perspective of control, but it can also be considered separately.)
It’s possible you don’t think any of this stuff gets off the ground. Fair enough if so.