Iterative amplification schemes work by having each version i+1 trained by previous iteration i; and, whenever version i fails at finding a good answer (low confidence in the prediction), punting the question to i−1 , until it reaches the human overseer at i=0, which is the ground truth for our purposes.
There is a dynamic like this in amplification, but I don’t think this is quite what happens.
In particular, the AI at level i-1 generally isn’t any more expensive than the AI at level i. The main dynamic for punting down is some way of breaking the problem into simpler pieces (security amplification requires you to take out-of-distribution data and, after enough steps, to reduce it to in-distribution subtasks), rather than punting to a weaker but more robust agent.
The problem is that, while training the anomaly detection at level i , we cannot sample from the distribution Di+N , because we simply don’t know it yet. As we run amplification, we extend both the capability of answering questions and the range of questions that come up.
I do agree with the basic point here though: as you do amplification the distribution shifts, and you need to be able to get a guarantee on a distribution that you can’t sample from. I talk about this problem in this post. It’s clearly pretty hard, but it does look significantly easier than the full problem to me.
There is a dynamic like this in amplification, but I don’t think this is quite what happens.
In particular, the AI at level i-1 generally isn’t any more expensive than the AI at level i. The main dynamic for punting down is some way of breaking the problem into simpler pieces (security amplification requires you to take out-of-distribution data and, after enough steps, to reduce it to in-distribution subtasks), rather than punting to a weaker but more robust agent.
I do agree with the basic point here though: as you do amplification the distribution shifts, and you need to be able to get a guarantee on a distribution that you can’t sample from. I talk about this problem in this post. It’s clearly pretty hard, but it does look significantly easier than the full problem to me.