Is it rather that the model space might not have the capacity to correctly imitate the human?
Paul wrote in a parallel subthread, “Against mimicry is mostly motivated by the case of imitating an amplified agent.” In the case of IDA, you’re bound to run out of model capacity eventually as you keep iterating the amplification and distillation.
I guess instead we’re instead imagining a problem where there’s no great metric (e.g. text answers to questions)?
Paul wrote in a parallel subthread, “Against mimicry is mostly motivated by the case of imitating an amplified agent.” In the case of IDA, you’re bound to run out of model capacity eventually as you keep iterating the amplification and distillation.
I’m pretty sure that’s it.