Just to make another note, “Solving the problem in theory” is also equivalent to the [forward training algorithm](https://www.cs.cmu.edu/~sross1/publications/Ross-AIStats10-paper.pdf), which preceded DAgger by the same authors.
I do think there are some interesting ideas to consider in the alignment setting. For example, the chunk size k is equivalent to the number of roll-out steps in IL. “Chunking” the roll-out to a fixed window is a common optimization if the task has a long time horizon and the expert is expensive to query. On the other hand, longer roll-outs provide stronger guarantees on how well the learned policy matches the expert.
Classically, this is a simple tradeoff between performance and speed. But, as you mention k must also be kept intentionally small so that the AI can not detect it is being trained on human generations. How does one choose the chunk size to favor both strong alignment and avoid discrimination? Dynamic roll-out strategies have been proposed in the IL literature, though I’m not very familiar.
Output tokens certainly do not scale linearly, even with a KV cache. The KV cache means you don’t need to recompute the k/q/v vectors for each of the previous tokens, but you still need to compute n kq dot products for the (n+1)’st token.