I agree that this is worth investigating. The problem seems to be that there are many confounders that are very difficult to control for. That makes it difficult to tackle the problem in practice.
For example, in the simulated world you mention I would not be surprised to see a large number of hallucinations show up. How could we be sure that any given issue we find is a meaningful problem that is worth investigating, and not a random hallucination?
I agree that this is worth investigating. The problem seems to be that there are many confounders that are very difficult to control for. That makes it difficult to tackle the problem in practice.
For example, in the simulated world you mention I would not be surprised to see a large number of hallucinations show up. How could we be sure that any given issue we find is a meaningful problem that is worth investigating, and not a random hallucination?
If you think about it from a “dangerous capability eval” perspective, the fact that it can happen at all is enough evidence of concern