Here’s a problem for you, which I’m not sure fits the requirements, but might: How do you learn whether an AI has been trained to use Gricean communication (e.g. “I interpret your words by modeling you as saying them because you model me as interpreting them, and so on until further recursion isn’t fruitful”) without being able to read its source code and check its functioning against some specification of recursive agential modeling?
Here’s a problem for you, which I’m not sure fits the requirements, but might: How do you learn whether an AI has been trained to use Gricean communication (e.g. “I interpret your words by modeling you as saying them because you model me as interpreting them, and so on until further recursion isn’t fruitful”) without being able to read its source code and check its functioning against some specification of recursive agential modeling?