But it also provides incredibly easy interpretability, because these systems think in English.
I’m not sure this point will stand because it might be cheaper to have them think in their own language: https://www.lesswrong.com/posts/bNCDexejSZpkuu3yz/you-can-use-gpt-4-to-create-prompt-injections-against-gpt-4
So will the maximally curious AI be curious about what would happen if you genetically modified all humans to become unicorns?