I like this frame, and I don’t recall seeing it already addressed.
What I have seen written about deceptiveness generally seems to assume that the AGI would be sufficiently capable of obfuscating its thoughts from direct queries and from any interpretability tools we have available that it could effectively make its plans for world domination in secret, unobserved by humans. That does seem like an even more effective strategy for optimizing its actual utility function than not bothering to think through such plans at all, if it’s able to do it. But it’s hard to do, and even thinking about it is risky.
I can imagine something like what you describe happening as a middle stage, for entities that are agentic enough to have (latent, probably misaligned since alignment is probably hard) goals, but not yet capable enough to think hard about how to optimize for them without being detected. It seems more likely if (1) almost all sufficiently powerful AI systems created by humans will actually have misaligned goals, (2) AIs are optimized very hard against having visibly misaligned cognition (selection of which AIs to keep being a form of optimization, in this context), and (3) our techniques for making misaligned cognition visible are more reliably able to detect an active process / subsystem doing planning towards goals than the mere latent presence of such goals. (3) seems likely, at least for a while and assuming we have any meaningful interpretability tools at all; it’s hard for me to imagine a detector of latent properties that doesn’t just always say “well, there are some off-distribution inputs that would make it do something very bad” for every sufficiently powerful AI, even one that was aligned-in-practice because those inputs would reliably never be given to it.
It seems like there might be a problem with this argument if the true vi are not just unknown, but adversarially chosen. For example, suppose the true v2 are the actual locations of a bunch of landmines, from a full set of possible landmine positions V2. We are trying to get a vehicle from A to B, and all possible paths go over some of the V2. We may know that the opponent placing the landmines only has n2 landmines to place. Furthermore, suppose each landmine only goes off with some probability p even if the vehicle drives over it. If we can mechanistically predict where the opponent placed the landmines, or even mechanistically derive a probability distribution over the landmine placements, this is no problem, we can just use that to minimize the expected probability of driving over a landmine that goes off. However, suppose we can’t predict the opponent that way, but we do know the opponent is trying to maximize the probability that the vehicle drives over a landmine that isn’t a dud. It seems like we need to use game theory here, not just probability theory, to figure out what mixed strategy the opponent would be using to maximize the probability that we drive over a landmine, and then use that game-theoretic strategy to choose a mixed strategy for which path to take. It seems like the game theory here involves a step where we look for the worst (according to our utility function) probability distribution over where the landmines are placed, because this is how the opponent will have actually chosen where to put the landmines. Doesn’t this look a lot like using μ rather than U as our utility function?