I have actually thought about that a lot, too, and my conclusion is that we should not beat ourselves up trying to figure out precisely what the difference might be between these nuanced versions of the idea, because the people who are proposing this idea in the first place have not themselves been clear enough about what is meant
They are clear that they don’t mean AIs rigid behaviour is the result of it assessing its own inferrential processes as infallible … that is what the controversy is all about..
The problem is, we are talking about an AI, and some people talk as if the AI can run its planning engine, then feel compelled to obey the planning engine … while at the same time being fully cognizant of evidence that the planning engine produced a crappy plan.
That is just what The Genie Knows but doesn’t Care is supposed to answer. I think it succeeds in showing that a fairly specific architecture would behave that way, but fails in it’s intended goal of showing that this behaviour is universal or likely.
They are clear that they don’t mean AIs rigid behaviour is the result of it assessing its own inferrential processes as infallible … that is what the controversy is all about..
That is just what The Genie Knows but doesn’t Care is supposed to answer. I think it succeeds in showing that a fairly specific architecture would behave that way, but fails in it’s intended goal of showing that this behaviour is universal or likely.