How much have you explored the REASONS that brainwashing is seen as not cool, while quiet rational-seeming chat is perfectly fine? Are you sure it’s only about efficacy?
I worry that there’s some underlying principle missing from the conversation, about agentiness and “free will” of humans, which you’re trying to preserve without defining. It’d be much stronger to identify the underlying goals and include them as terms in the AI’s utility function(s).
No, but I’m pretty sure efficacy plays a role. Look at the (stereotypical) freakout from some conservative parents about their kids attending university; it’s not really about the content or the methods, but because changes in values or beliefs are expected to some degree.
Ok. The obvious followup is “under what conditions is it a bad thing?” Your college example is a good one—are you saying you want to prevent AIs from making similar changes (but on a perhaps larger scale) that university does to students?
Well, there’s a formal answer: if an AI can, in condition C, convince any human of belief B for any B, then condition C is not sufficient to constrain the AI’s power, and the process is unlikely to be truth-tracking.
That’s a sufficient condition for C being insufficient, but not a necessary one.
How much have you explored the REASONS that brainwashing is seen as not cool, while quiet rational-seeming chat is perfectly fine? Are you sure it’s only about efficacy?
I worry that there’s some underlying principle missing from the conversation, about agentiness and “free will” of humans, which you’re trying to preserve without defining. It’d be much stronger to identify the underlying goals and include them as terms in the AI’s utility function(s).
No, but I’m pretty sure efficacy plays a role. Look at the (stereotypical) freakout from some conservative parents about their kids attending university; it’s not really about the content or the methods, but because changes in values or beliefs are expected to some degree.
Ok. The obvious followup is “under what conditions is it a bad thing?” Your college example is a good one—are you saying you want to prevent AIs from making similar changes (but on a perhaps larger scale) that university does to students?
Well, there’s a formal answer: if an AI can, in condition C, convince any human of belief B for any B, then condition C is not sufficient to constrain the AI’s power, and the process is unlikely to be truth-tracking.
That’s a sufficient condition for C being insufficient, but not a necessary one.