While I’m not particularly optimistic about BCI solutions either, I don’t think this story is strong evidence against them. Suppose that the BCI took the form of an exocortex that expanded the person’s brain functions and also significantly increased their introspective awareness to the level of an inhumanly good meditator. This would effectively allow for constant monitoring of what subagents within the person’s mind were getting activated in conversation, flagging those to the person’s awareness in real time and letting the person notice when they were getting manipulated in ways that the rest of their mind-system didn’t endorse. That kind of awareness tends to also allow defending against manipulation attempts since one does not blend with the subagents to a similar degree and can then better integrate them with the rest of the system after the issue has been noticed.
Ordinary humans can learn to get higher introspective awareness through practices such as meditation, but it’s very hard if not impossible to get to a point where you’d never be emotionally triggered since sufficiently strong emotions seem to trigger some kind of biological override. But an exocortex might be built to remain unaffected by that override and allow one to maintain high introspective awareness regardless. In that case, one might be able to more directly communicate with untrusted entities without getting hacked by them.
While I’m not particularly optimistic about BCI solutions either, I don’t think this story is strong evidence against them. Suppose that the BCI took the form of an exocortex that expanded the person’s brain functions and also significantly increased their introspective awareness to the level of an inhumanly good meditator. This would effectively allow for constant monitoring of what subagents within the person’s mind were getting activated in conversation, flagging those to the person’s awareness in real time and letting the person notice when they were getting manipulated in ways that the rest of their mind-system didn’t endorse. That kind of awareness tends to also allow defending against manipulation attempts since one does not blend with the subagents to a similar degree and can then better integrate them with the rest of the system after the issue has been noticed.
Ordinary humans can learn to get higher introspective awareness through practices such as meditation, but it’s very hard if not impossible to get to a point where you’d never be emotionally triggered since sufficiently strong emotions seem to trigger some kind of biological override. But an exocortex might be built to remain unaffected by that override and allow one to maintain high introspective awareness regardless. In that case, one might be able to more directly communicate with untrusted entities without getting hacked by them.