Zero day exploits in general are not a thing you have to worry about; it isn’t an analogy that applies to humans because we’re far more robust than software. A zero day exploit on an operating system can give you total control of it; a ‘zero day exploit’ like junk food can make you consume 5% more calories per day than you otherwise do.
The “just five percent more calories” example reveals nicely how meaningless this heuristic is. The vast majority of people alive today are the effective mental subjects of some religion, political party, national identity, or combination of the three, no magical backdoor access necessary; the confirmed tools and techniques are sufficient to ruin lives or convince people to do things completely counter to their own interests. And there are intermediate stages of effectiveness that political lobbying can ratchet up along, between the ones they’re at now and total control.
AI companies have not devoted significant effort to human thought steering, unless you mean “try to drive engagement on a social media website”; they are too busy working on AI.
AI companies are not going to try to weaponize “human thought steering” against AI safety
The premise of the above post is not that AI companies are going to try to weaponize “human thought steering” against AI safety. The premise of the above post is that AI companies are going to develop technology that can be used to manipulate people’s affinities and politics, Intel agencies will pilfer it or ask for it, and then it’s going to be weaponized, to a degree of much greater effectiveness than they have been able to afford historically. I’m ambivalent about the included story in particular being carried out, but if you care about anything (such as AI safety), it’s probably necessary that you keep your utilityfunction intact.
The “just five percent more calories” example reveals nicely how meaningless this heuristic is. The vast majority of people alive today are the effective mental subjects of some religion, political party, national identity, or combination of the three, no magical backdoor access necessary; the confirmed tools and techniques are sufficient to ruin lives or convince people to do things completely counter to their own interests. And there are intermediate stages of effectiveness that political lobbying can ratchet up along, between the ones they’re at now and total control.
The premise of the above post is not that AI companies are going to try to weaponize “human thought steering” against AI safety. The premise of the above post is that AI companies are going to develop technology that can be used to manipulate people’s affinities and politics, Intel agencies will pilfer it or ask for it, and then it’s going to be weaponized, to a degree of much greater effectiveness than they have been able to afford historically. I’m ambivalent about the included story in particular being carried out, but if you care about anything (such as AI safety), it’s probably necessary that you keep your utilityfunction intact.