We caution against purely technical interpretations of privacy such as “the data never leaves the device.” Meredith Whittaker argues that on-device fraud detection normalizes always-on surveillance and that the infrastructure can be repurposed for more oppressive purposes. That said, technical innovations can definitely help.
I really do not know what you are expecting. On-device calculation using existing data and other data you choose to store only, the current template, is more privacy protecting than existing technologies.
She’s expecting, or at least asking, that certain things not be done on or off of the device, and that the distinction between on-device and off-device not be made excessively central to that choice.
If an outsider can access your device, they can always use their own AI to analyze the same data.
The experience that’s probably framing her thoughts here is Apple’s proposal to search through photos on people’s phones, and flag “suspicious” ones. The argument was that the photos would never leave your device… but that doesn’t really matter, because the results would have. And even if they had not, any photo that generated a false positive would have become basically unusable, with the phone refusing to do anything with it, or maybe even outright deleting it.
Similarly, a system that tries to detect fraud against you can easily be repurposed to detect fraud by you. To act on that detection, it has to report you to somebody or restrict what you can do. On-device processing of whatever kind can still be used against the interests of the owner of the device.
Suppose that there was a debate around the privacy implications of some on-device scanning that actually acted only in the user’s interest, but that involved some privacy concerns. Further suppose that the fact that it was on-device was used as an argument that there wasn’t a privacy problem. The general zeitgeist might absorb the idea that “on-device” was the same as “privacy-preserving”. “On device good, off device bad”.
A later transition from “in your interest” to “against your interest” could easily get obscured in any debate, buried under insistence that “It’s on-device”.
Yes, some people with real influence really, truly are that dumb, even when they’re paying close attention. And the broad sweep of opinion tends to come from people who aren’t much paying attention to begin with. It happens all the time in complicated policy arguments.
She’s expecting, or at least asking, that certain things not be done on or off of the device, and that the distinction between on-device and off-device not be made excessively central to that choice.
The experience that’s probably framing her thoughts here is Apple’s proposal to search through photos on people’s phones, and flag “suspicious” ones. The argument was that the photos would never leave your device… but that doesn’t really matter, because the results would have. And even if they had not, any photo that generated a false positive would have become basically unusable, with the phone refusing to do anything with it, or maybe even outright deleting it.
Similarly, a system that tries to detect fraud against you can easily be repurposed to detect fraud by you. To act on that detection, it has to report you to somebody or restrict what you can do. On-device processing of whatever kind can still be used against the interests of the owner of the device.
Suppose that there was a debate around the privacy implications of some on-device scanning that actually acted only in the user’s interest, but that involved some privacy concerns. Further suppose that the fact that it was on-device was used as an argument that there wasn’t a privacy problem. The general zeitgeist might absorb the idea that “on-device” was the same as “privacy-preserving”. “On device good, off device bad”.
A later transition from “in your interest” to “against your interest” could easily get obscured in any debate, buried under insistence that “It’s on-device”.
Yes, some people with real influence really, truly are that dumb, even when they’re paying close attention. And the broad sweep of opinion tends to come from people who aren’t much paying attention to begin with. It happens all the time in complicated policy arguments.