One of boyd’s examples is a pretty straightforward feedback loop, recognizable to anyone with the slightest degree of systems engineering:
Consider, for example, what’s happening with policing practices, especially as computational systems allow precincts to distribute their officers “fairly.” In many jurisdictions, more officers are placed into areas that are deemed “high risk.” This is deemed to be appropriate at a societal level. And yet, people don’t think about the incentive structures of policing, especially in communities where the law is expected to clear so many warrants and do so many arrests per month. When they’re stationed in algorithmically determined “high risk” communities, they arrest in those communities, thereby reinforcing the algorithms’ assumptions.
This system — putting more crime-detecting police officers (who have a nontrivial false-positive rate) in areas that are currently considered “high crime”, and shifting them out of areas currently considered “low crime” — diverges under many sets of initial conditions and incentive structures. You don’t even have to posit racism or classism to get these effects (although those may contribute to failing to recognize them as a problem); under the right (wrong) conditions, as t → ∞, the noise (that is, the error in the original believed distribution of crime) dominates the signal.
The ninth of Robert Peel’s principles of ethical policing is surprisingly relevant: “To recognise always that the test of police efficiency is the absence of crime and disorder, and not the visible evidence of police action in dealing with them.”[1]
One of boyd’s examples is a pretty straightforward feedback loop, recognizable to anyone with the slightest degree of systems engineering:
This system — putting more crime-detecting police officers (who have a nontrivial false-positive rate) in areas that are currently considered “high crime”, and shifting them out of areas currently considered “low crime” — diverges under many sets of initial conditions and incentive structures. You don’t even have to posit racism or classism to get these effects (although those may contribute to failing to recognize them as a problem); under the right (wrong) conditions, as t → ∞, the noise (that is, the error in the original believed distribution of crime) dominates the signal.
The ninth of Robert Peel’s principles of ethical policing is surprisingly relevant: “To recognise always that the test of police efficiency is the absence of crime and disorder, and not the visible evidence of police action in dealing with them.” [1]