It’s worse, even, in a certain way, than that: the existence of optimizing systems organized around a certain idea of “natural class” feeds back into more observers observing data that is distributed according to this idea of “natural class”, leading to more optimizing systems being built around that idea of “natural class”, and so on.
Once a certain idea of “natural class” gains a foothold somewhere, observers will make real changes in the world that further suggest this particular idea of “natural class” to others, and this forms a feedback loop.
It’s worse, even, in a certain way, than that: the existence of optimizing systems organized around a certain idea of “natural class” feeds back into more observers observing data that is distributed according to this idea of “natural class”, leading to more optimizing systems being built around that idea of “natural class”, and so on.
Once a certain idea of “natural class” gains a foothold somewhere, observers will make real changes in the world that further suggest this particular idea of “natural class” to others, and this forms a feedback loop.