In theory, I don’t understand how to do optimization at all—other than methods that look like finding a bunch of stuff that I don’t understand, and seeing if it accomplishes my goal. But this is exactly the kind of thing that’s most prone to spinning up adversarial subsystems.
One form of optimization that comes to mind that is importantly different, is to carefully consider a prototypical system, think about how the parts interplay, and identify how the system can be improved, and create a new prototype that one can expect to be better. While practical application of this type of optimization will still often involve producing and testing multiple prototypes, it differs from back-propogation or stochastic hill-climbing because the new system will be better than the prototype it is based on due to reasons that the optimizing agent actually understands.
I think capitalism staddles the line between these two modes: an inventor or well-function firm will optimize by making modifications that they actually understand, but the way the market optimizes products is how Scott and Abram describe it: you get a lot of stuff that you don’t attempt to understand deeply, and choose whichever one looks best. While I am generally a fan of capitalism, there are examples of “adversarial subsystems” that have been spun up as a result of markets—the slave trade and urban pollution (e.g. smog) come to mind.
In “Emedded Agency”, Scott and Abram write:
One form of optimization that comes to mind that is importantly different, is to carefully consider a prototypical system, think about how the parts interplay, and identify how the system can be improved, and create a new prototype that one can expect to be better. While practical application of this type of optimization will still often involve producing and testing multiple prototypes, it differs from back-propogation or stochastic hill-climbing because the new system will be better than the prototype it is based on due to reasons that the optimizing agent actually understands.
I think capitalism staddles the line between these two modes: an inventor or well-function firm will optimize by making modifications that they actually understand, but the way the market optimizes products is how Scott and Abram describe it: you get a lot of stuff that you don’t attempt to understand deeply, and choose whichever one looks best. While I am generally a fan of capitalism, there are examples of “adversarial subsystems” that have been spun up as a result of markets—the slave trade and urban pollution (e.g. smog) come to mind.