In my experience, computer systems currently get out of my control by doing exactly what I ordered them to do, which is frequently different than I what I wanted them to do.
Whether or not a system is “just following orders” doesn’t seem to be a good metric for it being under your control.
While I agree that it is out of control if the behavior is neither ordered nor wanted, I think it’s also very possible for the system to get out of control while doing exactly what you ordered it to, but not what you meant for it to.
The argument I’m making is approximately the same as the one we see in the outcome pump example.
This is to say, while a system that is doing something neither ordered nor wanted is definitely out of control, it does not follow that a system that is doing exactly what it was ordered to do is necessarily under your control.
In my experience, computer systems currently get out of my control by doing exactly what I ordered them to do, which is frequently different than I what I wanted them to do.
Whether or not a system is “just following orders” doesn’t seem to be a good metric for it being under your control.
How does “just following orders,” a la Nuremberg, bear upon this issue? It’s out of control when its behavior is neither ordered nor wanted.
While I agree that it is out of control if the behavior is neither ordered nor wanted, I think it’s also very possible for the system to get out of control while doing exactly what you ordered it to, but not what you meant for it to.
The argument I’m making is approximately the same as the one we see in the outcome pump example.
This is to say, while a system that is doing something neither ordered nor wanted is definitely out of control, it does not follow that a system that is doing exactly what it was ordered to do is necessarily under your control.