If the paperclip maker’s architecture is a set of constrained boxes, where each box does a tiny, well defined part of the problem of making paperclips, and is being evaluated by other boxes that ultimately trace their goals and outputs to human defined goals and sensor data, it’s not going anywhere. It’s not even sentient in that there’s no memory in the system for anything like self reflection. Every piece of memory is specific to the needs of a component. You have to build reliable real-time systems like this, other architectures won’t function in a way that wouldn’t fail so often as to be economically infeasible. (because paperclips have very low value, while robotic waldos and human lives are expensive)
This is what I mean by I’m on the side of the spaceplane designers. I don’t know how another, more flexible architecture would even function, in the same way in this story they don’t know how to build a vehicle that doesn’t depend on air.
If the paperclip maker’s architecture is a set of constrained boxes, where each box does a tiny, well defined part of the problem of making paperclips, and is being evaluated by other boxes that ultimately trace their goals and outputs to human defined goals and sensor data, it’s not going anywhere. It’s not even sentient in that there’s no memory in the system for anything like self reflection. Every piece of memory is specific to the needs of a component. You have to build reliable real-time systems like this, other architectures won’t function in a way that wouldn’t fail so often as to be economically infeasible. (because paperclips have very low value, while robotic waldos and human lives are expensive)
This is what I mean by I’m on the side of the spaceplane designers. I don’t know how another, more flexible architecture would even function, in the same way in this story they don’t know how to build a vehicle that doesn’t depend on air.