If you get paperclips slightly wrong, you get something equally bad (staples is the usual example, but the point is that any slight difference is about equally bad), but if you get FAI slightly wrong, you don’t get something equally good. This breaks the symmetry.
I think if you get paperclips slightly wrong, you get a crash of some kind. If I get a ray-tracer slightly wrong, it doesn’t trace electrons instead of photons.
edit: To clarify. It’s about definition of person vs definition of paperclip. You need a very broad definition of person for FAI, so that it won’t misidentify a person as non-person (misidentifying dolphins as persons won’t be a big problem), and you need a very narrow definition of paperclip for uFAI, so that a person holding two papers together is not a paperclip. It’s not always intuitive how broad definitions compare to narrow in difficulty, but it is worth noting that it is ridiculously hard to define paperclip making so that a Soviet factory anxious to maximize the paperclips would make anything at all, while it wasn’t particularly difficult to define what a person is (or to define what ‘money’ are so that capitalist paperclip factory would make paperclips to maximize profit).
If you get paperclips slightly wrong, you get something equally bad (staples is the usual example, but the point is that any slight difference is about equally bad), but if you get FAI slightly wrong, you don’t get something equally good. This breaks the symmetry.
I think if you get paperclips slightly wrong, you get a crash of some kind. If I get a ray-tracer slightly wrong, it doesn’t trace electrons instead of photons.
edit: To clarify. It’s about definition of person vs definition of paperclip. You need a very broad definition of person for FAI, so that it won’t misidentify a person as non-person (misidentifying dolphins as persons won’t be a big problem), and you need a very narrow definition of paperclip for uFAI, so that a person holding two papers together is not a paperclip. It’s not always intuitive how broad definitions compare to narrow in difficulty, but it is worth noting that it is ridiculously hard to define paperclip making so that a Soviet factory anxious to maximize the paperclips would make anything at all, while it wasn’t particularly difficult to define what a person is (or to define what ‘money’ are so that capitalist paperclip factory would make paperclips to maximize profit).