“So any identity needs to be constructed by systems within physics, and the boundaries are arbitrary.” is very insightful. Even very smart people still wonder “will my upload really be myself?” and “why would an AI want to rewrite itself?” without realizing that the definition of “self” might be less rigid than they imagine.
On the other hand, “If a system identifies a human as an important part of itself it will strive to protect it and its normal functioning, as we instinctively protect important parts of ourselves” is nothing more than anthroporphizing. The word that should have set off warning sirens here is “instinctively”: artificial systems act on a combination of designs and mistakes, but not on instincts.
I’d taboo the word instinct. I was going for as “a pre-created quick acting method for achieving a goal that doesn’t involve huge amount of computation”. What do you mean by it?
“The inherent inclination of an organism toward a particular behavior” seems to be the clearest definition I could find. The catch is then that AIs have no inherent inclinations. Animals all instinctively try to protect themselves, because the ones that didn’t mostly died. But an AI will only protect itself if that goal happens to be part of the design decisions and bugs it was programmed with. Self-protection may be more likely than many other goals, since “protect myself” is a useful subgoal for a huge class of other instrumental goals, but in those cases the definition of “myself” isn’t subject to redefinition.
“So any identity needs to be constructed by systems within physics, and the boundaries are arbitrary.” is very insightful. Even very smart people still wonder “will my upload really be myself?” and “why would an AI want to rewrite itself?” without realizing that the definition of “self” might be less rigid than they imagine.
On the other hand, “If a system identifies a human as an important part of itself it will strive to protect it and its normal functioning, as we instinctively protect important parts of ourselves” is nothing more than anthroporphizing. The word that should have set off warning sirens here is “instinctively”: artificial systems act on a combination of designs and mistakes, but not on instincts.
I’d taboo the word instinct. I was going for as “a pre-created quick acting method for achieving a goal that doesn’t involve huge amount of computation”. What do you mean by it?
“The inherent inclination of an organism toward a particular behavior” seems to be the clearest definition I could find. The catch is then that AIs have no inherent inclinations. Animals all instinctively try to protect themselves, because the ones that didn’t mostly died. But an AI will only protect itself if that goal happens to be part of the design decisions and bugs it was programmed with. Self-protection may be more likely than many other goals, since “protect myself” is a useful subgoal for a huge class of other instrumental goals, but in those cases the definition of “myself” isn’t subject to redefinition.