Hmm, you’ve convinced me that it would be best to start with a non-person. In fact, if becoming a person was Friendly, the FAI would self-program that way anyhow. (Assumption: a FAI is still a runaway unstoppable super-mind, it’s just one whose goals are aligned with Friendliness. So if it decides X is Friendly, bet your bippy X will happen, and fast, and it won’t take no for an answer.)
What I’m still confused by is: what does person / non-person really mean? When I try to think of the idea of “person” I keep running into human assumptions.
What human / humane traits would you exclude or build in to seed AI? Which ones do you expect to be emergent given high enough universal intelligence? (I’m thinking like: “is it possible to speak, and understand, and not be a person?” Am I being parochial?)
Hmm, you’ve convinced me that it would be best to start with a non-person. In fact, if becoming a person was Friendly, the FAI would self-program that way anyhow. (Assumption: a FAI is still a runaway unstoppable super-mind, it’s just one whose goals are aligned with Friendliness. So if it decides X is Friendly, bet your bippy X will happen, and fast, and it won’t take no for an answer.)
What I’m still confused by is: what does person / non-person really mean? When I try to think of the idea of “person” I keep running into human assumptions.
What human / humane traits would you exclude or build in to seed AI? Which ones do you expect to be emergent given high enough universal intelligence? (I’m thinking like: “is it possible to speak, and understand, and not be a person?” Am I being parochial?)