how moral or otherwise desirable would the story have been if half a billion years’ of sentient minds had been made to think, act and otherwise be in perfect accordance to what three days of awkward-tentacled, primitive rock fans would wish if they knew more, thought faster, were more the people they wished they were...
Eliezer answers,
A Friendly AI should not be a person. I would like to know at least enough about this “consciousness” business to ensure a Friendly AI doesn’t have (think it has) it. An even worse critical failure is if the AI’s models of people are people.
Suppose consciousness and personhood are mistaken concepts. Well, since personhood is an important concept in our legal systems, there is something in reality (namely, in the legal environment) that corresponds to the term “person”, but suppose there is not any “objective” way to determine whether an intelligent agent is a person where “objective” means without someone creating a legal definition or taking a vote or something like that. And suppose consciousness is a mistaken concept like phlogiston, the aether and the immortal soul are mistaken concepts. Then would not CEV be morally unjustifiable because there is no way to justify the enslavement—or “entrainment” if you want a less loaded term—of the FAI to the (extrapolated) desires of the humans?
Well, as it stands, words like “person” and “consciousness” do pick out certain things (Bill Gates is a conscious person) and leave out other things (this ballpoint pen is not a conscious person). So there must be SOME meaning there, even if it’s a confused or morally irrelevant meaning.
Hence, they’re not as mistaken as the idea of the immortal soul, since that is as far as I can tell incoherent. (It’s what moves my body… but it is non-physical? What does that MEAN?)
They could be as mistaken as the aether or phlogiston, but that’s actually not so bad. These are scientific concepts that turned out to be wrong and were replaced by better concepts (relativity and oxidation reactions respectively) that explained the same phenomena and also explained many other phenomena. So if we replace “consciousness” in this way, we’ll replace it with something that works just as well in the obvious cases and works even better in the tricky ones. (I am hopeful that we will do exactly this.)
RI asks,
how moral or otherwise desirable would the story have been if half a billion years’ of sentient minds had been made to think, act and otherwise be in perfect accordance to what three days of awkward-tentacled, primitive rock fans would wish if they knew more, thought faster, were more the people they wished they were...
Eliezer answers,
A Friendly AI should not be a person. I would like to know at least enough about this “consciousness” business to ensure a Friendly AI doesn’t have (think it has) it. An even worse critical failure is if the AI’s models of people are people.
Suppose consciousness and personhood are mistaken concepts. Well, since personhood is an important concept in our legal systems, there is something in reality (namely, in the legal environment) that corresponds to the term “person”, but suppose there is not any “objective” way to determine whether an intelligent agent is a person where “objective” means without someone creating a legal definition or taking a vote or something like that. And suppose consciousness is a mistaken concept like phlogiston, the aether and the immortal soul are mistaken concepts. Then would not CEV be morally unjustifiable because there is no way to justify the enslavement—or “entrainment” if you want a less loaded term—of the FAI to the (extrapolated) desires of the humans?
This is almost certainly the case IMO.
Well, as it stands, words like “person” and “consciousness” do pick out certain things (Bill Gates is a conscious person) and leave out other things (this ballpoint pen is not a conscious person). So there must be SOME meaning there, even if it’s a confused or morally irrelevant meaning.
Hence, they’re not as mistaken as the idea of the immortal soul, since that is as far as I can tell incoherent. (It’s what moves my body… but it is non-physical? What does that MEAN?)
They could be as mistaken as the aether or phlogiston, but that’s actually not so bad. These are scientific concepts that turned out to be wrong and were replaced by better concepts (relativity and oxidation reactions respectively) that explained the same phenomena and also explained many other phenomena. So if we replace “consciousness” in this way, we’ll replace it with something that works just as well in the obvious cases and works even better in the tricky ones. (I am hopeful that we will do exactly this.)