Do you think that the outputs of human philosophers of mind, or physicists thinking about consciousness, can’t be accurately modeled by computational processes, even with access to humans? If they can be predicted or heard, then they can be deferred to.
CEV is supposed to extrapolate our wishes “if we knew more”, and the AI may be so sure that consciousness doesn’t really exist in some fundamental ontological sense that it will override human philosophers’ conclusions and extrapolate them as if they also thought consciousness doesn’t exist in this ontological sense. (ETA: I think Eliezer has talked specifically about fixing people’s wrong beliefs before starting to extrapolate them.) I share a similar concern, not so much about this particular philosophical problem, but that the AI will be wrong on some philosophical issue and reach some kind of disastrous or strongly suboptimal conclusion.
I share a similar concern, not so much about this particular philosophical problem, but that the AI will be wrong on some philosophical issue and reach some kind of disastrous or strongly suboptimal conclusion.
There’s a possibility that we are disastrously wrong about our own philosophical conclusions. Consciousness itself may be ethically monstrous in a truly rational moral framework. Especially when you contrast the desire for immortality with the heat death. What is the utility of 3^^^3 people facing an eventual certain death versus even just 2^^^2 or a few trillion?
I don’t think there’s a high probability that consciousness itself will turn out to be the ultimate evil but it’s at least a possibility. A more subtle problem may be that allowing consciousness to exist in this universe is evil. It may be far more ethical to only allow consciousness inside simulations with no defined end and just run them as long as possible with the inhabitants blissfully unaware of their eventual eternal pause. They won’t cease to exist so much as wait for some random universe to exist that just happens to encode their next valid state...
They won’t cease to exist so much as wait for some random universe to exist that just happens to encode their next valid state...
You could say the same of anyone who has ever died, for some sense of “valid” … This, and similar waterfall-type arguments lead me to suspect that we haven’t satisfactorily defined what it means for something to “happen.”
You could say the same of anyone who has ever died, for some sense of “valid” … This, and similar waterfall-type arguments lead me to suspect that we haven’t satisfactorily defined what it means for something to “happen.”
It depends on the natural laws the person lived under. The next “valid” state of a dead person is decomposition. I don’t find the waterfall argument compelling because the information necessary to specify the mappings is more complex than the computed function itself.
I’m hearing an invocation of the Anti-Zombie Principle here, i.e.: “If simulations of human philosophers of mind will talk about consciousness, they will do so for the same reasons that human philosophers do, namely, that they actually have consciousness to talk about” …
I’m hearing an invocation of the Anti-Zombie Principle here, i.e.: “If simulations of human philosophers of mind will talk about consciousness, they will do so for the same reasons that human philosophers do,
Yes.
namely, that they actually have consciousness to talk about” …
Okay, to clarify: If ‘consciousness’ refers to anything, it refers to something possessed both by human philosophers and accurate simulations of human philosophers. So one of the following must be true: ① human philosophers can’t be accurately simulated, ② simulated human philosophers have consciousness, or ③ ‘consciousness’ doesn’t refer to anything.
Dualists needn’t grant your first sentence, claiming epiphenomena. I am talking about whether mystical mind features would screw up the ability of an AI to carry out our aims, not arguing for physicalism (here).
I agree that a totally accurate simulation of a philosopher ought to arrive at the same conclusions as the original. But a totally accurate simulation of a human being is incredibly hard to obtain.
I’ve mentioned that I have a problem with outsourcing FAI design to sim-humans, and that I have a problem with the assumption of “state-machine materialism”. These are mostly different concerns. Outsourcing to sim-humans is just wildly impractical, and it distracts real humans from gearing up to tackle the problems of FAI design directly. Adopting state-machine materialism is something you can do, right now, and it will shape your methods and your goals.
The proverbial 500-subjective-year congress of sim-philosophers might be able to resolve the problem of state-machine materialism for you, but then so would the discovery of communications from an alien civilization which had solved the problem. I just don’t think you can rely on either method, and I also think real humans do have a chance of solving the ontological problem by working on it directly.
Do you think that the outputs of human philosophers of mind, or physicists thinking about consciousness, can’t be accurately modeled by computational processes, even with access to humans? If they can be predicted or heard, then they can be deferred to.
CEV is supposed to extrapolate our wishes “if we knew more”, and the AI may be so sure that consciousness doesn’t really exist in some fundamental ontological sense that it will override human philosophers’ conclusions and extrapolate them as if they also thought consciousness doesn’t exist in this ontological sense. (ETA: I think Eliezer has talked specifically about fixing people’s wrong beliefs before starting to extrapolate them.) I share a similar concern, not so much about this particular philosophical problem, but that the AI will be wrong on some philosophical issue and reach some kind of disastrous or strongly suboptimal conclusion.
There’s a possibility that we are disastrously wrong about our own philosophical conclusions. Consciousness itself may be ethically monstrous in a truly rational moral framework. Especially when you contrast the desire for immortality with the heat death. What is the utility of 3^^^3 people facing an eventual certain death versus even just 2^^^2 or a few trillion?
I don’t think there’s a high probability that consciousness itself will turn out to be the ultimate evil but it’s at least a possibility. A more subtle problem may be that allowing consciousness to exist in this universe is evil. It may be far more ethical to only allow consciousness inside simulations with no defined end and just run them as long as possible with the inhabitants blissfully unaware of their eventual eternal pause. They won’t cease to exist so much as wait for some random universe to exist that just happens to encode their next valid state...
You could say the same of anyone who has ever died, for some sense of “valid” … This, and similar waterfall-type arguments lead me to suspect that we haven’t satisfactorily defined what it means for something to “happen.”
It depends on the natural laws the person lived under. The next “valid” state of a dead person is decomposition. I don’t find the waterfall argument compelling because the information necessary to specify the mappings is more complex than the computed function itself.
I’m hearing an invocation of the Anti-Zombie Principle here, i.e.: “If simulations of human philosophers of mind will talk about consciousness, they will do so for the same reasons that human philosophers do, namely, that they actually have consciousness to talk about” …
Yes.
Not necessarily, in the mystical sense.
Okay, to clarify: If ‘consciousness’ refers to anything, it refers to something possessed both by human philosophers and accurate simulations of human philosophers. So one of the following must be true: ① human philosophers can’t be accurately simulated, ② simulated human philosophers have consciousness, or ③ ‘consciousness’ doesn’t refer to anything.
Dualists needn’t grant your first sentence, claiming epiphenomena. I am talking about whether mystical mind features would screw up the ability of an AI to carry out our aims, not arguing for physicalism (here).
I agree that a totally accurate simulation of a philosopher ought to arrive at the same conclusions as the original. But a totally accurate simulation of a human being is incredibly hard to obtain.
I’ve mentioned that I have a problem with outsourcing FAI design to sim-humans, and that I have a problem with the assumption of “state-machine materialism”. These are mostly different concerns. Outsourcing to sim-humans is just wildly impractical, and it distracts real humans from gearing up to tackle the problems of FAI design directly. Adopting state-machine materialism is something you can do, right now, and it will shape your methods and your goals.
The proverbial 500-subjective-year congress of sim-philosophers might be able to resolve the problem of state-machine materialism for you, but then so would the discovery of communications from an alien civilization which had solved the problem. I just don’t think you can rely on either method, and I also think real humans do have a chance of solving the ontological problem by working on it directly.