Why I expect AGIs to be sentient / conscious, whether we wanted that or not
I think it could be worse than that.
As you speculate, we will earlier or later figure out how to engineer consciousness. And I mean your strong version of consciousness where people agree that the thing is conscious, e.g., not just because it responds like a conscious agent but because we can point out how it happens, observe the process, and generally see the analogy to how we do it. If we can engineer it, we can optimize it and reduce it to the minimum needed components and computational power to admit consciousness in this sense. Humans got consciousness because it was evolutionary useful or even a side-effect. It is not the core feature of the human brain, and most of the processing and learning of the brain deals with other things. Therefore I conjecture that not that much CPU/RAM will be needed for pure consciousness. I would guess that my laptop has enough for that. What will happen if some evil actor engineers small consciousnesses into their devices or even apps? May we reformat or uninstall them?
As you speculate, we will earlier or later figure out how to engineer consciousness.
I think we’re much further away from this than we are other problems with AGI. I agree that we will, at some point, be able to define consciousness in a way that will be accepted by those we currently agree are conscious by dint of their similarity to ourselves). I don’t know that it will match my or your intuitions about it today, and depending on what that agreement is, we may or may not be able to engineer it very much.
I strongly expect that as we progress in understanding, we’ll decide that it’s not sacred, and it’s OK to create and destroy some consciousnesses for the convenience of others. Heck, we don’t spend very much of our personal energy in preventing death of distant human strangers, though we try not to be personally directly responsible for deaths. I’m certainly not going to worry about reformatting a device that has a tiny consciousness any more than I worry about killing an ant colony that’s too close to my house. I may or may not worry about longevity of a human-sized consciousness, if it’s one of billions that are coming and going all the time. I have no intuitions about giant consciousnesses—maybe they’re utility monsters, maybe they’re just blobs of matter like the rest of us.
I strongly expect that as we progress in understanding, we’ll decide that it’s not sacred, and it’s OK to create and destroy some consciousnesses for the convenience of others.
That might be an outcome. In that case, we might decide that the sacredness of life is not tied to consciousness but something else.
Creating or preventing conscious experiences from happening has a moral valence equivalent to how that conscious experience feels. I expect most “artificial” conscious experiences created by machines to be neutral with respect to the pain-pleasure axis, for the same reason that randomly generated bitmaps rarely depict anything.
I expect most “artificial” conscious experiences created by machines to be neutral with respect to the pain-pleasure axis, for the same reason that randomly generated bitmaps rarely depict anything.
What if the machine is an AGI algorithm, and right now it’s autonomously inventing a new better airplane design? Would you still expect that?
The space of possible minds/algorithms is so vast, and that problem is so open-ended, that it would be a remarkable coincidence if such an AGI had a consciousness that was anything like ours. Most details of our experience are just accidents of evolution and history.
Does an airplane have a consciousness like a bird? “Design an airplane” sounds like a more specific goal, but in the space of all possible minds/algorithms that goal’s solutions are quite undetermined, just like flight.
My airplane comment above was a sincere question, not a gotcha or argument or anything. I was a bit confused about what you were saying and was trying to suss it out. :) Thanks.
I do disagree with you though. Hmm, here’s an argument. Humans invented TD learning, and then it was discovered that human brains (and other animals) incorporate TD learning too. Similarly, self-supervised learning is widely used in both AI and human brains, as are distributed representations and numerous other things.
If our expectation is “The space of possible minds/algorithms is so vast…” then it would be a remarkable coincidence for TD learning to show up independently in brains & AI, right? How would you explain that?
I would propose instead an alternative picture, in which there are a small number of practical methods which can build intelligent systems. In that picture (which I subscribe to, more or less), we shouldn’t be too surprised if future AGI has a similar architecture to the human brain. Or in the most extreme version of that picture, we should be surprised if it doesn’t! (At least, they’d be similar in terms of how they use RL and other types of learning / inference algorithms; I don’t expect the innate drives a.k.a. reward functions to be remotely the same, at least not by default.)
I agree with Stephen’s point about convergent results from directed design (or evolution in the case of animals). I don’t agree that consciousness and moral valence are closely coupled such that it would incur a performance loss to decouple them. Therefore, I suspect it will be a nearly costless choice to make morally relevant vs irrelevant AGI, and that we very much morally ought to choose to make morally-irrelevant AGI. To do otherwise would be possible, as Gunnar describes, but morally monstrous. Unfortunately some people do morally monstrous things sometimes. I am unclear on how to prevent this particular form of monstrosity.
I think it could be worse than that.
As you speculate, we will earlier or later figure out how to engineer consciousness. And I mean your strong version of consciousness where people agree that the thing is conscious, e.g., not just because it responds like a conscious agent but because we can point out how it happens, observe the process, and generally see the analogy to how we do it. If we can engineer it, we can optimize it and reduce it to the minimum needed components and computational power to admit consciousness in this sense. Humans got consciousness because it was evolutionary useful or even a side-effect. It is not the core feature of the human brain, and most of the processing and learning of the brain deals with other things. Therefore I conjecture that not that much CPU/RAM will be needed for pure consciousness. I would guess that my laptop has enough for that. What will happen if some evil actor engineers small consciousnesses into their devices or even apps? May we reformat or uninstall them?
I think we’re much further away from this than we are other problems with AGI. I agree that we will, at some point, be able to define consciousness in a way that will be accepted by those we currently agree are conscious by dint of their similarity to ourselves). I don’t know that it will match my or your intuitions about it today, and depending on what that agreement is, we may or may not be able to engineer it very much.
I strongly expect that as we progress in understanding, we’ll decide that it’s not sacred, and it’s OK to create and destroy some consciousnesses for the convenience of others. Heck, we don’t spend very much of our personal energy in preventing death of distant human strangers, though we try not to be personally directly responsible for deaths. I’m certainly not going to worry about reformatting a device that has a tiny consciousness any more than I worry about killing an ant colony that’s too close to my house. I may or may not worry about longevity of a human-sized consciousness, if it’s one of billions that are coming and going all the time. I have no intuitions about giant consciousnesses—maybe they’re utility monsters, maybe they’re just blobs of matter like the rest of us.
That might be an outcome. In that case, we might decide that the sacredness of life is not tied to consciousness but something else.
Creating or preventing conscious experiences from happening has a moral valence equivalent to how that conscious experience feels. I expect most “artificial” conscious experiences created by machines to be neutral with respect to the pain-pleasure axis, for the same reason that randomly generated bitmaps rarely depict anything.
What if the machine is an AGI algorithm, and right now it’s autonomously inventing a new better airplane design? Would you still expect that?
The space of possible minds/algorithms is so vast, and that problem is so open-ended, that it would be a remarkable coincidence if such an AGI had a consciousness that was anything like ours. Most details of our experience are just accidents of evolution and history.
Does an airplane have a consciousness like a bird? “Design an airplane” sounds like a more specific goal, but in the space of all possible minds/algorithms that goal’s solutions are quite undetermined, just like flight.
My airplane comment above was a sincere question, not a gotcha or argument or anything. I was a bit confused about what you were saying and was trying to suss it out. :) Thanks.
I do disagree with you though. Hmm, here’s an argument. Humans invented TD learning, and then it was discovered that human brains (and other animals) incorporate TD learning too. Similarly, self-supervised learning is widely used in both AI and human brains, as are distributed representations and numerous other things.
If our expectation is “The space of possible minds/algorithms is so vast…” then it would be a remarkable coincidence for TD learning to show up independently in brains & AI, right? How would you explain that?
I would propose instead an alternative picture, in which there are a small number of practical methods which can build intelligent systems. In that picture (which I subscribe to, more or less), we shouldn’t be too surprised if future AGI has a similar architecture to the human brain. Or in the most extreme version of that picture, we should be surprised if it doesn’t! (At least, they’d be similar in terms of how they use RL and other types of learning / inference algorithms; I don’t expect the innate drives a.k.a. reward functions to be remotely the same, at least not by default.)
I agree with Stephen’s point about convergent results from directed design (or evolution in the case of animals). I don’t agree that consciousness and moral valence are closely coupled such that it would incur a performance loss to decouple them. Therefore, I suspect it will be a nearly costless choice to make morally relevant vs irrelevant AGI, and that we very much morally ought to choose to make morally-irrelevant AGI. To do otherwise would be possible, as Gunnar describes, but morally monstrous. Unfortunately some people do morally monstrous things sometimes. I am unclear on how to prevent this particular form of monstrosity.