It seems Kurzweil’s view of AI is different from this groups. This group seems to concentrate on autonomous AI on the one hand, against which humanity as it exists now must be protected. It seems to me Kurzweil, on the other hand, sees AI as enhancements to the human. That the human would be able to install hardware which would augment her brain (and her body, actually.)
Extrapolating my view of Kurzweil’s view, I suppose that over time, the humans become less and less human as the enhancements they can adopt become an increasing fraction of the “human” which is designing and adopting the next generation of enhanced enhancements. Maybe the Kurzweilian view takes the universe to the same place that lesswrongian autonomous UAI’s take the universe: with mechanical intelligence taking over primary spot from biological. But the transition seems more comfortable to contemplate, since at every step something that at least used to be human is taking the next step.
In some sense, I see CEV and attempts to insure FAI over UAI as a modern equivalent of Chasidic Judaism. Preserving a “better” past against the encroachment of modern “sin” without actually succeeding in doing that. In some sense, scientifically educated rational atheists are the UAI threat to the orthodox believers.
In some sense, I think I’d rather be extinct than to live in a community guarded from the harsh realities by FAI, living out a cartoon version of a life which is centuries out of date.
I suppose I think of my in-group as “dominant intelligences” instead of “humanity.” And so I feel more kinship with the dominant intelligences, even when they are no longer human.
In some sense, I think I’d rather be extinct than to live in a community guarded from the harsh realities by FAI, living out a cartoon version of a life which is centuries out of date.
If CEV produces whatever people value, do you think it would produce the above because you have different values than other people, or from some other error?
Also, it seems to me that avoiding a new technology (CEV) specifically because it will make your life too easy has a lot in common with living in a false world which is centuries out of date.
If CEV produces whatever people value, do you think it would produce the above because you have different values than other people… ?
Yes. And thank you for phrasing it that way so I understand that is at least one explanation for my concern.
It seems beyond likely to me that the CEV you get will depend heavily on just who you include in your definition of “humans” whos volition must be considered in defining CEV. Even if CEV were intended to be just that subset of volitions that “everybody” would agree on (if they were smart enough), will your definition of everybody include paranoid schizophrenics? People born with severely deformed brains? Sociopaths? Republicans? The French? My point being that our intuition is of a “common definition of human we can all agree on” but the reality of 7 billion live humans plus a few billion easy to anticipate might have a non-intuitively large variation across its volition.
So if CEV includes a “veto power” in its definition granted to all humans defined broadly enough to include sociopaths, we lose many of the values that allow us to work cooperatively.
Further concerning me, I think it is likely that humanity benefits from a diversity in values. At one level, societies with different values have different levels of success under different challenges, and in something like survival of the fittest, the societies that thrive have values that work better than those that don’t. At another level, within a society diversity in values serves the group: the nurturers are caretakers, the nerds technologists, the sociopaths become leaders and work in security.
CEV as I have heard it described sounds like a core of values, a kernel that all FAI operating systems would have to include. It doesn’t sound like a set of values or a core of meta-values that would somehow incorporate in a single kernel all the variation in values that has served humanity so well.
So yes, I am concerned that CEV is impossible, but perhaps not provably impossible, that any actual attempts to build a CEV will have more to do with the values of the people building CEV rather than some undefinable generalization of humanity.
Another concern: AI with a CEV constraint will necessarily be less adaptable than AI with no CEV constraint. So in the absence of complete totalitarian control over where AI can come from, non-CEV AI once created would eventually out-compete CEV-based AI anyway, and all that effort would have been for naught.
Finally, what I think of as a Kurzweilian paradigm of AI makes more sense to me than the idea of independent AIs that exist separately from humans. Kurzweil seems to me talks more of enhancing existing humans, building on modules, interfacing us better, and so on. Eventually, perhaps, the enhanced human is 99% enhancement and 1% human and so it becomes a matter of attitude whether you still think of it as human. Do you think CEV is something that applies to building up enhanced humans (instead of independent entities)?
Also, it seems to me that avoiding a new technology (CEV) specifically because it will make your life too easy has a lot in common with living in a false world which is centuries out of date.
The ultimate technology that makes my life too easy is wireheading. That was just fun to say, I don’t actually recognize it as a great response to your point, but I throw it out there because there might be more to it than I am allowing.
I suppose one man’s dystopia is another woman’s brave new world. I don’t think being locked in the matrix by FAIs who know this is the way to keep us safe is something I reject because it is too easy. I reject it because it is essentially wireheading.
My main concern about CEV is that it winds up protecting us from nothing, that it was a waste of time and effort. But this does go along with my belief that any CEV would be one that would not incorporate my values around the importance of diversity in values, and my values around being able to reject and fight violently against other humans that had values that I found sufficiently threatening to my values.
I appreciate the questions, it is nice to sharpen my ideas a little bit. I admit I have hardly sharpened them to a mathematical precision by any means, but if you see any obvious intuition pumps working against me, I’d love to hear them.
I read the book several times already and it makes me more and more pessimistic. Even if we make SI to follow CEV, at some point it might decide to drop it. Its SI above all, it can find ways to do anything. Yet we can’t survive without SI.
So SEV proposal is as good and as bad as any other proposal.
My only hope is that moral values could be as fundamental as laws of nature. So a very superintelligent AI will be very moral. Then we’ll be saved. If not, then it will create Hell for all people and keep them there for eternity (meaning that even death could be a better way out yet SI will not let people die). What should we do?
I think the burden of answering your “why?” question falls to those who feel sure that we have the wisdom to create superintelligent, super-creative lifeforms who could think outside the box regarding absolutely everything except ethical values. For those, they would inevitably stay on the rails that we designed for them. The thought “human monkey-minds wouldn’t on reflection approve of x” would forever stop them from doing x.
In effect, we want superintelligent creatures to ethically defer to us the way Euthyphro deferred to the gods. But as we all know, Socrates had a devastating comeback to Euthyphro’s blind deference: We should not follow the gods simply because they want something, or because they command something. We should only follow them if the things they want are right. Insofar as the gods have special insight into what’s right, then we should do what they say, but only because what they want is right. On the other hand, if the gods’ preferences are morally arbitrary, we have no obligation to heed them.
How long will it take a superintelligence to decide that Socrates won this argument? Milliseconds? Then how do we convince the superintelligence that our preferences (or CEV extrapolated preferences) track genuine moral rightness, rather than evolutionary happenstance? How good a case do we have that humans possess a special insight into what is right that the superintelligence doesn’t have, so that the superintelligence will feel justified in deferring to our values?
If you think this is an automatic slam dunk for humans.… Why?
I don’t think there’s any significant barrier to making a superintelligence that deferred to us for approval on everything. It would be a pretty lousy superintelligence, because it would essentially be crippled by its strict adherence to our wishes (making it excruciatingly slow) but it would work, and it would be friendly.
Given that there is a very significant barrier to making children that deferred to us for approval on everything, why do you think the barrier would be reduced if instead of children, we made a superintelligent AI?
I thought it’s supposed to work like this: The first generation of AI are designed by us. The superintelligence is designed by them, the AI. We have initial control over what their utility functions are. I’m looking for a good reason for we should expect to retain that control beyond the superintelligence transition. No such reasons have been given here.
A different way to put a my point: Would a superintelligence be able to reason about ends? If so, then it might find itself disagreeing with our conclusions. But if not—if we design it to have what for humans would be a severe cognitive handicap—why should we think that subsequent generations of SuperAI will not repair that handicap?
You’re making the implicit assumption that a runaway scenario will happen. A ‘cognitive handicap’ would, in this case, simply prevent the next generation AI from being built at all.
As I’m saying, it would be a lousy SI and not very useful. But it would be friendly.
We never bother running a computer program unless we don’t know the output and we know an important fact about the output.
—Marcello Herreshoff
In this case, one of the important facts must be that it won’t go around changing its motivational structure. If it isn’t, we’re screwed for the reason you give.
What do you think of CEV as a proposal?
It seems Kurzweil’s view of AI is different from this groups. This group seems to concentrate on autonomous AI on the one hand, against which humanity as it exists now must be protected. It seems to me Kurzweil, on the other hand, sees AI as enhancements to the human. That the human would be able to install hardware which would augment her brain (and her body, actually.)
Extrapolating my view of Kurzweil’s view, I suppose that over time, the humans become less and less human as the enhancements they can adopt become an increasing fraction of the “human” which is designing and adopting the next generation of enhanced enhancements. Maybe the Kurzweilian view takes the universe to the same place that lesswrongian autonomous UAI’s take the universe: with mechanical intelligence taking over primary spot from biological. But the transition seems more comfortable to contemplate, since at every step something that at least used to be human is taking the next step.
In some sense, I see CEV and attempts to insure FAI over UAI as a modern equivalent of Chasidic Judaism. Preserving a “better” past against the encroachment of modern “sin” without actually succeeding in doing that. In some sense, scientifically educated rational atheists are the UAI threat to the orthodox believers.
In some sense, I think I’d rather be extinct than to live in a community guarded from the harsh realities by FAI, living out a cartoon version of a life which is centuries out of date.
I suppose I think of my in-group as “dominant intelligences” instead of “humanity.” And so I feel more kinship with the dominant intelligences, even when they are no longer human.
If CEV produces whatever people value, do you think it would produce the above because you have different values than other people, or from some other error?
Also, it seems to me that avoiding a new technology (CEV) specifically because it will make your life too easy has a lot in common with living in a false world which is centuries out of date.
Yes. And thank you for phrasing it that way so I understand that is at least one explanation for my concern.
It seems beyond likely to me that the CEV you get will depend heavily on just who you include in your definition of “humans” whos volition must be considered in defining CEV. Even if CEV were intended to be just that subset of volitions that “everybody” would agree on (if they were smart enough), will your definition of everybody include paranoid schizophrenics? People born with severely deformed brains? Sociopaths? Republicans? The French? My point being that our intuition is of a “common definition of human we can all agree on” but the reality of 7 billion live humans plus a few billion easy to anticipate might have a non-intuitively large variation across its volition.
So if CEV includes a “veto power” in its definition granted to all humans defined broadly enough to include sociopaths, we lose many of the values that allow us to work cooperatively.
Further concerning me, I think it is likely that humanity benefits from a diversity in values. At one level, societies with different values have different levels of success under different challenges, and in something like survival of the fittest, the societies that thrive have values that work better than those that don’t. At another level, within a society diversity in values serves the group: the nurturers are caretakers, the nerds technologists, the sociopaths become leaders and work in security.
CEV as I have heard it described sounds like a core of values, a kernel that all FAI operating systems would have to include. It doesn’t sound like a set of values or a core of meta-values that would somehow incorporate in a single kernel all the variation in values that has served humanity so well.
So yes, I am concerned that CEV is impossible, but perhaps not provably impossible, that any actual attempts to build a CEV will have more to do with the values of the people building CEV rather than some undefinable generalization of humanity.
Another concern: AI with a CEV constraint will necessarily be less adaptable than AI with no CEV constraint. So in the absence of complete totalitarian control over where AI can come from, non-CEV AI once created would eventually out-compete CEV-based AI anyway, and all that effort would have been for naught.
Finally, what I think of as a Kurzweilian paradigm of AI makes more sense to me than the idea of independent AIs that exist separately from humans. Kurzweil seems to me talks more of enhancing existing humans, building on modules, interfacing us better, and so on. Eventually, perhaps, the enhanced human is 99% enhancement and 1% human and so it becomes a matter of attitude whether you still think of it as human. Do you think CEV is something that applies to building up enhanced humans (instead of independent entities)?
The ultimate technology that makes my life too easy is wireheading. That was just fun to say, I don’t actually recognize it as a great response to your point, but I throw it out there because there might be more to it than I am allowing.
I suppose one man’s dystopia is another woman’s brave new world. I don’t think being locked in the matrix by FAIs who know this is the way to keep us safe is something I reject because it is too easy. I reject it because it is essentially wireheading.
My main concern about CEV is that it winds up protecting us from nothing, that it was a waste of time and effort. But this does go along with my belief that any CEV would be one that would not incorporate my values around the importance of diversity in values, and my values around being able to reject and fight violently against other humans that had values that I found sufficiently threatening to my values.
I appreciate the questions, it is nice to sharpen my ideas a little bit. I admit I have hardly sharpened them to a mathematical precision by any means, but if you see any obvious intuition pumps working against me, I’d love to hear them.
I read the book several times already and it makes me more and more pessimistic. Even if we make SI to follow CEV, at some point it might decide to drop it. Its SI above all, it can find ways to do anything. Yet we can’t survive without SI. So SEV proposal is as good and as bad as any other proposal. My only hope is that moral values could be as fundamental as laws of nature. So a very superintelligent AI will be very moral. Then we’ll be saved. If not, then it will create Hell for all people and keep them there for eternity (meaning that even death could be a better way out yet SI will not let people die). What should we do?
Why?
I think the burden of answering your “why?” question falls to those who feel sure that we have the wisdom to create superintelligent, super-creative lifeforms who could think outside the box regarding absolutely everything except ethical values. For those, they would inevitably stay on the rails that we designed for them. The thought “human monkey-minds wouldn’t on reflection approve of x” would forever stop them from doing x.
In effect, we want superintelligent creatures to ethically defer to us the way Euthyphro deferred to the gods. But as we all know, Socrates had a devastating comeback to Euthyphro’s blind deference: We should not follow the gods simply because they want something, or because they command something. We should only follow them if the things they want are right. Insofar as the gods have special insight into what’s right, then we should do what they say, but only because what they want is right. On the other hand, if the gods’ preferences are morally arbitrary, we have no obligation to heed them.
How long will it take a superintelligence to decide that Socrates won this argument? Milliseconds? Then how do we convince the superintelligence that our preferences (or CEV extrapolated preferences) track genuine moral rightness, rather than evolutionary happenstance? How good a case do we have that humans possess a special insight into what is right that the superintelligence doesn’t have, so that the superintelligence will feel justified in deferring to our values?
If you think this is an automatic slam dunk for humans.… Why?
I don’t think there’s any significant barrier to making a superintelligence that deferred to us for approval on everything. It would be a pretty lousy superintelligence, because it would essentially be crippled by its strict adherence to our wishes (making it excruciatingly slow) but it would work, and it would be friendly.
Given that there is a very significant barrier to making children that deferred to us for approval on everything, why do you think the barrier would be reduced if instead of children, we made a superintelligent AI?
The ‘child’ metaphor for SI is not very accurate. SIs can be designed and, most importantly, we have control over what their utility functions are.
I thought it’s supposed to work like this: The first generation of AI are designed by us. The superintelligence is designed by them, the AI. We have initial control over what their utility functions are. I’m looking for a good reason for we should expect to retain that control beyond the superintelligence transition. No such reasons have been given here.
A different way to put a my point: Would a superintelligence be able to reason about ends? If so, then it might find itself disagreeing with our conclusions. But if not—if we design it to have what for humans would be a severe cognitive handicap—why should we think that subsequent generations of SuperAI will not repair that handicap?
You’re making the implicit assumption that a runaway scenario will happen. A ‘cognitive handicap’ would, in this case, simply prevent the next generation AI from being built at all.
As I’m saying, it would be a lousy SI and not very useful. But it would be friendly.
As friendly as we are, anyway.
Because we are not SI, so we don’t know what it will do and why. It might.
We never bother running a computer program unless we don’t know the output and we know an important fact about the output. —Marcello Herreshoff
In this case, one of the important facts must be that it won’t go around changing its motivational structure. If it isn’t, we’re screwed for the reason you give.