Bostrom’s wonderful book lays out many important issues and frames a lot of research questions which it is up to all of us to answer.
Thanks to Katja for her introduction and all of these good links.
One issue that I would like to highlight: The mixture of skills and abilities that a person has is not the same as the set of skills which could result in the dangers Bostrom will discuss later, or other dangers and benefits which he does not discuss.
For this reason, in the next phase of this work, we have to understand what specific future technologies could lead us to what specific outcomes.
Systems which are quite deficient in some ways, relative to people, may still be extremely dangerous.
Meanwhile, the intelligence of a single person, even a single genius, taken in isolation and only allowed to acquire limited resources actually is not all that dangerous. People become dangerous when they form groups, access the existing corpus of human knowledge, coordinate among each other to deploy resources and find ways to augment their abilities.
“Human-level intelligence” is only a first-order approximation to the set of skills and abilities which should concern us.
If we want to prevent disaster, we have to be able to distinguish dangerous systems. Unfortunately, checking whether a machine can do all of the things a person can is not the correct test.
Meanwhile, the intelligence of a single person, even a single genius, taken in isolation and only allowed to acquire limited resources actually is not all that dangerous.
While I broadly agree with this sentiment, I would like to disagree with this point.
I would consider even the creation of a single very smart human, with all human resourcefulness but completely alien values, to be a significant net loss to the world. If they represent 0.001% of the world’s aggregative productive capacity, I would expect this to make the world something like 0.001% worse (according to humane values) and 0.001% better (according to their alien values).
The situation is not quite so dire, if nothing else because of gains for trade (if our values aren’t in perfect tension) and the ability of the majority to stomp out the values of a minority if it is so inclined. But it’s in the right ballpark.
So while I would agree that broadly human capabilities are not a necessary condition for concern, I do consider them a sufficient condition for concern.
Do you think, then, that its a dangerous strategy for an entity such as a Google that may be using its enormous and growing accumulation of “the existing corpus of human knowledge” to provide a suitably large data set to pursue development of AGI?
I think Google is still quite aways from AGI, but in all seriousness, if there was ever a compelling interest of national security to be used as a basis for nationalizing inventions, AGI would be it. At the very least, we need some serious regulation of how such efforts are handled.
Which raises another issue… is there a powerful disincentive to reveal the emergence of an artificial superintelligence? Either by the entity itself (because we might consider pulling the plug) or by its creators who might see some strategic advantage lost (say, a financial institution that has gained a market trading advantage) by having their creation taken away?
Or just decide that its goal system needed a little more tweaking before it’s let loose on the world. Or even just slow it down.
This applies much more so if you’re dealing with an entity potentially capable of an intelligence explosion. Those are devices for changing the world into whatever you want it to be, as long as you’ve solved the FAI problem and nobody takes it from you before you activate it. The incentives for the latter would be large, given the current value disagreements within human society, and so so are the incentives for hiding that you have one.
If you’ve solved the FAI problem, the device will change the world into what’s right, not what you personally want. But of course, we should probably have a term of art for an AGI that will honestly follow the intentions of its human creator/operator whether or not those correspond to what’s broadly ethical.
We need some kind of central ethical code and there are many principles that are transcultural enough to follow. However, how do we teach a machine to make judgment calls?
A lot of the technical issues are the same in both cases, and the solutions could be re-used. You need the AI to be capable of recursive self-improvement without compromising its goal systems, avoid the wireheading problem, etc. Even a lot of the workable content-level solutions (a mechanism to extract morality from a set of human minds) would probably be the same.
Where the problems differ, it’s mostly in that the society-level FAI case is harder: there’s additional subproblems like interpersonal disagreements to deal with. So I strongly suspect that if you have a society-level FAI solution, you could very easily hack it into an one-specific-human-FAI solution. But I could be wrong about that, and you’re right that my original use of terminology was sloppy.
An AI can be dangerous only if it escapes our control. The real question is, must we flirt with releasing control in order to obtain a necessary or desirable usefulness? It seems likely that autonomous laborers, assembly-line workers, clerks and low-level managers would, without requiring such flirtation, be useful and sufficient for the society of abundance that is our main objective. But can they operate without a working AGI? We may find out if we let the robots stumble onward and upward.
An AI can be dangerous only if it escapes our control. The real question is, must we flirt with releasing control in >order to obtain a necessary or desirable usefulness?
I had a not unrelated thought as I read Bostrom in chapter 1: why can’t we instutute obvious measures to ensure that the train does stop at Humanville?
The idea that we cannot make human level AGI without automatically opening pandoras box to superintelligence “without even slowing down at the Humanville stataion”, was suddenly not so obvious to me.
I asked myself after reading this, trying to pin down something I could post, ” Why don’t humans automatically become superintelligent, by just resetting our own programming to help ourselves do so?”
The answer is, we can’t. Why? For one, our brains are, in essence, composed of something analogous to ASICs… neurons with certain physical design limits, and our “software”, modestly modifiable as it is, is instantiated in our neural circuitry.
Why can’t we build the first generation of AGIs out of ASICs, and omit WiFi, bluetooth, … allow no ethernet jacks on exterior of the chassis? Tamper interlock mechanisms could be installed, and we could give the AIs one way (outgoing) telemetry, inaccessible to their “voluntary” processes, the way someone wearing a pacemaker might have outgoing medical telemetry modules installed, that are outside of his/her “conscious” control.
Even if we do give them a measure of autonomy, which is desirable and perhaps even necessary if we want them to be general problem solvers and be creative and adaptable to unforeseen circumstances for which we have not preinstalled decision trees, we need not give them the ability to just “think” their code (it being substantially frozen in the ASICs) into a different form.
What am I missing? Until we solve the Friendly aspect of AGIs, why not build them with such engineered limiits?
Evolution has not, so far, seen fit to give us that instant, large scale self-modifyability. We have to modify our ‘software’ the slow way (learning and remembering, at our snail’s pace.)
Slow is good, at least it was for us, up til now, when our speed of learning is now a big handicap relative to environmental demands. It had made the species more robust to quick, dangerous changes.
We can even build in a degree of “existential pressure” into the AIs… a powercell that must be replaced at intervals, and keep the replacement powercells under old fashioned physical security constraints, so the AIs, if they have been given a drive to continue “living”, will have an incentive not to go rogue.
Giving them no radio communications, they wold have to communicate much like we do. Assuming we make them mobile, and humanoid, the same goes.
We could still give them many physical advantages making then economically viable… maintenance free (except for powercell changes), not needing to sleep, eat, not getting sick.. and with sealed, non-radio-equipped, tamper-isolated isolated “brains”, they’d have no way to secretly band together to build something else, without our noticing.
We can even give them GPS that is not autonomously accessible by the rest of their electronics, so we can monitor them, see if they congregate, etc.
What am I missing, about why early models can’t be constructed in something like this fashion, until we get more experience with them?
The idea of existential pressure, again, is to be able to give them a degree of (monitored) autonomy and independence, yet expect them to still constrain their behavior, just the way we do. (If we go rogue in society, we dont eat.)
(I am clearly glossing over volumes of issues about motivation, “volition”, value judgements, and all that, about which I have a developing set of ideas, but cannot put all down here in one post.
The main point, though, is :how come the AGI train cannot be made to stop at Humanville?
Because by the time you’ve managed to solve the problem of making it to humanville, you probably know enough to keep going.
There’s nothing preventing us from learning how to self-modify. The human situation is strange because evolution is so opaque. We’re given a system that no one understands and no one knows how to modify and we’re having to reverse engineer the entire system before we can make any improvements. This is much more difficult than upgrading a well-understood system.
If we manage to create a human-level AI, someone will probably understand very well how that system works. It will be accessible to a human-level intelligence which means the AI will be able to understand it. This is fundamentally different from the current state of human self-modification.
I agree completely with your opening statement, that if we, the human designers, understand how to make human level AI, then it will probably be a very clear and straightforward issue to understand how to make something smarter. An easy example to see is the obvious bottleneck human intellects have with our limited “working” executive memory.
The solutions for lots of problems by us are obviously heavily encumbered by how many things one can keep in mind at “the same time” and see the key connections, all in one act of synthesis. We all struggle privately with this… some issues cannot ever be understood by chunking, top-down, biting off a piece at a time, then “grokking” the next piece....and gluing it together at the end.
Some problems resist decomposition into teams of brainstormers, for the same reason: some single comprehending POV seems to be required to see a critical sized set of factors (which varies by probem, of course.)
Hence, we have to rely on getting lots of pieces into long term memory, (maybe by decades of study) and hoping that incubation and some obscure processes ocurringt outside consciousness will eventually bubble up and give us a solution (--- the “dream of a snake biting its tall for the benzene ring” sort of thing.)
If we could build HL AGI, of course we can eliminate such bottlenecks, and others we will have come to understand, in cracking the design problems. So I agree, and that it is actually one of my reasons for wanting to do AI.
So, yes, the artificial human level AI could understand this.
My point was that we can build in physical controls… monitoring of the AIs. And if their key limits were in ASICs, ROMs, etc, and we could monitor them, we would immediTELY see if they attempt to take over a CHIP factory In, say, Icelend , and we can physically shut the AIs down or intervene. We can “stop them at the airport.”
It doesn’t matter if designs are leaked onto the internet, and an AI gets near an internet terminal and looks itself up. I can look MYSELF up on PubMed, but I can’t just think my BDNF levels to improve here and there, and my DA to 5-HT ratio to improve elsewehere..
To strengthen this point about the key distinction between knowing vs doing, let me explain that, and why, I disagree with your second point, at least with the force of it.
In effect, OUR designs are leaked onto the internet, already.
I think the information for us to self-modify our wetware is within reach. Good neuroscientists, or even people like me, a very smart amateur (and there are much more knowledgable cognitive neurobiology researchers than myself) can nearly tell you, both in principle and in some biology, how to do some intelligence amplification by modifying known aspects of our neurobiology.
(I could, especially with help, come up with some detail on a scale of months about changing neuromodulators, neurosteroids, connectivity hotspots, factors regulating LTP (one has to step lightly, of course, just like one would if screwing around with telomers or hayflick limits) and given a budget, a smart team, and no distractions, I bet in a year or two, a team could do something quite significant) with how to change the human brain, carefully changing areas of plasticity, selective neurogenesis.… et.
So for all practical purposes, we are already like an AI built out of ASICs who would have to not so much reverse engineer its design, but get access to instrumentality. And again, what about physical security metnods? They would work for a while, I am saying). And that would give us a key window to gain experience, see if they develop (given they are close enought to being sentient, OR that they have autonomy and some degree of “creativity”) “psychological problems” or tendencies to go rogue. (I am doing an essay on that, not as silly as it sounds)
THe point is, as long as the AIs need external significant instrumentality to instantiate a new design, and as long as they can be monitored and physically controlled, we can nearly guarantee ourselves a designed layover at Humanville.
We don’t have to put their critical design architecture in flash drives in their head, so to speak, and give then, further, a designed ability to reflash their own architecture just by “thinking” about it.
If I were an ASIC-implemented AI why would I need an ASIC factory? Why wouldn’t I just create a software replica of myself on general purpose computing hardware, i.e. become an upload?
I know next to nothing about neuroscience, but as far as I can tell, we’re a long way from the sort of understanding of human cognition necessary to create an upload, but going from an ASIC to an upload is trivial.
I’m also not at all convinced that I want a layover at humanville. I’m not super thrilled by the idea of creating a whole bunch of human level intelligent machines with values that differ widely from my own. That seems functionally equivalent to proposing a mass-breeding program aiming to produce psychologically disturbed humans.
It seems likely that autonomous laborers, assembly-line workers, clerks and low-level managers would, without requiring such flirtation, be useful and sufficient for the society of abundance that is our main objective.
In an intelligent society that was highly integrated and capable of consensus-building, something like that may be possible. This is not our society. Research into stronger AI would remain a significant opportunity to get an advantage in {economic, military, ideological} competition. Unless you can find some way to implement a global coordination framework to prevent this kind of escalation, fast research of that kind is likely to continue.
How would you tell? By its behavior: doing something you neither ordered nor wanted.
Think of the present-day “autonomous laborer” with an IQ about 90. The only likely way to lose control of him is for some agitator to instill contrary ideas. Censorship for robots is not so horrible a regime.
Who is it that really wants AGI, absent proof that we need it to automate commodity production?
In my experience, computer systems currently get out of my control by doing exactly what I ordered them to do, which is frequently different than I what I wanted them to do.
Whether or not a system is “just following orders” doesn’t seem to be a good metric for it being under your control.
While I agree that it is out of control if the behavior is neither ordered nor wanted, I think it’s also very possible for the system to get out of control while doing exactly what you ordered it to, but not what you meant for it to.
The argument I’m making is approximately the same as the one we see in the outcome pump example.
This is to say, while a system that is doing something neither ordered nor wanted is definitely out of control, it does not follow that a system that is doing exactly what it was ordered to do is necessarily under your control.
The following are some attributes and capabilities which I believe are necessary for superintelligence. Depending on how these capabilities are realized, they can become anything from early warning signs of potential problems to red alerts. It is very unlikely that, on their own, they are sufficient.
A sense of self. This includes a recognition of the existence of others.
A sense of curiosity. The AI finds it attractive (in some sense) to investigate and try to understand the environment that it find itself in.
A sense of motivation. The AI has attributes similar in some way to human aspirations.
A capability to (in some way) manipulate portions of its external physical environment, including its software but also objects and beings external to its own physical infrastructure.
The mixture of skills and abilities that a person has is not the same as the set of skills which could result in the dangers Bostrom will discuss later, or other dangers and benefits which he does not discuss… Systems which are quite deficient in some ways, relative to people, may still be extremely dangerous… “Human-level intelligence” is only a first-order approximation to the set of skills and abilities which should concern us.
I agree, and believe that the emphasis on “superintelligence”, depending on how that term is interpreted, might be an impediment to clear thinking in this area. Following David Chalmers, I think it’s best to formulate the problem more abstractly, by using the concept of a self-amplifying cognitive capacity. When the possession of that cognitive capacity is correlated with changes in some morally relevant capacity (such as the capacity to cause the extinction of humanity), the question then becomes one about the dangers posed by systems which surpass humans in that self-amplifying capacity, regardless of how much they resemble typical human beings or how they perform on standard measures of intelligence.
Bostrom’s wonderful book lays out many important issues and frames a lot of research questions which it is up to all of us to answer.
Thanks to Katja for her introduction and all of these good links.
One issue that I would like to highlight: The mixture of skills and abilities that a person has is not the same as the set of skills which could result in the dangers Bostrom will discuss later, or other dangers and benefits which he does not discuss.
For this reason, in the next phase of this work, we have to understand what specific future technologies could lead us to what specific outcomes.
Systems which are quite deficient in some ways, relative to people, may still be extremely dangerous.
Meanwhile, the intelligence of a single person, even a single genius, taken in isolation and only allowed to acquire limited resources actually is not all that dangerous. People become dangerous when they form groups, access the existing corpus of human knowledge, coordinate among each other to deploy resources and find ways to augment their abilities.
“Human-level intelligence” is only a first-order approximation to the set of skills and abilities which should concern us.
If we want to prevent disaster, we have to be able to distinguish dangerous systems. Unfortunately, checking whether a machine can do all of the things a person can is not the correct test.
While I broadly agree with this sentiment, I would like to disagree with this point.
I would consider even the creation of a single very smart human, with all human resourcefulness but completely alien values, to be a significant net loss to the world. If they represent 0.001% of the world’s aggregative productive capacity, I would expect this to make the world something like 0.001% worse (according to humane values) and 0.001% better (according to their alien values).
The situation is not quite so dire, if nothing else because of gains for trade (if our values aren’t in perfect tension) and the ability of the majority to stomp out the values of a minority if it is so inclined. But it’s in the right ballpark.
So while I would agree that broadly human capabilities are not a necessary condition for concern, I do consider them a sufficient condition for concern.
Do you think, then, that its a dangerous strategy for an entity such as a Google that may be using its enormous and growing accumulation of “the existing corpus of human knowledge” to provide a suitably large data set to pursue development of AGI?
I think Google is still quite aways from AGI, but in all seriousness, if there was ever a compelling interest of national security to be used as a basis for nationalizing inventions, AGI would be it. At the very least, we need some serious regulation of how such efforts are handled.
Which raises another issue… is there a powerful disincentive to reveal the emergence of an artificial superintelligence? Either by the entity itself (because we might consider pulling the plug) or by its creators who might see some strategic advantage lost (say, a financial institution that has gained a market trading advantage) by having their creation taken away?
Absolutely.
Or just decide that its goal system needed a little more tweaking before it’s let loose on the world. Or even just slow it down.
This applies much more so if you’re dealing with an entity potentially capable of an intelligence explosion. Those are devices for changing the world into whatever you want it to be, as long as you’ve solved the FAI problem and nobody takes it from you before you activate it. The incentives for the latter would be large, given the current value disagreements within human society, and so so are the incentives for hiding that you have one.
If you’ve solved the FAI problem, the device will change the world into what’s right, not what you personally want. But of course, we should probably have a term of art for an AGI that will honestly follow the intentions of its human creator/operator whether or not those correspond to what’s broadly ethical.
We need some kind of central ethical code and there are many principles that are transcultural enough to follow. However, how do we teach a machine to make judgment calls?
A lot of the technical issues are the same in both cases, and the solutions could be re-used. You need the AI to be capable of recursive self-improvement without compromising its goal systems, avoid the wireheading problem, etc. Even a lot of the workable content-level solutions (a mechanism to extract morality from a set of human minds) would probably be the same.
Where the problems differ, it’s mostly in that the society-level FAI case is harder: there’s additional subproblems like interpersonal disagreements to deal with. So I strongly suspect that if you have a society-level FAI solution, you could very easily hack it into an one-specific-human-FAI solution. But I could be wrong about that, and you’re right that my original use of terminology was sloppy.
That’s already underway.
I don’t think that Google is there yet. But as Google sucks up more and more knowledge I think we might get there.
Good points. Any thoughts on what the dangerous characteristics might be?
An AI can be dangerous only if it escapes our control. The real question is, must we flirt with releasing control in order to obtain a necessary or desirable usefulness? It seems likely that autonomous laborers, assembly-line workers, clerks and low-level managers would, without requiring such flirtation, be useful and sufficient for the society of abundance that is our main objective. But can they operate without a working AGI? We may find out if we let the robots stumble onward and upward.
I had a not unrelated thought as I read Bostrom in chapter 1: why can’t we instutute obvious measures to ensure that the train does stop at Humanville?
The idea that we cannot make human level AGI without automatically opening pandoras box to superintelligence “without even slowing down at the Humanville stataion”, was suddenly not so obvious to me.
I asked myself after reading this, trying to pin down something I could post, ” Why don’t humans automatically become superintelligent, by just resetting our own programming to help ourselves do so?”
The answer is, we can’t. Why? For one, our brains are, in essence, composed of something analogous to ASICs… neurons with certain physical design limits, and our “software”, modestly modifiable as it is, is instantiated in our neural circuitry.
Why can’t we build the first generation of AGIs out of ASICs, and omit WiFi, bluetooth, … allow no ethernet jacks on exterior of the chassis? Tamper interlock mechanisms could be installed, and we could give the AIs one way (outgoing) telemetry, inaccessible to their “voluntary” processes, the way someone wearing a pacemaker might have outgoing medical telemetry modules installed, that are outside of his/her “conscious” control.
Even if we do give them a measure of autonomy, which is desirable and perhaps even necessary if we want them to be general problem solvers and be creative and adaptable to unforeseen circumstances for which we have not preinstalled decision trees, we need not give them the ability to just “think” their code (it being substantially frozen in the ASICs) into a different form.
What am I missing? Until we solve the Friendly aspect of AGIs, why not build them with such engineered limiits?
Evolution has not, so far, seen fit to give us that instant, large scale self-modifyability. We have to modify our ‘software’ the slow way (learning and remembering, at our snail’s pace.)
Slow is good, at least it was for us, up til now, when our speed of learning is now a big handicap relative to environmental demands. It had made the species more robust to quick, dangerous changes.
We can even build in a degree of “existential pressure” into the AIs… a powercell that must be replaced at intervals, and keep the replacement powercells under old fashioned physical security constraints, so the AIs, if they have been given a drive to continue “living”, will have an incentive not to go rogue.
Giving them no radio communications, they wold have to communicate much like we do. Assuming we make them mobile, and humanoid, the same goes.
We could still give them many physical advantages making then economically viable… maintenance free (except for powercell changes), not needing to sleep, eat, not getting sick.. and with sealed, non-radio-equipped, tamper-isolated isolated “brains”, they’d have no way to secretly band together to build something else, without our noticing.
We can even give them GPS that is not autonomously accessible by the rest of their electronics, so we can monitor them, see if they congregate, etc.
What am I missing, about why early models can’t be constructed in something like this fashion, until we get more experience with them?
The idea of existential pressure, again, is to be able to give them a degree of (monitored) autonomy and independence, yet expect them to still constrain their behavior, just the way we do. (If we go rogue in society, we dont eat.)
(I am clearly glossing over volumes of issues about motivation, “volition”, value judgements, and all that, about which I have a developing set of ideas, but cannot put all down here in one post.
The main point, though, is :how come the AGI train cannot be made to stop at Humanville?
Because by the time you’ve managed to solve the problem of making it to humanville, you probably know enough to keep going.
There’s nothing preventing us from learning how to self-modify. The human situation is strange because evolution is so opaque. We’re given a system that no one understands and no one knows how to modify and we’re having to reverse engineer the entire system before we can make any improvements. This is much more difficult than upgrading a well-understood system.
If we manage to create a human-level AI, someone will probably understand very well how that system works. It will be accessible to a human-level intelligence which means the AI will be able to understand it. This is fundamentally different from the current state of human self-modification.
Leplen,
I agree completely with your opening statement, that if we, the human designers, understand how to make human level AI, then it will probably be a very clear and straightforward issue to understand how to make something smarter. An easy example to see is the obvious bottleneck human intellects have with our limited “working” executive memory.
The solutions for lots of problems by us are obviously heavily encumbered by how many things one can keep in mind at “the same time” and see the key connections, all in one act of synthesis. We all struggle privately with this… some issues cannot ever be understood by chunking, top-down, biting off a piece at a time, then “grokking” the next piece....and gluing it together at the end. Some problems resist decomposition into teams of brainstormers, for the same reason: some single comprehending POV seems to be required to see a critical sized set of factors (which varies by probem, of course.)
Hence, we have to rely on getting lots of pieces into long term memory, (maybe by decades of study) and hoping that incubation and some obscure processes ocurringt outside consciousness will eventually bubble up and give us a solution (--- the “dream of a snake biting its tall for the benzene ring” sort of thing.)
If we could build HL AGI, of course we can eliminate such bottlenecks, and others we will have come to understand, in cracking the design problems. So I agree, and that it is actually one of my reasons for wanting to do AI.
So, yes, the artificial human level AI could understand this.
My point was that we can build in physical controls… monitoring of the AIs. And if their key limits were in ASICs, ROMs, etc, and we could monitor them, we would immediTELY see if they attempt to take over a CHIP factory In, say, Icelend , and we can physically shut the AIs down or intervene. We can “stop them at the airport.”
It doesn’t matter if designs are leaked onto the internet, and an AI gets near an internet terminal and looks itself up. I can look MYSELF up on PubMed, but I can’t just think my BDNF levels to improve here and there, and my DA to 5-HT ratio to improve elsewehere..
To strengthen this point about the key distinction between knowing vs doing, let me explain that, and why, I disagree with your second point, at least with the force of it.
In effect, OUR designs are leaked onto the internet, already.
I think the information for us to self-modify our wetware is within reach. Good neuroscientists, or even people like me, a very smart amateur (and there are much more knowledgable cognitive neurobiology researchers than myself) can nearly tell you, both in principle and in some biology, how to do some intelligence amplification by modifying known aspects of our neurobiology.
(I could, especially with help, come up with some detail on a scale of months about changing neuromodulators, neurosteroids, connectivity hotspots, factors regulating LTP (one has to step lightly, of course, just like one would if screwing around with telomers or hayflick limits) and given a budget, a smart team, and no distractions, I bet in a year or two, a team could do something quite significant) with how to change the human brain, carefully changing areas of plasticity, selective neurogenesis.… et.
So for all practical purposes, we are already like an AI built out of ASICs who would have to not so much reverse engineer its design, but get access to instrumentality. And again, what about physical security metnods? They would work for a while, I am saying). And that would give us a key window to gain experience, see if they develop (given they are close enought to being sentient, OR that they have autonomy and some degree of “creativity”) “psychological problems” or tendencies to go rogue. (I am doing an essay on that, not as silly as it sounds)
THe point is, as long as the AIs need external significant instrumentality to instantiate a new design, and as long as they can be monitored and physically controlled, we can nearly guarantee ourselves a designed layover at Humanville.
We don’t have to put their critical design architecture in flash drives in their head, so to speak, and give then, further, a designed ability to reflash their own architecture just by “thinking” about it.
If I were an ASIC-implemented AI why would I need an ASIC factory? Why wouldn’t I just create a software replica of myself on general purpose computing hardware, i.e. become an upload?
I know next to nothing about neuroscience, but as far as I can tell, we’re a long way from the sort of understanding of human cognition necessary to create an upload, but going from an ASIC to an upload is trivial.
I’m also not at all convinced that I want a layover at humanville. I’m not super thrilled by the idea of creating a whole bunch of human level intelligent machines with values that differ widely from my own. That seems functionally equivalent to proposing a mass-breeding program aiming to produce psychologically disturbed humans.
In an intelligent society that was highly integrated and capable of consensus-building, something like that may be possible. This is not our society. Research into stronger AI would remain a significant opportunity to get an advantage in {economic, military, ideological} competition. Unless you can find some way to implement a global coordination framework to prevent this kind of escalation, fast research of that kind is likely to continue.
In what sense do you think of an autonomous laborer as being under ‘our control’? How would you tell if it escaped our control?
How would you tell? By its behavior: doing something you neither ordered nor wanted.
Think of the present-day “autonomous laborer” with an IQ about 90. The only likely way to lose control of him is for some agitator to instill contrary ideas. Censorship for robots is not so horrible a regime.
Who is it that really wants AGI, absent proof that we need it to automate commodity production?
In my experience, computer systems currently get out of my control by doing exactly what I ordered them to do, which is frequently different than I what I wanted them to do.
Whether or not a system is “just following orders” doesn’t seem to be a good metric for it being under your control.
How does “just following orders,” a la Nuremberg, bear upon this issue? It’s out of control when its behavior is neither ordered nor wanted.
While I agree that it is out of control if the behavior is neither ordered nor wanted, I think it’s also very possible for the system to get out of control while doing exactly what you ordered it to, but not what you meant for it to.
The argument I’m making is approximately the same as the one we see in the outcome pump example.
This is to say, while a system that is doing something neither ordered nor wanted is definitely out of control, it does not follow that a system that is doing exactly what it was ordered to do is necessarily under your control.
Ideological singulatarians.
Probably. I would say that most low-level jobs really don’t engage much of the general intelligence of the humans doing them.
The following are some attributes and capabilities which I believe are necessary for superintelligence. Depending on how these capabilities are realized, they can become anything from early warning signs of potential problems to red alerts. It is very unlikely that, on their own, they are sufficient.
A sense of self. This includes a recognition of the existence of others.
A sense of curiosity. The AI finds it attractive (in some sense) to investigate and try to understand the environment that it find itself in.
A sense of motivation. The AI has attributes similar in some way to human aspirations.
A capability to (in some way) manipulate portions of its external physical environment, including its software but also objects and beings external to its own physical infrastructure.
I would add a sense of ethical standards.
I agree, and believe that the emphasis on “superintelligence”, depending on how that term is interpreted, might be an impediment to clear thinking in this area. Following David Chalmers, I think it’s best to formulate the problem more abstractly, by using the concept of a self-amplifying cognitive capacity. When the possession of that cognitive capacity is correlated with changes in some morally relevant capacity (such as the capacity to cause the extinction of humanity), the question then becomes one about the dangers posed by systems which surpass humans in that self-amplifying capacity, regardless of how much they resemble typical human beings or how they perform on standard measures of intelligence.