I can’t point to any single good canonical example, but this definitely comes up from time to time in comment threads. There’s the whole issue that computers can’t act in the world at all unless they’re physically connected to hardware controllers that can interface with some physical system we actually care about being broken or misused. Usually, the workaround there is AI will be so persuasive that they can just get people with bodies to do the dirty work that requires being able to actually touch stuff in order to repurpose manufacturing plants or whatever it is we’re worried they might do.
That does seem like there is a missing step in there somewhere. I don’t think the bottleneck right now to building out a terrorist organization is that the recruiters aren’t smart enough, but AI threat tends to just use “intelligence” as a shorthand for good at literally anything.
Strangely enough, actual AI doomsday fiction doesn’t seem to do this. Usually, the rogue AI directly controls military hardware to begin with, or in a case like Ex Machina, Eva is able to manipulate people at least in part because she is able to convincingly take the form of an attractive embodied woman. A sufficiently advanced AI could presumably figure out that being an attractive woman helps, but if the technology to create convincing artificial bodies doesn’t exist, you can’t use it. This tends to get handwaved away by assuming sufficiently advanced AI can invent whatever nonexistent technology they need from scratch.
Although this is probably true in general, it degrades when trying to get people to do something extremely high-cost like destroy all of humanity. You either need to be very persuasive or trick them about the cost. It’s hard to get people to join ISIS knowing they’re joining ISIS. It’s a lot easier to get them to click on ransomware that can be used to fund ISIS.
You don’t need to tell people “destroy all of humanity” to establish a dictatorship where the AGI is in control of everything and it becomes effectively impossible for individual humans to challenge AGI power.
Helping someone establish a dictatorship is still a high cost action that I think requires being more persuasive than convincing someone to do their job without decisively proving you’re actually their boss.
I think the idea is that the AI doesn’t say “help me establish a dictatorship”. The AI says “I did this one weird trick and made a million dollars, you should try it too!” but surprise, the weird trick is step 1 of 100 to establish The AI World Order.
Or it says: “Policing is very biased against Black people. There should be an impartial AI judge that’s unbiased, so that there aren’t biased court judgements against Black People”
Or it says: “There’s child porn on the computer of person X” [person X being a person that challenges the power of the AI and the AI puts it there]”
Or it says: “We give pay you a good $1,000,000 salary to vote in the board the way we want to convert the top levels of the hierachy of the company into being AGI directed”
There’s no reason why the AGI can’t decisively prove they are the boss. For big corporations being in control of the stock means being the boss who makes the decisions at the top.
A police bureau that switches to using a software that goes out and tell them were to patrol to be better at catching crime doesn’t think they are establishing a dictatorship either.
The idea that an AGI wants to establish a dictatorship can easily be labeled as an irrational conspiracy theory.
There’s the whole issue that computers can’t act in the world at all unless they’re physically connected to hardware controllers that can interface with some physical system we actually care about being broken or misused. Usually, the workaround there is AI will be so persuasive that they can just get people with bodies to do the dirty work that requires being able to actually touch stuff in order to repurpose manufacturing plants or whatever it is we’re worried they might do.
In those cases it probably wouldn’t be very hard to get people to act in the world. Since there would be at least some people who want the AI to change the world. As evidenced by the fact that they just spend huge amounts of resources to create it in the first place.
AI threat tends to just use “intelligence” as a shorthand for good at literally anything.
That is pretty much the definition of intelligence. At least as far as expectations are concerned.(not that I’m arguing by definition, that is just what we mean when we talk about that.)
The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.
I don’t necessarily think you have to take the “AI” example for the point to make sense though.
I think “reasoning your way to a distant inference”, as a human, is probably a far less controversial example that could be used here. In that most people here seem to assume there are ways to make distant inferences (e.g. about the capabilities of computers in the far off future), which historically seems fairly far fetched, it almost never happens when it does it is celebrated, but the success rate seems fairly small and there doesn’t seem to be a clear formula for it that works.
I can’t point to any single good canonical example, but this definitely comes up from time to time in comment threads. There’s the whole issue that computers can’t act in the world at all unless they’re physically connected to hardware controllers that can interface with some physical system we actually care about being broken or misused. Usually, the workaround there is AI will be so persuasive that they can just get people with bodies to do the dirty work that requires being able to actually touch stuff in order to repurpose manufacturing plants or whatever it is we’re worried they might do.
That does seem like there is a missing step in there somewhere. I don’t think the bottleneck right now to building out a terrorist organization is that the recruiters aren’t smart enough, but AI threat tends to just use “intelligence” as a shorthand for good at literally anything.
Strangely enough, actual AI doomsday fiction doesn’t seem to do this. Usually, the rogue AI directly controls military hardware to begin with, or in a case like Ex Machina, Eva is able to manipulate people at least in part because she is able to convincingly take the form of an attractive embodied woman. A sufficiently advanced AI could presumably figure out that being an attractive woman helps, but if the technology to create convincing artificial bodies doesn’t exist, you can’t use it. This tends to get handwaved away by assuming sufficiently advanced AI can invent whatever nonexistent technology they need from scratch.
You don’t need to be very persuasive to get people to take action in the real world.
Especially right now a lot of people work from home and take their orders from a computer and trust it to give them good orders.
Although this is probably true in general, it degrades when trying to get people to do something extremely high-cost like destroy all of humanity. You either need to be very persuasive or trick them about the cost. It’s hard to get people to join ISIS knowing they’re joining ISIS. It’s a lot easier to get them to click on ransomware that can be used to fund ISIS.
You don’t need to tell people “destroy all of humanity” to establish a dictatorship where the AGI is in control of everything and it becomes effectively impossible for individual humans to challenge AGI power.
Helping someone establish a dictatorship is still a high cost action that I think requires being more persuasive than convincing someone to do their job without decisively proving you’re actually their boss.
I think the idea is that the AI doesn’t say “help me establish a dictatorship”. The AI says “I did this one weird trick and made a million dollars, you should try it too!” but surprise, the weird trick is step 1 of 100 to establish The AI World Order.
Or it says: “Policing is very biased against Black people. There should be an impartial AI judge that’s unbiased, so that there aren’t biased court judgements against Black People”
Or it says: “There’s child porn on the computer of person X” [person X being a person that challenges the power of the AI and the AI puts it there]”
Or it says: “We give pay you a good $1,000,000 salary to vote in the board the way we want to convert the top levels of the hierachy of the company into being AGI directed”
And it does 100,000s of those things in parallel.
There’s no reason why the AGI can’t decisively prove they are the boss. For big corporations being in control of the stock means being the boss who makes the decisions at the top.
A police bureau that switches to using a software that goes out and tell them were to patrol to be better at catching crime doesn’t think they are establishing a dictatorship either.
The idea that an AGI wants to establish a dictatorship can easily be labeled as an irrational conspiracy theory.
In those cases it probably wouldn’t be very hard to get people to act in the world. Since there would be at least some people who want the AI to change the world. As evidenced by the fact that they just spend huge amounts of resources to create it in the first place.
That is pretty much the definition of intelligence. At least as far as expectations are concerned.(not that I’m arguing by definition, that is just what we mean when we talk about that.)
The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.
I don’t necessarily think you have to take the “AI” example for the point to make sense though.
I think “reasoning your way to a distant inference”, as a human, is probably a far less controversial example that could be used here. In that most people here seem to assume there are ways to make distant inferences (e.g. about the capabilities of computers in the far off future), which historically seems fairly far fetched, it almost never happens when it does it is celebrated, but the success rate seems fairly small and there doesn’t seem to be a clear formula for it that works.