There’s the whole issue that computers can’t act in the world at all unless they’re physically connected to hardware controllers that can interface with some physical system we actually care about being broken or misused. Usually, the workaround there is AI will be so persuasive that they can just get people with bodies to do the dirty work that requires being able to actually touch stuff in order to repurpose manufacturing plants or whatever it is we’re worried they might do.
In those cases it probably wouldn’t be very hard to get people to act in the world. Since there would be at least some people who want the AI to change the world. As evidenced by the fact that they just spend huge amounts of resources to create it in the first place.
AI threat tends to just use “intelligence” as a shorthand for good at literally anything.
That is pretty much the definition of intelligence. At least as far as expectations are concerned.(not that I’m arguing by definition, that is just what we mean when we talk about that.)
The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.
In those cases it probably wouldn’t be very hard to get people to act in the world. Since there would be at least some people who want the AI to change the world. As evidenced by the fact that they just spend huge amounts of resources to create it in the first place.
That is pretty much the definition of intelligence. At least as far as expectations are concerned.(not that I’m arguing by definition, that is just what we mean when we talk about that.)
The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.