The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.
The distinction in this specific case here is between intelligence and persuasiveness. To the extent that some elements of persuasiveness are inherently embodied, as in people are more likely to trust you if you’re also a person, that is at best orthogonal to intelligence.
More generally, “effectiveness” as some general purpose quality of agents that can do things is limited by the ability to acquire and process information, but also by the ability to act on it. You may know that being tall makes you more likely to be elected to office, but if you can’t make yourself any taller, you can’t use the information to make your campaign more likely to succeed.
As a more fantastical but maybe more relevant example, people often mention something like turning the moon into comptronium. Part of doing that is knowing how to do it. But we already know how to do it. We understand at the level of fusion and fission how to transmute elements into different elements, and we understand, given some elements that act as semiconductors, how to produce general-purpose computational processors. The actual reason we can’t do it, aside from not wanting to disrupt the earth’s orbit and potentially end human civilization, is (1) there is inherent propagation delay in moving material from wherever it is created to wherever it needs to be used and this delay is much greater when the distances to move are greater than planet-scale, (2) machines that can actually transmute rocks to silicon don’t presently exist and there is non-zero manufacturing delay in creating them, and (3) we have no means of harnessing sufficient energy to actually transmute matter at the necessary scale.
Can gaining more information solve these problems? Maybe. There might exist unknown physics that enable easier or faster methods than we presently know of, but there is non-zero propagation delay in creation of new knowledge of physics as well. You have to conduct experiments. At high-energy, sub-particle scale, these have become extremely expensive and time consuming. AI threat analysis tends to get around this one by proposing they can just simulate physics to such perfect fidelity that experimentation is no longer necessary, but this seem question-begging because you need to already know rules of physics that haven’t been discovered yet to be able to do this.
While presumably a collection of brains better than human brains can figure out a way to make this happen faster, maybe even decades rather than centuries faster, “foom” type analyses that claim the ability to recursively rewrite one’s own source code better than the original coder means it will happen in days or even hours come across more as mysticism than real risk analysis.