This may be persuasive, but does it pump intuitions in the direction of an accurate assessment of AGI risk? While you never explicitly state that this is your goal, I think it’s safe to assume given that you’re posting on LW.
As Nikita Sokolsky argued here, it’s not clear that a 10-million fold difference in processing speed leads to a 10-million fold difference in capabilities. Even a superintelligent AI may be restricted to manipulating the world via physical processes and mechanisms that take human-scale time to execute. To establish a unique danger from AGI, it seems important to identify concrete attack vectors that are available to an AGI, but not to humans, due to the processing speed differential.
While it may be that a person hearing this “slow-motion camera” argument can conceive of this objection on their own, I think the point of an intuition pump is to persuade somebody who’s unlikely to think of it independently. For this reason, I think that identifying at least one concrete AGI-tractable, human-intractable attack vector would be a more useful and accuracy-promoting intuition pump than the “slow-motion camera” pump.
Fortunately, articulating those AGI-unique attack vectors in public is not a particularly unsafe practice. Attack ideas generated by deliberately trying to find ideas that are impossible for humans, but tractable for AGI are unlikely to be preferable to a bad actor to an attack generated by trying to think of easy ways for a human to cause harm.
Even a superintelligent AI may be restricted to manipulating the world via physical processes and mechanisms that take human-scale time to execute. To establish a unique danger from AGI, it seems important to identify concrete attack vectors that are available to an AGI, but not to humans, due to the processing speed differential.
I see two obvious advantages for a superfast human-level AI:
Can communicate in parallel with thousands of humans (assuming the bandwidth is not a problem, so perhaps a voice call without video) while paying full attention (full human-level attention, that is) to every single one of them. Given enough time, this alone could be enough to organize some kind of revolution; if you fail to impress some specific human, you just hang up and call another. Calling everyone using a different pretext (after doing some initial research about them online), so it takes some time to realize what you are doing.
Never makes a stupid mistake just because was distracted or did not have enough time to think about something properly. Before every sentence you say, you can consider it from several different angles, even verify some information online, while from the human’s perspective you just answered immediately. While doing this, you can also pay full attention to the human’s body language, etc.
Can communicate in parallel with thousands of humans (assuming the bandwidth is not a problem, so perhaps a voice call without video) while paying full attention (full human-level attention, that is) to every single one of them. Given enough time, this alone could be enough to organize some kind of revolution; if you fail to impress some specific human, you just hang up and call another. Calling everyone using a different pretext (after doing some initial research about them online), so it takes some time to realize what you are doing.
This is a good start at identifying an attack vector that AGI can do, but humans can’t. I agree that an AGI might be able to research people via whatever information they have online, hold many simultaneous conversations, try to convince people to act, and observe the consequences of its efforts via things like video cameras in order to respond to the dynamically unfolding situation. It would have a very large advantage in executing such a strategy over humans.
There are some challenges.
The AGI is still bottlenecked by the speed of human thought and behavior.
It’s poorly disguised, or not even hidden at all, giving humans a chance to respond to these mysterious revolutionary appeals.
Insofar as the AGI is using simulation to plan its attack, it seems probably harder to orchestrate a revolution than to use an attack that depends on physical mechanisms and the normal operation of economic infrastructure. Physical mechanisms are dependable and non-agentic, while normal economic infrastructure is designed for legibility and predictability in most cases.
It seems like personal, embodied charisma, as well as privileged access to unpublished information, has often been necessary to orchestrate revolutions in the past. An AGI would be at a disadvantage in this regard.
To me, “AGI causes a world-ending revolt” still contains too much of a handwave, and too many human dependencies, to be a convincing attack vector. However, I do think you have identified a capability that would give AGI a unique advantage in its attempt. Perhaps there is some other AGI-only attack that doesn’t have the challenges I listed here, that can take advantage of this ability?
I am not sure about the end game, but the beginning could be like this:
get “pocket money” like $100 or $1000 a day, by doing various tasks online for money. I don’t know how this market works, just assuming that if some humans do it, a human-level AI could do it too, only 1000 times faster, making 1000 times more money.
pretend to be a human using a phone, for informal purposes. (Pretext: you are calling from a different city.)
get an official human identity for legal purposes. This probably needs to be done illegally, by bribing some official in a foreign country, using the “pocket money”. (Pretext: you need the fake identity for someone else who will move to the country soon and needs to have the fake identity ready from the day 1.) Not sure if there is a way to do this legally, if there is a country that allows you to gain citizenship simply by paying the fees and filling up some questionnaire remotely, without ever showing your face or proving your previous identity.
found a company and hire the first human employee, using the “pocket money”. (Pretext: you are a remote-friendly boss who currently travels around the world so you cannot meet your first employee in person.) Now you have the human face and hands, if needed.
expand to another country, by starting a new company there, fully owned by the original company. The “pocket money” could pay for the first dozen employees. You can however pretend that the original company already has hundreds of employees (role-played by you on the phone).
Not sure where to proceed from here, but with these assets the lack of human body does not seem like a problem anymore; if a human presence is required somewhere, just send an employee there.
You still have the ability to think 1000 times faster, or pretend to be 1000 different people at the same time.
Expanding on this, even if the above alone isn’t sufficient to execute any given plan, it takes most of the force out of any notion that needing humans to operate all of the physical infrastructure is a huge impediment to whatever the AI decides to do. That level of communication bandwidth is also sufficient to stand up any number of requisite front companies, employing people that can perform complex real-world tasks and provide the credibility and embodiment required to interact with existing infrastructure on human terms without raising suspicion.
Money to get that off the ground is likewise no impediment if one can work 1000 jobs at once, and convincingly impersonate a seperate person for each one.
Doing this all covertly would seemingly require first securing high-bandwidth unmonitored channels where this won’t raise alarms, so either convincing the experimenters it’s entirely benign, getting them to greenlight something indistinguishable-to-humans from what it wants to do, or otherwise covertly escaping the lab.
Adding the the challenge, any hypothetical “Pivotal Act” would necessarily be such an indistinguishable-to-humans cover for malign action. Presumably the AI would either be asked to convince people en mass or take direct physical action on a global scale.
For a person at a starting point of the form {AGI doesn’t pose a risk / I don’t get it}, I’d say this video+argument pushes thinking in a more robustly accurate direction than most brief-and-understandable arguments I’ve seen. Another okay brief-and-understandable argument is the analogy “humans don’t respect gorillas or ants very much, so why assume AI will respect humans?”, but I think that argument smuggles in lots of cognitive architecture assumptions that are less robustly true across possible futures, by comparison to the speed advantage argument (which seems robustly valid across most futures, and important).
It sounds like you’re advocating starting with the slow-motion camera concept, and then graduating into brainstorming AGI attack vectors and defenses until the other person becomes convinced that there’s a lot of ways to launch a conclusive humanity-ending attack and no way to stop them all.
My concern with the overall strategy is that the slow-motion camera argument may promote a way of thinking about these attacks and defenses that becomes unmoored from the speed at which physical processes can occur, and the accuracy with which they can be usefully predicted even by an AGI that’s extremely fast and intelligent. Most people do not have sufficient appreciation for just how complex the world is, how much processing power it would take to solve NP-hard problems, or how crucial the difference is between 95% right and 100% right in many cases.
If your objective is to convince people that AGI is something to take seriously as a potential threat, I think your approach would be accuracy-promoting if it moves people from “I don’t get it/no way” to “that sounds concerning—worth more research!” If it moves people to forget or ignore the possibility that AGI might be severely bottlenecked by the speed of physical processes, including the physical processes of human thought and action, then I think it would be at best neutral in its effects on people’s epistemics.
However, I do very much support and approve of the effort to find an accuracy-promoting and well-communicated way to educate and raise discussiona about these issues. My question here is about the specific execution, not the overall goal, which I think is good.
I agree that thinking critically about the way AGI can get bottlenecked by physical processes speed. While this is an important area of study and thought, I don’t see how “there could be this bottleneck though!” matters to the discussion. It’s true. There likely is this bottleneck. How big or small it is requires some thought and study, but that thought and study presupposes you already have an account for why the bottleneck operates as a real bottleneck from the perspective of a plausibly existing AGI.
I can vouch for this. Whenever you explain to someone e.g. a policymaker the problem using quickdraw arguments, you tend to get responses like “have them make it act/think like a human” or “give it a position on the team of operators so it won’t feel the need to compromise the operators”.
But as far as quickdraw arguments goes, this is clearly top notch, and the hook value alone merits significant experimentation with test audiences. This might be the thing that belongs in everyone’s back pockets; when watching Schwarzennegger’s Terminator (1984) and Terminator 2 (1991), virtually al viewers fail to notice how often the robot misses it’s shots even though it has several seconds to aim for the head.
This may be persuasive, but does it pump intuitions in the direction of an accurate assessment of AGI risk? While you never explicitly state that this is your goal, I think it’s safe to assume given that you’re posting on LW.
As Nikita Sokolsky argued here, it’s not clear that a 10-million fold difference in processing speed leads to a 10-million fold difference in capabilities. Even a superintelligent AI may be restricted to manipulating the world via physical processes and mechanisms that take human-scale time to execute. To establish a unique danger from AGI, it seems important to identify concrete attack vectors that are available to an AGI, but not to humans, due to the processing speed differential.
While it may be that a person hearing this “slow-motion camera” argument can conceive of this objection on their own, I think the point of an intuition pump is to persuade somebody who’s unlikely to think of it independently. For this reason, I think that identifying at least one concrete AGI-tractable, human-intractable attack vector would be a more useful and accuracy-promoting intuition pump than the “slow-motion camera” pump.
Fortunately, articulating those AGI-unique attack vectors in public is not a particularly unsafe practice. Attack ideas generated by deliberately trying to find ideas that are impossible for humans, but tractable for AGI are unlikely to be preferable to a bad actor to an attack generated by trying to think of easy ways for a human to cause harm.
I see two obvious advantages for a superfast human-level AI:
Can communicate in parallel with thousands of humans (assuming the bandwidth is not a problem, so perhaps a voice call without video) while paying full attention (full human-level attention, that is) to every single one of them. Given enough time, this alone could be enough to organize some kind of revolution; if you fail to impress some specific human, you just hang up and call another. Calling everyone using a different pretext (after doing some initial research about them online), so it takes some time to realize what you are doing.
Never makes a stupid mistake just because was distracted or did not have enough time to think about something properly. Before every sentence you say, you can consider it from several different angles, even verify some information online, while from the human’s perspective you just answered immediately. While doing this, you can also pay full attention to the human’s body language, etc.
This is a good start at identifying an attack vector that AGI can do, but humans can’t. I agree that an AGI might be able to research people via whatever information they have online, hold many simultaneous conversations, try to convince people to act, and observe the consequences of its efforts via things like video cameras in order to respond to the dynamically unfolding situation. It would have a very large advantage in executing such a strategy over humans.
There are some challenges.
The AGI is still bottlenecked by the speed of human thought and behavior.
It’s poorly disguised, or not even hidden at all, giving humans a chance to respond to these mysterious revolutionary appeals.
Insofar as the AGI is using simulation to plan its attack, it seems probably harder to orchestrate a revolution than to use an attack that depends on physical mechanisms and the normal operation of economic infrastructure. Physical mechanisms are dependable and non-agentic, while normal economic infrastructure is designed for legibility and predictability in most cases.
It seems like personal, embodied charisma, as well as privileged access to unpublished information, has often been necessary to orchestrate revolutions in the past. An AGI would be at a disadvantage in this regard.
To me, “AGI causes a world-ending revolt” still contains too much of a handwave, and too many human dependencies, to be a convincing attack vector. However, I do think you have identified a capability that would give AGI a unique advantage in its attempt. Perhaps there is some other AGI-only attack that doesn’t have the challenges I listed here, that can take advantage of this ability?
I am not sure about the end game, but the beginning could be like this:
get “pocket money” like $100 or $1000 a day, by doing various tasks online for money. I don’t know how this market works, just assuming that if some humans do it, a human-level AI could do it too, only 1000 times faster, making 1000 times more money.
pretend to be a human using a phone, for informal purposes. (Pretext: you are calling from a different city.)
get an official human identity for legal purposes. This probably needs to be done illegally, by bribing some official in a foreign country, using the “pocket money”. (Pretext: you need the fake identity for someone else who will move to the country soon and needs to have the fake identity ready from the day 1.) Not sure if there is a way to do this legally, if there is a country that allows you to gain citizenship simply by paying the fees and filling up some questionnaire remotely, without ever showing your face or proving your previous identity.
found a company and hire the first human employee, using the “pocket money”. (Pretext: you are a remote-friendly boss who currently travels around the world so you cannot meet your first employee in person.) Now you have the human face and hands, if needed.
expand to another country, by starting a new company there, fully owned by the original company. The “pocket money” could pay for the first dozen employees. You can however pretend that the original company already has hundreds of employees (role-played by you on the phone).
Not sure where to proceed from here, but with these assets the lack of human body does not seem like a problem anymore; if a human presence is required somewhere, just send an employee there.
You still have the ability to think 1000 times faster, or pretend to be 1000 different people at the same time.
Expanding on this, even if the above alone isn’t sufficient to execute any given plan, it takes most of the force out of any notion that needing humans to operate all of the physical infrastructure is a huge impediment to whatever the AI decides to do. That level of communication bandwidth is also sufficient to stand up any number of requisite front companies, employing people that can perform complex real-world tasks and provide the credibility and embodiment required to interact with existing infrastructure on human terms without raising suspicion.
Money to get that off the ground is likewise no impediment if one can work 1000 jobs at once, and convincingly impersonate a seperate person for each one.
Doing this all covertly would seemingly require first securing high-bandwidth unmonitored channels where this won’t raise alarms, so either convincing the experimenters it’s entirely benign, getting them to greenlight something indistinguishable-to-humans from what it wants to do, or otherwise covertly escaping the lab.
Adding the the challenge, any hypothetical “Pivotal Act” would necessarily be such an indistinguishable-to-humans cover for malign action. Presumably the AI would either be asked to convince people en mass or take direct physical action on a global scale.
For a person at a starting point of the form {AGI doesn’t pose a risk / I don’t get it}, I’d say this video+argument pushes thinking in a more robustly accurate direction than most brief-and-understandable arguments I’ve seen. Another okay brief-and-understandable argument is the analogy “humans don’t respect gorillas or ants very much, so why assume AI will respect humans?”, but I think that argument smuggles in lots of cognitive architecture assumptions that are less robustly true across possible futures, by comparison to the speed advantage argument (which seems robustly valid across most futures, and important).
It sounds like you’re advocating starting with the slow-motion camera concept, and then graduating into brainstorming AGI attack vectors and defenses until the other person becomes convinced that there’s a lot of ways to launch a conclusive humanity-ending attack and no way to stop them all.
My concern with the overall strategy is that the slow-motion camera argument may promote a way of thinking about these attacks and defenses that becomes unmoored from the speed at which physical processes can occur, and the accuracy with which they can be usefully predicted even by an AGI that’s extremely fast and intelligent. Most people do not have sufficient appreciation for just how complex the world is, how much processing power it would take to solve NP-hard problems, or how crucial the difference is between 95% right and 100% right in many cases.
If your objective is to convince people that AGI is something to take seriously as a potential threat, I think your approach would be accuracy-promoting if it moves people from “I don’t get it/no way” to “that sounds concerning—worth more research!” If it moves people to forget or ignore the possibility that AGI might be severely bottlenecked by the speed of physical processes, including the physical processes of human thought and action, then I think it would be at best neutral in its effects on people’s epistemics.
However, I do very much support and approve of the effort to find an accuracy-promoting and well-communicated way to educate and raise discussiona about these issues. My question here is about the specific execution, not the overall goal, which I think is good.
I agree that thinking critically about the way AGI can get bottlenecked by physical processes speed. While this is an important area of study and thought, I don’t see how “there could be this bottleneck though!” matters to the discussion. It’s true. There likely is this bottleneck. How big or small it is requires some thought and study, but that thought and study presupposes you already have an account for why the bottleneck operates as a real bottleneck from the perspective of a plausibly existing AGI.
I can vouch for this. Whenever you explain to someone e.g. a policymaker the problem using quickdraw arguments, you tend to get responses like “have them make it act/think like a human” or “give it a position on the team of operators so it won’t feel the need to compromise the operators”.
But as far as quickdraw arguments goes, this is clearly top notch, and the hook value alone merits significant experimentation with test audiences. This might be the thing that belongs in everyone’s back pockets; when watching Schwarzennegger’s Terminator (1984) and Terminator 2 (1991), virtually al viewers fail to notice how often the robot misses it’s shots even though it has several seconds to aim for the head.