How do you address the “capabilities overhang” argument?
If you keep capabilities research at the same pace, or speed it up, capable AGIs will be built at an earlier date.
Iff they are dangerous/uncontrollable, there will be direct evidence of this. Perhaps science fiction scenarios where the AGIs commit murders before getting caught and shut down—something that clearly demonstrates the danger. (I hope no one is ever killed by rogue AI, but in the past, most of the rules for industrial safety were written using the blood of killed workers as the ink. I’m not sure there are OSHA rules that aren’t directly from someone dying.)
Right now you are telling people about a wolf that doesn’t even exist yet.
If the AGIs are built in this earlier era, it means there is less compute available. Right now, it might require an unfeasible amount of compute to train and host an AI—easily tens, maybe hundreds of millions of dollars in hardware just to host 1 instance. This makes “escapes” difficult if there is only a few sets of the necessary hardware on the planet.
There is also less robotics available—with the recent tech company funding cuts to robotics efforts, even less robotics than that.
By working on capabilities, “you” could do two things:
Create a trail, a path to a working system that is probably safe. This creates evolutionary lock in—people are going to build on the existing code and explore around it, they won’t start over. Example: Linux Torvalds banned the use of C++ in the Linux kernel, a decision that sticks to the present.
Accelerate slightly the date to the first AGI, which has the advantages mentioned above.
By obstructing capabilities, “you”:
1. Leave it to the other people who create the AGI instead to decide the approach
2. Delay slightly the date to the first AGI, where more compute and robotics will be available for bad outcomes.
By “you” I mean every user of lesswrong who is in a position knowledge/career wise combined, so a few thousand people.
Are broader outcomes even possible? If EY really did spur the creation of OpenAI, he personally subtracted several years from the time until first AGI...
All good points. I suspect the best path into the future looks like: everyone’s optimistic and then a ‘survivable disaster’ happens with AI. Ideally, we’d want all the panic to happen in one big shot—it’s the best way to motivate real change.
Yeah. Around here, there’s Zvi, EY himself, and many others who are essentially arguing that capabilities research is itself evil.
The problem with that take is
(1) alignment research will probably never achieve success without capabilities to test their theories on. Why wasn’t alignment worked on since 1955, when AI research began? Because there was no credible belief to think it was a threat.
(2) the world we live in has a number of terribad things happening to people by the millions, with that nasty virus that went around being only a recent particularly bad example, and we have a bunch of problems where we humans are probably not capable of solving them. Too many independent variables, too stochastic, correct theories are probably too complex for a human to keep the entire theory “in their head” at once. Examples: medical problems, how the economy works.
How do you address the “capabilities overhang” argument?
If you keep capabilities research at the same pace, or speed it up, capable AGIs will be built at an earlier date.
Iff they are dangerous/uncontrollable, there will be direct evidence of this. Perhaps science fiction scenarios where the AGIs commit murders before getting caught and shut down—something that clearly demonstrates the danger. (I hope no one is ever killed by rogue AI, but in the past, most of the rules for industrial safety were written using the blood of killed workers as the ink. I’m not sure there are OSHA rules that aren’t directly from someone dying.)
Right now you are telling people about a wolf that doesn’t even exist yet.
If the AGIs are built in this earlier era, it means there is less compute available. Right now, it might require an unfeasible amount of compute to train and host an AI—easily tens, maybe hundreds of millions of dollars in hardware just to host 1 instance. This makes “escapes” difficult if there is only a few sets of the necessary hardware on the planet.
There is also less robotics available—with the recent tech company funding cuts to robotics efforts, even less robotics than that.
>How do you address the “capabilities overhang” argument?
I don’t, I’m just not willing to bet the Earth that a world-ender will be proceed by mere mass murders. It might be, but let’s not risk it.
As I say, if there’s a 1% chance of doom, then we should treat it as nearly 100%.
By working on capabilities, “you” could do two things:
Create a trail, a path to a working system that is probably safe. This creates evolutionary lock in—people are going to build on the existing code and explore around it, they won’t start over. Example: Linux Torvalds banned the use of C++ in the Linux kernel, a decision that sticks to the present.
Accelerate slightly the date to the first AGI, which has the advantages mentioned above.
By obstructing capabilities, “you”:
1. Leave it to the other people who create the AGI instead to decide the approach
2. Delay slightly the date to the first AGI, where more compute and robotics will be available for bad outcomes.
By “you” I mean every user of lesswrong who is in a position knowledge/career wise combined, so a few thousand people.
Are broader outcomes even possible? If EY really did spur the creation of OpenAI, he personally subtracted several years from the time until first AGI...
All good points. I suspect the best path into the future looks like: everyone’s optimistic and then a ‘survivable disaster’ happens with AI. Ideally, we’d want all the panic to happen in one big shot—it’s the best way to motivate real change.
Yeah. Around here, there’s Zvi, EY himself, and many others who are essentially arguing that capabilities research is itself evil.
The problem with that take is
(1) alignment research will probably never achieve success without capabilities to test their theories on. Why wasn’t alignment worked on since 1955, when AI research began? Because there was no credible belief to think it was a threat.
(2) the world we live in has a number of terribad things happening to people by the millions, with that nasty virus that went around being only a recent particularly bad example, and we have a bunch of problems where we humans are probably not capable of solving them. Too many independent variables, too stochastic, correct theories are probably too complex for a human to keep the entire theory “in their head” at once. Examples: medical problems, how the economy works.
I wouldn’t call it evil, but I would say that it’s playing with fire.