By working on capabilities, “you” could do two things:
Create a trail, a path to a working system that is probably safe. This creates evolutionary lock in—people are going to build on the existing code and explore around it, they won’t start over. Example: Linux Torvalds banned the use of C++ in the Linux kernel, a decision that sticks to the present.
Accelerate slightly the date to the first AGI, which has the advantages mentioned above.
By obstructing capabilities, “you”:
1. Leave it to the other people who create the AGI instead to decide the approach
2. Delay slightly the date to the first AGI, where more compute and robotics will be available for bad outcomes.
By “you” I mean every user of lesswrong who is in a position knowledge/career wise combined, so a few thousand people.
Are broader outcomes even possible? If EY really did spur the creation of OpenAI, he personally subtracted several years from the time until first AGI...
All good points. I suspect the best path into the future looks like: everyone’s optimistic and then a ‘survivable disaster’ happens with AI. Ideally, we’d want all the panic to happen in one big shot—it’s the best way to motivate real change.
Yeah. Around here, there’s Zvi, EY himself, and many others who are essentially arguing that capabilities research is itself evil.
The problem with that take is
(1) alignment research will probably never achieve success without capabilities to test their theories on. Why wasn’t alignment worked on since 1955, when AI research began? Because there was no credible belief to think it was a threat.
(2) the world we live in has a number of terribad things happening to people by the millions, with that nasty virus that went around being only a recent particularly bad example, and we have a bunch of problems where we humans are probably not capable of solving them. Too many independent variables, too stochastic, correct theories are probably too complex for a human to keep the entire theory “in their head” at once. Examples: medical problems, how the economy works.
>How do you address the “capabilities overhang” argument?
I don’t, I’m just not willing to bet the Earth that a world-ender will be proceed by mere mass murders. It might be, but let’s not risk it.
As I say, if there’s a 1% chance of doom, then we should treat it as nearly 100%.
By working on capabilities, “you” could do two things:
Create a trail, a path to a working system that is probably safe. This creates evolutionary lock in—people are going to build on the existing code and explore around it, they won’t start over. Example: Linux Torvalds banned the use of C++ in the Linux kernel, a decision that sticks to the present.
Accelerate slightly the date to the first AGI, which has the advantages mentioned above.
By obstructing capabilities, “you”:
1. Leave it to the other people who create the AGI instead to decide the approach
2. Delay slightly the date to the first AGI, where more compute and robotics will be available for bad outcomes.
By “you” I mean every user of lesswrong who is in a position knowledge/career wise combined, so a few thousand people.
Are broader outcomes even possible? If EY really did spur the creation of OpenAI, he personally subtracted several years from the time until first AGI...
All good points. I suspect the best path into the future looks like: everyone’s optimistic and then a ‘survivable disaster’ happens with AI. Ideally, we’d want all the panic to happen in one big shot—it’s the best way to motivate real change.
Yeah. Around here, there’s Zvi, EY himself, and many others who are essentially arguing that capabilities research is itself evil.
The problem with that take is
(1) alignment research will probably never achieve success without capabilities to test their theories on. Why wasn’t alignment worked on since 1955, when AI research began? Because there was no credible belief to think it was a threat.
(2) the world we live in has a number of terribad things happening to people by the millions, with that nasty virus that went around being only a recent particularly bad example, and we have a bunch of problems where we humans are probably not capable of solving them. Too many independent variables, too stochastic, correct theories are probably too complex for a human to keep the entire theory “in their head” at once. Examples: medical problems, how the economy works.
I wouldn’t call it evil, but I would say that it’s playing with fire.