I strongly agree with Section 1. Even if we would have aligned superintelligence, how are we going to make sure no one runs an unaligned superintelligence? A pivotal act? If so, which one? Or does defense trump offense? If so, why? Or are we still going to regulate heavily? If so, wouldn’t the same regulation be able to stop superintelligence altogether?
Would love to see an argument landing at 1% p(doom) or lower, even if alignment would be easy.
The argument is probably something like superintelligence allows robust monitoring of other people’s attempts at building superintelligences, and likely could resolve whatever prisoners dilemmas people have to build them. We don’t need an authoritarian regime.
I expect an ASI can denoise easily available data enough to confidently figure out who is trying to build a superintelligence, and either stop them through soft power/argumentation or implore them to make it aligned.
I think most problems are probably easy to solve with a truly aligned superintelligence. It may come up with good solutions we haven’t even thought of.
I think you may be right that this is what people think of. It seems pretty incompatible with any open source-ish vision of AGI. But what I’m most surprised at, is that people call supervision by humans dystopian/authoritarian, but the same supervision by an ASI (apparently able to see all your data, stop anyone from doing anything, subtly manipulate anyone, etc etc) a utopia. What am I missing here?
Personally, by the way, I imagine a regulation regime to look like regulating a few choke points in the hardware supply chain, plus potentially limits to the hardware or data a person can possess. This doesn’t require an authoritarian regime at all, it’s just regular regulation as we have in many domains already.
In any case, the point was, is something like this going to lead to <=1% xrisk? I think it doesn’t, and definitely not mixed with a democratic/open source AGI vision.
If this is your definition of a dystopia we already live in a dystopia. You can’t make nuclear bombs without being picked up by the FBI/CIA and you’ll probably be arrested in the process. Making something illegal doesn’t define an authoritarian regime. Governments already try to stop international players from building nukes. It just lacks teeth because you can ultimately live with sanctions.
The other problem is it’s way too easy to avoid surveillance or defect in a human regime. For example, you can have a decentralized training network, claim you are training good AIIt’s also unusually easy to regulate AI training. Right now GPUs are easy to control because only Nvidia can make them. This won’t always be true. It’s also much easier to hide GPU production than making nukes because we need GPUs and CPUs for a ton of other useful things.
Theoretically an ASI could probably extrapolate attempts to use compute correctly from your internet signals. Further, if you have the benefits you want from an ASI, you have much less reason to build a 2nd one that’s possibly unaligned. “The digital god says you can’t build it” probably sounds a lot more compelling than “Joe Biden says you can’t build it”.
This is a good question to ask, and my general answer is to this question is a combination of 2 answers, defense doesn’t trump offense in all domains, but they are much more balanced than LWers think, with the exception of bio, and that domain mostly doesn’t have existential risky products. Regulation is necessary for superintelligence, but I don’t think this is anywhere near true:
If so, wouldn’t the same regulation be able to stop superintelligence altogether?
No, primarily because misuse/structural issues demand very different responses, and a lot of the policy making pretty much relies on the assumption that AI is hard to control.
Much more generally, I wish people would make distinctions between existential risk caused by lack of control vs existential risk caused by misuse vs mass death caused by structural forces, since each of these relies on very differing causes, and that matters since the policy conclusions are very different, and sometimes even opposed to each other.
Open Sourcing is a case in point. It’s negative in the cases where misuse and loss control is the dominant risk factor, but turns into a positive if we instead assume structural forces are at work, like in dr_s’s story here:
I strongly agree with Section 1. Even if we would have aligned superintelligence, how are we going to make sure no one runs an unaligned superintelligence? A pivotal act? If so, which one? Or does defense trump offense? If so, why? Or are we still going to regulate heavily? If so, wouldn’t the same regulation be able to stop superintelligence altogether?
Would love to see an argument landing at 1% p(doom) or lower, even if alignment would be easy.
The argument is probably something like superintelligence allows robust monitoring of other people’s attempts at building superintelligences, and likely could resolve whatever prisoners dilemmas people have to build them. We don’t need an authoritarian regime.
I expect an ASI can denoise easily available data enough to confidently figure out who is trying to build a superintelligence, and either stop them through soft power/argumentation or implore them to make it aligned.
I think most problems are probably easy to solve with a truly aligned superintelligence. It may come up with good solutions we haven’t even thought of.
I think you may be right that this is what people think of. It seems pretty incompatible with any open source-ish vision of AGI. But what I’m most surprised at, is that people call supervision by humans dystopian/authoritarian, but the same supervision by an ASI (apparently able to see all your data, stop anyone from doing anything, subtly manipulate anyone, etc etc) a utopia. What am I missing here?
Personally, by the way, I imagine a regulation regime to look like regulating a few choke points in the hardware supply chain, plus potentially limits to the hardware or data a person can possess. This doesn’t require an authoritarian regime at all, it’s just regular regulation as we have in many domains already.
In any case, the point was, is something like this going to lead to <=1% xrisk? I think it doesn’t, and definitely not mixed with a democratic/open source AGI vision.
If this is your definition of a dystopia we already live in a dystopia. You can’t make nuclear bombs without being picked up by the FBI/CIA and you’ll probably be arrested in the process. Making something illegal doesn’t define an authoritarian regime. Governments already try to stop international players from building nukes. It just lacks teeth because you can ultimately live with sanctions.
The other problem is it’s way too easy to avoid surveillance or defect in a human regime. For example, you can have a decentralized training network, claim you are training good AIIt’s also unusually easy to regulate AI training. Right now GPUs are easy to control because only Nvidia can make them. This won’t always be true. It’s also much easier to hide GPU production than making nukes because we need GPUs and CPUs for a ton of other useful things.
Theoretically an ASI could probably extrapolate attempts to use compute correctly from your internet signals. Further, if you have the benefits you want from an ASI, you have much less reason to build a 2nd one that’s possibly unaligned. “The digital god says you can’t build it” probably sounds a lot more compelling than “Joe Biden says you can’t build it”.
This is a good question to ask, and my general answer is to this question is a combination of 2 answers, defense doesn’t trump offense in all domains, but they are much more balanced than LWers think, with the exception of bio, and that domain mostly doesn’t have existential risky products. Regulation is necessary for superintelligence, but I don’t think this is anywhere near true:
No, primarily because misuse/structural issues demand very different responses, and a lot of the policy making pretty much relies on the assumption that AI is hard to control.
Much more generally, I wish people would make distinctions between existential risk caused by lack of control vs existential risk caused by misuse vs mass death caused by structural forces, since each of these relies on very differing causes, and that matters since the policy conclusions are very different, and sometimes even opposed to each other.
Open Sourcing is a case in point. It’s negative in the cases where misuse and loss control is the dominant risk factor, but turns into a positive if we instead assume structural forces are at work, like in dr_s’s story here:
https://www.lesswrong.com/posts/2ujT9renJwdrcBqcE/the-benevolence-of-the-butcher
More generally, policies for one scenario will not work for other scenarios for AI risk automatically, it’s a case by case basis.