I agree that unrestricted Internet access is Bad(TM). Given the Internet, a completely unbounded intelligence could very probably cause massive havoc, essentially at an x-risk level, damned fast… but it’s not a certainty, and I think “damned fast” is in the range of months to years even if your intelligence is truly unbounded. You have to work through tools that can only go so fast, and stealth will slow you down even more (while still necessarily being imperfect).
… but a lot of the talk on here is in the vein of “if it gets to say one sentence to one randomly selected person, it can destroy the world”. Even if it also has limited knowledge of the outside world. If people don’t actually believe that, it’s still sometimes seen as a necessary conservative assumption. That’s getting pretty far out there. While “conservative” in one sense, that strong an assumption could keep you from applying safety measures that would actually be effective, so it it can be “anti-conservative” in other senses. Admittedly the extreme view doesn’t seem to be so common among the people actually trying to figure out how to build stuff, but it still colors everybody’s thoughts.
Also, my points interact with one another. If you are a real superintelligence, with effectively instantaneous logical omniscience, Internet access is very probably enough (and I still claim being able to say one sentence is very probably not enough). But if you just have “a really high IQ”, Internet access may not be enough, and being able to say one sentence is definitely not enough. And even trying to figure out how to use what you have can be a problem, if you start sucking down expensive computing resources without producing visible results.
If those limitations give people a softer takeoff, more time to figure things out, and an ability to have some experience with AGI without the first bug destroying you, it seems like they have a better chance to survive.
Also, I’d like to defend myself by pointing out, that I said “more optimistic”, not “optimistic in an absolute sense”. I resist puttting numbers on these things, but I’m not sure I’d say survival was better than fifty-fifty, even considering the limitations I mentioned. Some days I’d go far worse, fewer days I’d go a bit better.
I agree that unrestricted Internet access is Bad(TM). Given the Internet, a completely unbounded intelligence could very probably cause massive havoc, essentially at an x-risk level, damned fast… but it’s not a certainty, and I think “damned fast” is in the range of months to years even if your intelligence is truly unbounded. You have to work through tools that can only go so fast, and stealth will slow you down even more (while still necessarily being imperfect).
… but a lot of the talk on here is in the vein of “if it gets to say one sentence to one randomly selected person, it can destroy the world”. Even if it also has limited knowledge of the outside world. If people don’t actually believe that, it’s still sometimes seen as a necessary conservative assumption. That’s getting pretty far out there. While “conservative” in one sense, that strong an assumption could keep you from applying safety measures that would actually be effective, so it it can be “anti-conservative” in other senses. Admittedly the extreme view doesn’t seem to be so common among the people actually trying to figure out how to build stuff, but it still colors everybody’s thoughts.
Also, my points interact with one another. If you are a real superintelligence, with effectively instantaneous logical omniscience, Internet access is very probably enough (and I still claim being able to say one sentence is very probably not enough). But if you just have “a really high IQ”, Internet access may not be enough, and being able to say one sentence is definitely not enough. And even trying to figure out how to use what you have can be a problem, if you start sucking down expensive computing resources without producing visible results.
If those limitations give people a softer takeoff, more time to figure things out, and an ability to have some experience with AGI without the first bug destroying you, it seems like they have a better chance to survive.
Also, I’d like to defend myself by pointing out, that I said “more optimistic”, not “optimistic in an absolute sense”. I resist puttting numbers on these things, but I’m not sure I’d say survival was better than fifty-fifty, even considering the limitations I mentioned. Some days I’d go far worse, fewer days I’d go a bit better.