One of the reasons I worry about cybersecurity, and the sorry state it’s in, is that it provides an easy path for human-level and even infrahuman-level AIs to acquire additional computation. In some plausible worlds, this turns a manageable infrahuman AI into an unmanageable superintelligence, when the creator’s decision would have been not to launch.
Unlike solving protein-design and constructing nanobots, this is something definitely within reach of human-level intelligence; many people have done it for ordinary criminal purposes, like mining cryptocurrency.
Current cybersecurity research incentives don’t seem quite right for getting people to mitigate this risk. If I were trying to stop an AI from taking over the internet and using the extra hardware to make itself superintelligent, I would worry a lot less about protecting user data, phones and whatnot, and worry a lot more about protecting software developer workstations in particular.
Am incline to agree, but I want to add that security is all connected. There are several direct causal paths from compromised user data to compromised dev workstation (and vice versa).
One of the reasons I worry about cybersecurity, and the sorry state it’s in, is that it provides an easy path for human-level and even infrahuman-level AIs to acquire additional computation. In some plausible worlds, this turns a manageable infrahuman AI into an unmanageable superintelligence, when the creator’s decision would have been not to launch.
Unlike solving protein-design and constructing nanobots, this is something definitely within reach of human-level intelligence; many people have done it for ordinary criminal purposes, like mining cryptocurrency.
Current cybersecurity research incentives don’t seem quite right for getting people to mitigate this risk. If I were trying to stop an AI from taking over the internet and using the extra hardware to make itself superintelligent, I would worry a lot less about protecting user data, phones and whatnot, and worry a lot more about protecting software developer workstations in particular.
Am incline to agree, but I want to add that security is all connected. There are several direct causal paths from compromised user data to compromised dev workstation (and vice versa).