like most all computer systems today, very well tested to assure that its behavior was aligned well with its owners’ goals across its domains of usage
I’m sorta skeptical about this point of Hanson’s—current software is already very imperfect and buggy. In the last decade, we had two whole-Internet-threatening critical bugs—Heartbleed & Log4Shell. Heartbleed was in the main encryption library most of the Internet uses for like 2 years before it was discovered, and most of the Internet was vulnerable to Log4Shell for like 7 years before anyone found it (persisting through 14 versions from 2.0 to 2.14). And Microsoft routinely finds & patches Windows bugs that impact years-old versions of its OS.
So it’s not like our current testing capabilities are infallibly robust or anything—Heartbleed was a classic out-of-bounds buffer read, and Log4Shell was straight-up poor design that should’ve been caught. And this code was on ALL SORTS of critical systems—Heartbleed affected everything from VOIP phones to World of Warcraft to payment providers like Stripe and major sites like AWS or Wikipedia. Log4J hit basically every Cloud provider, and resulted in ransomware attacks against several governments.
Another facet of this problem that this example highlights—OpenSSL was a critical infrastructure component for most of the internet, maintained by only one or two people. The risk of AI won’t necessarily come from a highly organized dev team at a Big Tech/MAMAA company, but could come from a scrappy startup or under-resourced foundation where the testing protocols and coding aren’t as robust. The idea that some sort of misalignment could creep in there and remain hidden for a few versions doesn’t seem outlandish at all.
I’m sorta skeptical about this point of Hanson’s—current software is already very imperfect and buggy. In the last decade, we had two whole-Internet-threatening critical bugs—Heartbleed & Log4Shell. Heartbleed was in the main encryption library most of the Internet uses for like 2 years before it was discovered, and most of the Internet was vulnerable to Log4Shell for like 7 years before anyone found it (persisting through 14 versions from 2.0 to 2.14). And Microsoft routinely finds & patches Windows bugs that impact years-old versions of its OS.
So it’s not like our current testing capabilities are infallibly robust or anything—Heartbleed was a classic out-of-bounds buffer read, and Log4Shell was straight-up poor design that should’ve been caught. And this code was on ALL SORTS of critical systems—Heartbleed affected everything from VOIP phones to World of Warcraft to payment providers like Stripe and major sites like AWS or Wikipedia. Log4J hit basically every Cloud provider, and resulted in ransomware attacks against several governments.
Another facet of this problem that this example highlights—OpenSSL was a critical infrastructure component for most of the internet, maintained by only one or two people. The risk of AI won’t necessarily come from a highly organized dev team at a Big Tech/MAMAA company, but could come from a scrappy startup or under-resourced foundation where the testing protocols and coding aren’t as robust. The idea that some sort of misalignment could creep in there and remain hidden for a few versions doesn’t seem outlandish at all.