The other critical thing that the current terms seem to miss is the deep and inscrutable nature of the systems
I have one comment here. Electronic systems that have a fault, and low level software constructs with a sporadic fault, can feel inscrutable when you are trying to chase down the bug. This also happens when you have poor layering in a software stack, the stack becomes ‘deep’ and difficult to debug because the layers are fused together, and so a bug can be at any layer. (aka it could be application/runtime libs/userspace driver/driver/comm protocol/pcie switch/firmware or a silicon bug). SWEs can separate into isolated layers using various techniques, the layering for TCP/IP is a famous example.
Early electronics, using less reliable gates, were also like this. Vacuum tubes blew often.
LLMs are effectively trained on a bunch of trash—whatever human text they were trained on, and so there’s a lot of dirt in there. Unreliable behavior is going to always be the norm, same as early electronics.
“Deepware” trained on synthetic data may be better. (have an LLM generate a bunch of text with a consistent voice and grounded reasoning, and train another one on that, as an example. this is also bootstrapping → early unreliable electronics controlled the machines that made their successors)
There also may be a way to layer ‘deepware’ into accountable and separable systems.
With that said, yes it’s a new paradigm. Just don’t assume the problems of today are eternal to the field.
Summary: { dirty human text is similar to impure materials in early electronics, and deep networks that perform all functions have no layer separation and are not debuggable, similar to early software. Neither is the standard now }
Good point, and I agree that it’s possible that what I see as essential features might go away—“floppy disks” turned out to be a bad name when they ended up inside hard plastic covers, and “deepware” could end up the same—but I am skeptical that it will.
I agree that early electronics were buggy until we learned to build them reliably—and perhaps we can solve this for gradient-descent based learning, though many are skeptical of that, since many of the problems have been shown to be pretty fundamental. I also agree that any system is inscrutable until you understand it, but unlike early electronics, no-one understands these massive lists of numbers that produce text, and human brains can’t build them, they just program a process to grow them. (Yes, composable NNs could solve some of this, as you point out when mentioning separable systems, but I still predict they won’t be well understood, because the components individually are still deepware.)
I agree that early electronics were buggy until we learned to build them reliably—and perhaps we can solve this for gradient-descent based learning, though many are skeptical of that, since many of the problems have been shown to be pretty fundamental. I also agree that any system is inscrutable until you understand it, but unlike early electronics, no-one understands these massive lists of numbers that produce text, and human brains can’t build them, they just program a process to grow them
I can say from experience that no one “understands” complex software and hardware systems. Any little bug can take weeks to isolate and patch, and you end up in calls with multiple domain specialists. Rare non deterministic bugs you will never find the cause for in the lifetime of the project.
This means you need to use black box methods for testing and reliability analysis. You empirically measure how often it fails, you don’t know how often it will fail by looking at the design.
These same methods apply mostly unchanged to AI. The only major difference I see with AI is the possibility of coordinated failure, where multiple AI systems conspire to all fail at the same time. This implies that AI alignment may ultimately be based around the simple idea of reducing complex AI systems that can fail in a coordinated way, to simpler AI systems that fail independently. (Note this doesn’t mean it will be simple to do. See what humans have to do to control fire. Just 3 ingredients you have to keep apart, yet you need all this infrastructure)
This is the “pinnacle” of software engineering at present : rip apart your software from this complex, coupled thing to a bunch of separate simpler things that are each testable, and a modern flavor is to do things like make every element in a complex GUI use a separate copy of the JavaScript libraries.
I have one comment here. Electronic systems that have a fault, and low level software constructs with a sporadic fault, can feel inscrutable when you are trying to chase down the bug. This also happens when you have poor layering in a software stack, the stack becomes ‘deep’ and difficult to debug because the layers are fused together, and so a bug can be at any layer. (aka it could be application/runtime libs/userspace driver/driver/comm protocol/pcie switch/firmware or a silicon bug). SWEs can separate into isolated layers using various techniques, the layering for TCP/IP is a famous example.
Early electronics, using less reliable gates, were also like this. Vacuum tubes blew often.
LLMs are effectively trained on a bunch of trash—whatever human text they were trained on, and so there’s a lot of dirt in there. Unreliable behavior is going to always be the norm, same as early electronics.
“Deepware” trained on synthetic data may be better. (have an LLM generate a bunch of text with a consistent voice and grounded reasoning, and train another one on that, as an example. this is also bootstrapping → early unreliable electronics controlled the machines that made their successors)
There also may be a way to layer ‘deepware’ into accountable and separable systems.
With that said, yes it’s a new paradigm. Just don’t assume the problems of today are eternal to the field.
Summary: { dirty human text is similar to impure materials in early electronics, and deep networks that perform all functions have no layer separation and are not debuggable, similar to early software. Neither is the standard now }
Good point, and I agree that it’s possible that what I see as essential features might go away—“floppy disks” turned out to be a bad name when they ended up inside hard plastic covers, and “deepware” could end up the same—but I am skeptical that it will.
I agree that early electronics were buggy until we learned to build them reliably—and perhaps we can solve this for gradient-descent based learning, though many are skeptical of that, since many of the problems have been shown to be pretty fundamental. I also agree that any system is inscrutable until you understand it, but unlike early electronics, no-one understands these massive lists of numbers that produce text, and human brains can’t build them, they just program a process to grow them. (Yes, composable NNs could solve some of this, as you point out when mentioning separable systems, but I still predict they won’t be well understood, because the components individually are still deepware.)
I can say from experience that no one “understands” complex software and hardware systems. Any little bug can take weeks to isolate and patch, and you end up in calls with multiple domain specialists. Rare non deterministic bugs you will never find the cause for in the lifetime of the project.
This means you need to use black box methods for testing and reliability analysis. You empirically measure how often it fails, you don’t know how often it will fail by looking at the design.
These same methods apply mostly unchanged to AI. The only major difference I see with AI is the possibility of coordinated failure, where multiple AI systems conspire to all fail at the same time. This implies that AI alignment may ultimately be based around the simple idea of reducing complex AI systems that can fail in a coordinated way, to simpler AI systems that fail independently. (Note this doesn’t mean it will be simple to do. See what humans have to do to control fire. Just 3 ingredients you have to keep apart, yet you need all this infrastructure)
This is the “pinnacle” of software engineering at present : rip apart your software from this complex, coupled thing to a bunch of separate simpler things that are each testable, and a modern flavor is to do things like make every element in a complex GUI use a separate copy of the JavaScript libraries.