I don’t think, from the perspective of humans monitoring single ML system running a concrete, quantifiable process—industry or mining or machine design—that it will be unexplainable. Just like today, tech stacks are already enormously complex, but at each layer someone does know how they work, and we know what they do at the layers that matter.
This seems like the key question.
Ever more complex designs for, say, a mining robot might start to resemble more and more some mix of living creatures and artwork out of a fractal, but we’ll still have reports that measure how much performance the design gives per cost.
I think that if we relate to our machines in the same way we relate to biological systems or ecologies, but AI systems actually understand those systems very well, then that’s basically what I mean.
Having reports about outcomes is a kind of understanding, but it’s basically the one I’m scared of (since e.g. it will be tough to learn about these kinds of systemic risks via outcome-driven reports, and attempts to push down near-misses may just transform them into full-blown catastrophes).
This seems like the key question.
I think that if we relate to our machines in the same way we relate to biological systems or ecologies, but AI systems actually understand those systems very well, then that’s basically what I mean.
Having reports about outcomes is a kind of understanding, but it’s basically the one I’m scared of (since e.g. it will be tough to learn about these kinds of systemic risks via outcome-driven reports, and attempts to push down near-misses may just transform them into full-blown catastrophes).