“That is, where are the Matt Levines of, say, chemistry or drug development1,”
You are looking for Derek Lowe’s “In the pipeline.” It appears on hacker news occasionally.
Maelstrom
Karma: 15
[Question] Searching for Impossibility Results or No-Go Theorems for provable safety.
The crux of these types of arguments seems to be conflating the provable safety of an agent in a system with the expectation of absolute safety. In my experience, this is the norm, not the exception, and needs to be explicitly addressed.
In agreement with what you posted above, I think it is formally trivial to construct a scenario in which a pedestrian jumps in front of a car, making it provably impossible for a vehicle to stop in time to avoid a collision using high school physics.
Likewise, I have the intuition that AI safety, in general, should have various “no-go theorems” about unprovability outside a reasonable problem scope or that finding such proofs would be np-hard or worse. If you know of any specific results( outside of general computability theory) , could you please share them? It would be nice if the community could avoid falling into the trap of trying to prove too much.
(Sorry if this isn’t the correct location for this post.)
One needs only to read 4 or so papers on category theory applied to AI to understand the problem. None of them share a common foundation on what type of constructions to use or formalize in category theory. The core issue is that category theory is a general language for all of mathematics, and as commonly used just exponentially increase the search space for useful mathematical ideas.
I want to be wrong about this, but I have yet to find category theory uniquely useful outside of some subdomains of pure math.