For any point on the spectrum there is some difficulty in achieving it.
We can approach that point from either direction, 1) starting with a “big blob of compute” and encountering the difficulty in extracting these pieces from the blob, or 2) starting with assembling the pieces, and encountering the difficulty in figuring out how to assemble them.
It’s not at all clear that (1) would be easier than (2).
Probably it’s best to do some of both.
Regarding difficulty of (1) vs. (2), OTMH, there may be some sort of complexity-style argument that engineering, say, a circuit is harder than recognizing it. However, the DNN doesn’t produce the circuit, we still need to do that using interpretability techniques. So I’m not sure how I feel about this argument.
I agree it’s a spectrum. I would put it this way:
For any point on the spectrum there is some difficulty in achieving it.
We can approach that point from either direction, 1) starting with a “big blob of compute” and encountering the difficulty in extracting these pieces from the blob, or 2) starting with assembling the pieces, and encountering the difficulty in figuring out how to assemble them.
It’s not at all clear that (1) would be easier than (2).
Probably it’s best to do some of both.
Regarding difficulty of (1) vs. (2), OTMH, there may be some sort of complexity-style argument that engineering, say, a circuit is harder than recognizing it. However, the DNN doesn’t produce the circuit, we still need to do that using interpretability techniques. So I’m not sure how I feel about this argument.