It’s not clear to me that your metaphors are pointing at something in particular.
Revenue of a nail factory is a good proxy for the quality of the nails produced, but only within a fairly small bubble around our current world. You can’t make the factory-owner too smart, or the economy too irrational, or allow for too many technological breakthroughs to happen, or else the proxy breaks. If this was all we needed, then yes, absolutely, I’m sure there’s a similarly neat and simple way to instrumentalize human values—it’s just going to fail if things are too smart, or too irrational, or too far in the future.
Biology being human-comprehensible is an interesting topic, and suppose I grant that it is—that we could have comprehensible explanatory stories for every thing our cells do, and that these stories aren’t collectively leaving anything out. First off, I would like to note that such a collection of stories would still be really complicated relative to simple abstractions in physics or economics! Second, this doesn’t connect directly to Goodhart’s law. We’re just talking about understanding biology, without mentioning purposes to which our understanding can be applied. Comprehending biology might help us generalize, in the sense of being able to predict what features will be conserved by mutation, or will adapt to a perturbed environment, but again this generalization only seems to work in a limited range, where the organism is doing all the same jobs with the same divisions between them.
The butterfly effect metaphor seems like the opposite of biology. In biology you can have lots of little important pieces—they’re not individually redirecting the whole hurricane/organism, but they’re doing locally-important jobs that follow comprehensible rules, and so we don’t disregard them as noise. None of the butterflies have such locally-useful stories about what they’re doing to the hurricane, they’re all just applying small incomprehensible perturbations to a highly chaotic system. The lesson I take is that messiness is not the total lack of structure—when I say my room is messy, I don’t mean that the arrangement of its component atoms has been sampled from the Boltzmann distribution—it’s just that the structure that’s there isn’t easy for humans to use.
I’d like to float one more metaphor: K-complexity and compression.
Suppose I have a bit string of length 10^9, and I can compress it down to length 10^8. The “True Name hypothesis” is that the compression looks like finding some simple, neat patterns that explain most of the data and we expect to generalize well, plus a lot of “diff” that’s the noisy difference between the simple rules and the full bitstring. The “fractal hypothesis” is that there are a few simple patterns that do some of the work, and a few less simple rules that do more of the work, and so on for as long as you have patience. The “total mess hypothesis” is that simple rules do a small amount of the work, and a lot of the 10^8 bits is big highly-interdependent programs that would output something very different if you flipped just a few bits. Does this seem about right?
Revenue of a nail factory is a good proxy for the quality of the nails produced, but only within a fairly small bubble around our current world. You can’t make the factory-owner too smart, or the economy too irrational, or allow for too many technological breakthroughs to happen, or else the proxy breaks.
I think you missed the point of that particular metaphor. The claim was not that revenue of a nail factory is a robust operationalization of nail value. The claim was that a competitive nail market plus nail-maker reputation tracking is a True Name for a pointer to nail value—i.e. such a system will naturally generate economically-valuable nails. Because we have a robust mathematical formalization of efficient markets, we know the conditions under which that pointer-to-nail-value will break down: things like the factory owner being smart enough to circumvent the market mechanism, or the economy too irrational, etc.
The lesson I take is that messiness is not the total lack of structure—when I say my room is messy, I don’t mean that the arrangement of its component atoms has been sampled from the Boltzmann distribution—it’s just that the structure that’s there isn’t easy for humans to use.
I agree with this, and it’s a good summary of the takeaway of the butterfly effect analogy. In this frame, I think our disagreement is about whether “structure which isn’t easy for humans to use” is generally hard to use because the humans haven’t yet figured it out (but they could easily use it if they did figure it out) vs structure which humans are incapable of using due to hardware limitations of the brain.
Suppose I have a bit string of length 10^9, and I can compress it down to length 10^8. …
This is an anology which I also considered bringing up, and I think you’ve analogized things basically correctly here. One important piece: if I can compress a bit string down to length 10^8, and I can’t compress it any further, then that program of length 10^8 is itself incompressible—i.e. it’s 10^8 random bits. As with the butterfly effect, we get a duality between structure and noise.
Actually, to be somewhat more precise: it may be that we could compress the length 10^8 program somewhat, but then we’d still need to run the decompressed program through an interpreter in order for it to generate our original bitstring. So the actual rule is something roughly like “any maximally-compressed string consists of a program shorter than roughly-the-length-of-the-shortest-interpreter, plus random bits” (with the obvious caveat that the short program and the random bits may not separate neatly).
It’s not clear to me that your metaphors are pointing at something in particular.
Revenue of a nail factory is a good proxy for the quality of the nails produced, but only within a fairly small bubble around our current world. You can’t make the factory-owner too smart, or the economy too irrational, or allow for too many technological breakthroughs to happen, or else the proxy breaks. If this was all we needed, then yes, absolutely, I’m sure there’s a similarly neat and simple way to instrumentalize human values—it’s just going to fail if things are too smart, or too irrational, or too far in the future.
Biology being human-comprehensible is an interesting topic, and suppose I grant that it is—that we could have comprehensible explanatory stories for every thing our cells do, and that these stories aren’t collectively leaving anything out. First off, I would like to note that such a collection of stories would still be really complicated relative to simple abstractions in physics or economics! Second, this doesn’t connect directly to Goodhart’s law. We’re just talking about understanding biology, without mentioning purposes to which our understanding can be applied. Comprehending biology might help us generalize, in the sense of being able to predict what features will be conserved by mutation, or will adapt to a perturbed environment, but again this generalization only seems to work in a limited range, where the organism is doing all the same jobs with the same divisions between them.
The butterfly effect metaphor seems like the opposite of biology. In biology you can have lots of little important pieces—they’re not individually redirecting the whole hurricane/organism, but they’re doing locally-important jobs that follow comprehensible rules, and so we don’t disregard them as noise. None of the butterflies have such locally-useful stories about what they’re doing to the hurricane, they’re all just applying small incomprehensible perturbations to a highly chaotic system. The lesson I take is that messiness is not the total lack of structure—when I say my room is messy, I don’t mean that the arrangement of its component atoms has been sampled from the Boltzmann distribution—it’s just that the structure that’s there isn’t easy for humans to use.
I’d like to float one more metaphor: K-complexity and compression.
Suppose I have a bit string of length 10^9, and I can compress it down to length 10^8. The “True Name hypothesis” is that the compression looks like finding some simple, neat patterns that explain most of the data and we expect to generalize well, plus a lot of “diff” that’s the noisy difference between the simple rules and the full bitstring. The “fractal hypothesis” is that there are a few simple patterns that do some of the work, and a few less simple rules that do more of the work, and so on for as long as you have patience. The “total mess hypothesis” is that simple rules do a small amount of the work, and a lot of the 10^8 bits is big highly-interdependent programs that would output something very different if you flipped just a few bits. Does this seem about right?
I think you missed the point of that particular metaphor. The claim was not that revenue of a nail factory is a robust operationalization of nail value. The claim was that a competitive nail market plus nail-maker reputation tracking is a True Name for a pointer to nail value—i.e. such a system will naturally generate economically-valuable nails. Because we have a robust mathematical formalization of efficient markets, we know the conditions under which that pointer-to-nail-value will break down: things like the factory owner being smart enough to circumvent the market mechanism, or the economy too irrational, etc.
I agree with this, and it’s a good summary of the takeaway of the butterfly effect analogy. In this frame, I think our disagreement is about whether “structure which isn’t easy for humans to use” is generally hard to use because the humans haven’t yet figured it out (but they could easily use it if they did figure it out) vs structure which humans are incapable of using due to hardware limitations of the brain.
This is an anology which I also considered bringing up, and I think you’ve analogized things basically correctly here. One important piece: if I can compress a bit string down to length 10^8, and I can’t compress it any further, then that program of length 10^8 is itself incompressible—i.e. it’s 10^8 random bits. As with the butterfly effect, we get a duality between structure and noise.
Actually, to be somewhat more precise: it may be that we could compress the length 10^8 program somewhat, but then we’d still need to run the decompressed program through an interpreter in order for it to generate our original bitstring. So the actual rule is something roughly like “any maximally-compressed string consists of a program shorter than roughly-the-length-of-the-shortest-interpreter, plus random bits” (with the obvious caveat that the short program and the random bits may not separate neatly).