Status: strong opinions, weakly held. not a control theorist; not only ready to eat my words, but I’ve already set the table.
As I understand it, the original good regulator theorem seems even dumber than you point out.
First, the original paper doesn’t make sense to me. Not surprising, old papers are often like that, and I don’t know any control theory… but here’s John Baez also getting stuck, giving up, and deriving his own version of what he imagines the theorem should say:
when I tried to read Conant and Ashby’s paper, I got stuck. They use some very basic mathematical notation in nonstandard ways, and they don’t clearly state the hypotheses and conclusion of their theorem...
However, I have a guess about the essential core of Conant and Ashby’s theorem. So, I’ll state that, and then say more about their setup.
Needless to say, I looked around to see if someone else had already done the work of figuring out what Conant and Ashby were saying...
As pointed out by the authors of [3], the importance and generality of this theorem in control theory makes it comparable in importance to Einstein’s E=mc2 for physics. However, as John C. Baez carefully argues in a blog post titled The Internal Model Principle it’s not clear that Conant and Ashby’s paper demonstrates what it sets out to prove. I’d like to add that many other researchers, besides myself, share John C. Baez’ perspective.
Hello?? Isn’t this one of the fundamental results of control theory? Where’s a good proof of it? It’s been cited 1,317 times and confidently brandished to make sweeping claims about how to e.g. organize society or make an ethical superintelligence.
It seems plausible that people just read the confident title (Every good regulator of a system must be a model of that system—of course the paper proves the claim in its title...), saw the citation count / assumed other people had checked it out (yay information cascades!), and figured it must be correct...
Motte and bailey
The paper starts off by introducing the components of the regulatory system:
OK, so we’re going to be talking about how regulators which ensure good outcomesalso model their environments, right? Sounds good.
Wait...
They defined the intuitively correct version, and then retreat to this.
Later...
We’re talking about the entire outcome space Z again. In the introduction we focused on regulators ensuring ‘good’ states, but we immediately gave that up to talk about entropy H(Z).
Why does this matter? Well...
The original theorem seems even dumber than John points out
John writes:
Also, though I don’t consider it a “problem” so much as a choice which I think most people here will find more familiar:
The theorem uses entropy-minimization as its notion of optimality, rather than expected-utility-maximization
I suppose my intuition is that this is actually a significant problem.
At first glance, Good Regulator seems to basically prove something like ‘there’s a deterministic optimal policy wrt the observations’, but even that’s too generous—it proves that there’s a deterministic way to minimize outcome entropy. But what does that guarantee us—how do we know that’s a ‘good regulator’? Like, imagine an environment with a strong “attractor” outcome, like the streets being full of garbage. The regulator can try to push against this, but they can’t always succeed due to the influence of random latent variables (this cuts against the determinism assumption, but you note that this can be rectified by reincluding X). However, by sitting back, they can ensure that the streets are full of garbage.
The regulator does so, minimizes the entropy over the unconditional outcome distribution Z, and is christened a ‘good regulator’ which has built a ‘model’ of the environment. In reality, we have a deterministic regulator which does nothing, and our streets are full of trash.
Now, I think it’s possible I’ve misunderstood, so I’d appreciate correction if I have. But if I haven’t, and if no control theorists have in fact repaired and expanded this theorem before John’s post...
If that’s true, what the heck happened? Control theorists just left a $100 bill on the ground for decades? A quick !scholar search doesn’t reveal any good critiques.
“Good regulator” is used here to mean that it is good at keeping the output “regular”. That is, reducing the entropy instead of “a regulator which produces good”.
On page 4 the authors acknowledge that there are numerous ways by which one could consider a regulator successful, then go on to say “In this paper we shall use the last measure, H(Z), and we define ‘successful regulation’ as equivalent, to ‘H(Z) is minimal’.”
The fact that reducing entropy doesn’t line up with maximizing utility is true, but the authors never claimed it did. Reducing entropy generalises to a lot of real world problems.
The reason I think entropy minimization is basically an ok choice here is that there’s not much restriction on which variable’s entropy is minimized. There’s enough freedom that we can transform an expected-utility-maximization problem into an entropy-minimization problem.
In particular, suppose we have a utility variable U, and we want to maximize E[U]. As long as possible values of U are bounded above, we can subtract a constant without changing anything, making U strictly negative. Then, we define a new random variable Z, which is generated from U in such a way that its entropy (given U) is -U bits. For instance, we could let Z be a list of ⌊−U⌋50⁄50 coin flips, plus one biased coin flip with bias chosen so the entropy of the coin flip is −U−⌊−U⌋, i.e. the fractional part of U. Then, minimizing entropy of Z (unconditional on U) is equivalent to maximizing E[U].
I had a little tinker with this. It’s straightforward to choose a utility function where maximising it is equivalent to minimizing H(Z) - just set U(z)=logp(z).
As far as I can see, the other way round is basically as you suggested, but a tiny bit more fiddly. We can indeed produce a nonpositive U′ from which we make a new RV Z′ with H(Z′|z)=−U(z) as you suggested (e.g. with coinflips etc). But a simple shift of U isn’t enough. We need U′(z)=mU(z)−logp(z)+c (for some scalar m and c) - note the logp(z) term.
We take the z′ outcomes to be partitioned by z, i.e. they’re ′z happened and also I got z′ coinflip outcome’. Then P(z′)=P(z)P(z′|z) (where z is understood to be the particular z associated with z′). That means H(Z|Z′)=0 so H(Z′)=H(Z)+H(Z′|Z) (you can check this by spelling things out pointfully and rearranging, but I realised that I was just rederiving conditional entropy laws).
so minimizing H(Z′) is equivalent to maximising EzU(z).
Requiring U′(z)=mU(z)−logp(z)+c to be nonpositive for all z maybe places more constraints on things? Certainly U needs to be bounded above as you said. It’s also a bit weird and embedded as you hinted, because this utility function depends on the probability of the outcome, which is the thing being controlled/regulated by the decisioner. I don’t know if there are systems where this might not be well-defined even for bounded U, haven’t dug into it.
Okay, I agree that if you remove their determinism & full observability assumption (as you did in the post), it seems like your construction should work.
I still think that the original paper seems awful (because it’s their responsibility to justify choices like this in order to explain how their result captures the intuitive meaning of a ‘good regulator’).
Status: strong opinions, weakly held. not a control theorist; not only ready to eat my words, but I’ve already set the table.
As I understand it, the original good regulator theorem seems even dumber than you point out.
First, the original paper doesn’t make sense to me. Not surprising, old papers are often like that, and I don’t know any control theory… but here’s John Baez also getting stuck, giving up, and deriving his own version of what he imagines the theorem should say:
An unanswered StackExchange question asks whether anyone has a rigorous proof:
Hello?? Isn’t this one of the fundamental results of control theory? Where’s a good proof of it? It’s been cited 1,317 times and confidently brandished to make sweeping claims about how to e.g. organize society or make an ethical superintelligence.
It seems plausible that people just read the confident title (Every good regulator of a system must be a model of that system—of course the paper proves the claim in its title...), saw the citation count / assumed other people had checked it out (yay information cascades!), and figured it must be correct...
Motte and bailey
The paper starts off by introducing the components of the regulatory system:
OK, so we’re going to be talking about how regulators which ensure good outcomes also model their environments, right? Sounds good.
Wait...
Later...
We’re talking about the entire outcome space Z again. In the introduction we focused on regulators ensuring ‘good’ states, but we immediately gave that up to talk about entropy H(Z).
Why does this matter? Well...
The original theorem seems even dumber than John points out
John writes:
I suppose my intuition is that this is actually a significant problem.
At first glance, Good Regulator seems to basically prove something like ‘there’s a deterministic optimal policy wrt the observations’, but even that’s too generous—it proves that there’s a deterministic way to minimize outcome entropy. But what does that guarantee us—how do we know that’s a ‘good regulator’? Like, imagine an environment with a strong “attractor” outcome, like the streets being full of garbage. The regulator can try to push against this, but they can’t always succeed due to the influence of random latent variables (this cuts against the determinism assumption, but you note that this can be rectified by reincluding X). However, by sitting back, they can ensure that the streets are full of garbage.
The regulator does so, minimizes the entropy over the unconditional outcome distribution Z, and is christened a ‘good regulator’ which has built a ‘model’ of the environment. In reality, we have a deterministic regulator which does nothing, and our streets are full of trash.
Now, I think it’s possible I’ve misunderstood, so I’d appreciate correction if I have. But if I haven’t, and if no control theorists have in fact repaired and expanded this theorem before John’s post...
If that’s true, what the heck happened? Control theorists just left a $100 bill on the ground for decades? A quick !scholar search doesn’t reveal any good critiques.
“Good regulator” is used here to mean that it is good at keeping the output “regular”. That is, reducing the entropy instead of “a regulator which produces good”.
On page 4 the authors acknowledge that there are numerous ways by which one could consider a regulator successful, then go on to say “In this paper we shall use the last measure, H(Z), and we define ‘successful regulation’ as equivalent, to ‘H(Z) is minimal’.”
The fact that reducing entropy doesn’t line up with maximizing utility is true, but the authors never claimed it did. Reducing entropy generalises to a lot of real world problems.
The reason I think entropy minimization is basically an ok choice here is that there’s not much restriction on which variable’s entropy is minimized. There’s enough freedom that we can transform an expected-utility-maximization problem into an entropy-minimization problem.
In particular, suppose we have a utility variable U, and we want to maximize E[U]. As long as possible values of U are bounded above, we can subtract a constant without changing anything, making U strictly negative. Then, we define a new random variable Z, which is generated from U in such a way that its entropy (given U) is -U bits. For instance, we could let Z be a list of ⌊−U⌋ 50⁄50 coin flips, plus one biased coin flip with bias chosen so the entropy of the coin flip is −U−⌊−U⌋, i.e. the fractional part of U. Then, minimizing entropy of Z (unconditional on U) is equivalent to maximizing E[U].
I had a little tinker with this. It’s straightforward to choose a utility function where maximising it is equivalent to minimizing H(Z) - just set U(z)=logp(z).
As far as I can see, the other way round is basically as you suggested, but a tiny bit more fiddly. We can indeed produce a nonpositive U′ from which we make a new RV Z′ with H(Z′|z)=−U(z) as you suggested (e.g. with coinflips etc). But a simple shift of U isn’t enough. We need U′(z)=mU(z)−logp(z)+c (for some scalar m and c) - note the logp(z) term.
We take the z′ outcomes to be partitioned by z, i.e. they’re ′z happened and also I got z′ coinflip outcome’. Then P(z′)=P(z)P(z′|z) (where z is understood to be the particular z associated with z′). That means H(Z|Z′)=0 so H(Z′)=H(Z)+H(Z′|Z) (you can check this by spelling things out pointfully and rearranging, but I realised that I was just rederiving conditional entropy laws).
Then
−H(Z′)=−H(Z)−H(Z′|Z)=−H(Z)+EzU′(z)=−H(Z)+mEzU(z)−Ezlogp(z)+c=mEzU(z)+c
so minimizing H(Z′) is equivalent to maximising EzU(z).
Requiring U′(z)=mU(z)−logp(z)+c to be nonpositive for all z maybe places more constraints on things? Certainly U needs to be bounded above as you said. It’s also a bit weird and embedded as you hinted, because this utility function depends on the probability of the outcome, which is the thing being controlled/regulated by the decisioner. I don’t know if there are systems where this might not be well-defined even for bounded U, haven’t dug into it.
Okay, I agree that if you remove their determinism & full observability assumption (as you did in the post), it seems like your construction should work.
I still think that the original paper seems awful (because it’s their responsibility to justify choices like this in order to explain how their result captures the intuitive meaning of a ‘good regulator’).
Oh absolutely, the original is still awful and their proof does not work with the construction I just gave.
BTW, this got a huge grin out of me: