(Writing a summary of the post and my reading of it, in large part for my own understanding.)
The first disagreement is about adaptation-executors vs utility-maximisers. I take the adaptation-executors side mostly, although once in a while a utility maximiser gets through and then lots of stuff happens, and it’s not clear if it’s good but it is exciting.
The second disagreement is whether Zvi can beat the status quo. I think the status quo is adaptation-executors, or blue-minimizing robots. Things that have learned to follow local gradients in a bunch of basic ways, and not really think about reality nor optimize it.
Still, it is not obvious to me that you can easily do better than the adaptation-executor that was selected for by the environment. Put me alone in the North Pole, and for all that I’m an intelligent human, a polar bear will do better.
I think that I’d bet against Zvi working out there, but not overwhelmingly, and it would be worth it to get him there given the potential upside. I’d give him 30% chance of winning and creating greatness. Though I could easily be persuaded further in either direction with more detailed info about the job and about Zvi.
The rest of the post is Zvi giving the details of why the adaptation-executor isn’t optimal, and it’s a lot of details, but all very good detail. In summary:
They are adaptation-executors, which almost by-definition never works well when the environment suddenly changes (e.g. pandemic)
They have been selected to not really think about reality, just to think about whether they’re getting blamed within the next 2 weeks.
Once again, the environment changed suddenly, and they don’t model it.
They’re just actually not that competent.
For some reason they’re incorrectly worried about the costs of good policy change (as I saw in the case of First Doses First). I don’t know why they’re making this mistake though.
They have no time to really think, and also all the information sources around them are adversarial.
The short time horizons of feedback with the public mean that even doing actually good things is kind of not going to be rewarded.
(Here Zvi just makes the argument that if the above is all true, you can’t reason from their not taking an action that the action will not work. Which is fine, but is exactly what Zvi is trying to argue in the first place, so I don’t get this point.)
Doing actually good things just gets you feedback in a very different way than covering your ass avoids getting you punished, and these agents haven’t learned to model that reward. This is nearly the same as 7.
So I do feel like they’re adaptation executors much more than they’re utility function maximisers. Give them an option to do good, where there‘s no risk of being punished, and I don’t think they’ll jump at it. I don’t think they’ll think much about it at all. They’re not built to really notice, they’re weird life forms that exist to be in exactly the weird role they’re usually in, and not get blamed on short time scales.
It isn’t all of my thinking, sometimes politicians actually make longer term bets, or have some semblance of cognition, and the appointees aren’t subject to this as much as the elected people. But this is most of what’s happening, I think.
This all gives me a much scarier feeling about politics. How does literally anything happen at all, if these are the strange beasts we’ve selected to run things? My my. The British TV show The Thick Of It is excellent at portraying many of these properties of people, weak people constantly just trying to cover their asses. (It’s also hilarious.) I suggest Zvi watch one or two episodes, it might help him see more clearly the things he’s trying to say.
Assuming this is an accurate summary, I feel skeptical about “the deal is that The People At The Top are Adaptation Executors”. It just… seems like it must be pretty hard to get to the top without having some kind of longterm planning going on (even if it’s purely manipulative)
The main issue here is that the people in question (heads of the FDA and CDC) are not really The People At The Top. They are bureaucrats promoted to the highest levels of the bureaucracy, and their attitudes and failures are those of career bureaucrats, not successful sociopaths (in the sense of Rao’s “Gervais Principle”).
It just… seems like it must be pretty hard to get to the top without having some kind of longterm planning going on (even if it’s purely manipulative)
I think I would bet against the quoted sentence, though I’m uncertain. The crux for me is whether the optimization-force that causes a single person to end up “at the top” (while many others don’t) is mostly that person’s own optimization-force (vs a set of preferences/flinches/optimization-bits distributed in many others, or in the organization as a whole, or similar).
(This overlaps with jaspax’s comment; but I wanted to state the more general version of the hypothesis.)
I don’t have a very strong guess here. I can imagine the world where being fully adaptation executer outcompetes deliberate strategy. It seems plausible that, at lower levels, adaptation-execution outcompetes strategy because people’s deliberate strategies just aren’t as good as copy-the-neighbors social wisdom.
But, to end up at the very top (even of a career bureaucrat ladder), you have to outcompete all the other people who were copying the neighbors / adaptation executing (I’m currently lumping these strategies together, not sure if that seems fair). It seems to me like this should require some kind of deliberate optimization, of some sort.
The crux for me is whether the optimization-force that causes a single person to end up “at the top” (while many others don’t) is mostly that person’s own optimization-force (vs a set of preferences/flinches/optimization-bits distributed in many others, or in the organization as a whole, or similar).
This phrasing feels overly strong – a world where 65% of the variance is distributed flinches/optimization but 35% personal optimization seems like the sort of world that might exist, in which personal optimization plays a significant role in who ends up at the top.
(Note that I’m basically arguing “the case for ‘there is some kind of deliberate optimization pressure at the top of bureaucracies’ seems higher than Zvi thinks it is”, not making any claims about “this seems most likely how it is”, or “the optimization at the top is altruistic.”)
One relevant question is how many smart, strategic, long-term optimizers exist. (In politics, vs. in business, or in academia, etc.)
E.g., if 1 in 3 people think this way at the start of their careers, that’s very different than if 1 in 1000 do, or 1 in 15. The rarer this way of thinking is, the more powerful it needs to be in order to overcome base rates.
I was also trying to think about the numbers, it felt important but I didn’t get anywhere with it. Where to start? Should I assume a base rate of 1 in a 1,000 people being strategic? More? Less?
I think that one natural way that you get to the top without being long-term strategic and being able to model reality, is nepotism. Some companies are literally handed down father-to-son, so the son doesn’t actually need to show great competence at the task. The Clinton families and the Bush families are also examples. There is some strategy here, but it isn’t strategy that’s closely connected to being competent at the job or even modeling reality that much (other than the social reality within the family).
(Writing a summary of the post and my reading of it, in large part for my own understanding.)
The first disagreement is about adaptation-executors vs utility-maximisers. I take the adaptation-executors side mostly, although once in a while a utility maximiser gets through and then lots of stuff happens, and it’s not clear if it’s good but it is exciting.
The second disagreement is whether Zvi can beat the status quo. I think the status quo is adaptation-executors, or blue-minimizing robots. Things that have learned to follow local gradients in a bunch of basic ways, and not really think about reality nor optimize it.
Still, it is not obvious to me that you can easily do better than the adaptation-executor that was selected for by the environment. Put me alone in the North Pole, and for all that I’m an intelligent human, a polar bear will do better.
I think that I’d bet against Zvi working out there, but not overwhelmingly, and it would be worth it to get him there given the potential upside. I’d give him 30% chance of winning and creating greatness. Though I could easily be persuaded further in either direction with more detailed info about the job and about Zvi.
The rest of the post is Zvi giving the details of why the adaptation-executor isn’t optimal, and it’s a lot of details, but all very good detail. In summary:
They are adaptation-executors, which almost by-definition never works well when the environment suddenly changes (e.g. pandemic)
They have been selected to not really think about reality, just to think about whether they’re getting blamed within the next 2 weeks.
Once again, the environment changed suddenly, and they don’t model it.
They’re just actually not that competent.
For some reason they’re incorrectly worried about the costs of good policy change (as I saw in the case of First Doses First). I don’t know why they’re making this mistake though.
They have no time to really think, and also all the information sources around them are adversarial.
The short time horizons of feedback with the public mean that even doing actually good things is kind of not going to be rewarded.
(Here Zvi just makes the argument that if the above is all true, you can’t reason from their not taking an action that the action will not work. Which is fine, but is exactly what Zvi is trying to argue in the first place, so I don’t get this point.)
Doing actually good things just gets you feedback in a very different way than covering your ass avoids getting you punished, and these agents haven’t learned to model that reward. This is nearly the same as 7.
So I do feel like they’re adaptation executors much more than they’re utility function maximisers. Give them an option to do good, where there‘s no risk of being punished, and I don’t think they’ll jump at it. I don’t think they’ll think much about it at all. They’re not built to really notice, they’re weird life forms that exist to be in exactly the weird role they’re usually in, and not get blamed on short time scales.
It isn’t all of my thinking, sometimes politicians actually make longer term bets, or have some semblance of cognition, and the appointees aren’t subject to this as much as the elected people. But this is most of what’s happening, I think.
This all gives me a much scarier feeling about politics. How does literally anything happen at all, if these are the strange beasts we’ve selected to run things? My my. The British TV show The Thick Of It is excellent at portraying many of these properties of people, weak people constantly just trying to cover their asses. (It’s also hilarious.) I suggest Zvi watch one or two episodes, it might help him see more clearly the things he’s trying to say.
Assuming this is an accurate summary, I feel skeptical about “the deal is that The People At The Top are Adaptation Executors”. It just… seems like it must be pretty hard to get to the top without having some kind of longterm planning going on (even if it’s purely manipulative)
The main issue here is that the people in question (heads of the FDA and CDC) are not really The People At The Top. They are bureaucrats promoted to the highest levels of the bureaucracy, and their attitudes and failures are those of career bureaucrats, not successful sociopaths (in the sense of Rao’s “Gervais Principle”).
I think I would bet against the quoted sentence, though I’m uncertain. The crux for me is whether the optimization-force that causes a single person to end up “at the top” (while many others don’t) is mostly that person’s own optimization-force (vs a set of preferences/flinches/optimization-bits distributed in many others, or in the organization as a whole, or similar).
(This overlaps with jaspax’s comment; but I wanted to state the more general version of the hypothesis.)
See also Kaj’s FB post from this morning.
(Now also on LW.)
I don’t have a very strong guess here. I can imagine the world where being fully adaptation executer outcompetes deliberate strategy. It seems plausible that, at lower levels, adaptation-execution outcompetes strategy because people’s deliberate strategies just aren’t as good as copy-the-neighbors social wisdom.
But, to end up at the very top (even of a career bureaucrat ladder), you have to outcompete all the other people who were copying the neighbors / adaptation executing (I’m currently lumping these strategies together, not sure if that seems fair). It seems to me like this should require some kind of deliberate optimization, of some sort.
This phrasing feels overly strong – a world where 65% of the variance is distributed flinches/optimization but 35% personal optimization seems like the sort of world that might exist, in which personal optimization plays a significant role in who ends up at the top.
(Note that I’m basically arguing “the case for ‘there is some kind of deliberate optimization pressure at the top of bureaucracies’ seems higher than Zvi thinks it is”, not making any claims about “this seems most likely how it is”, or “the optimization at the top is altruistic.”)
One relevant question is how many smart, strategic, long-term optimizers exist. (In politics, vs. in business, or in academia, etc.)
E.g., if 1 in 3 people think this way at the start of their careers, that’s very different than if 1 in 1000 do, or 1 in 15. The rarer this way of thinking is, the more powerful it needs to be in order to overcome base rates.
I was also trying to think about the numbers, it felt important but I didn’t get anywhere with it. Where to start? Should I assume a base rate of 1 in a 1,000 people being strategic? More? Less?
I think that one natural way that you get to the top without being long-term strategic and being able to model reality, is nepotism. Some companies are literally handed down father-to-son, so the son doesn’t actually need to show great competence at the task. The Clinton families and the Bush families are also examples. There is some strategy here, but it isn’t strategy that’s closely connected to being competent at the job or even modeling reality that much (other than the social reality within the family).