Imagine that you have a classroom of children that you want to behave well… You could intervene at the level of the child’s individual thoughts. Police each thought, make sure it is a well-behaved-child thought.
I want to briefly point to a different relevant axis. Your framing is primarily about policing bad thoughts, and generally making the group stable and well-behaved. If I had a group of 30 people to command, while there are some ways I’d try to satisfice for every person (e.g. make sure they all learn a certain level of math, all have a certain ceiling on the trauma they experience in the year) I actually will put a lot of effort (perhaps >50% of my focus) into children achieving the biggest wins possible (e.g. getting one child to a state where they are deeply curious about some aspect of the world and are spending a lot of self-directed effort to better understand that phenomena, or two children getting very excited about building something and spending most of their time doing that well). The motivation here is that a single child growing up and making breakthrough discoveries in fundamental physics is something I will trade-off against a lot of days of many ‘well-behaved’ children.
But this is an abstract point, and it’s easy for people to talk past one another or create a double illusion of transparency when talking in ungrounded abstractions, so I’ll write another, much more concrete, comment.
I also want to mention, as Geoff indicates in the OP, that once you start looking on the time scale of months and years, I think motivation becomes an obvious factor. One way you can think of it is that you have to ask not merely whether this epistemic heuristic a good fit for a person’s environment, but also ask how likely the person is to consistently using the heuristic when it’s appropriate. Heuristics with a high effort-to-information ratio often wear a person out and they use them less and less.
I want to briefly point to a different relevant axis. Your framing is primarily about policing bad thoughts, and generally making the group stable and well-behaved. If I had a group of 30 people to command, while there are some ways I’d try to satisfice for every person (e.g. make sure they all learn a certain level of math, all have a certain ceiling on the trauma they experience in the year) I actually will put a lot of effort (perhaps >50% of my focus) into children achieving the biggest wins possible (e.g. getting one child to a state where they are deeply curious about some aspect of the world and are spending a lot of self-directed effort to better understand that phenomena, or two children getting very excited about building something and spending most of their time doing that well). The motivation here is that a single child growing up and making breakthrough discoveries in fundamental physics is something I will trade-off against a lot of days of many ‘well-behaved’ children.
But this is an abstract point, and it’s easy for people to talk past one another or create a double illusion of transparency when talking in ungrounded abstractions, so I’ll write another, much more concrete, comment.
I also want to mention, as Geoff indicates in the OP, that once you start looking on the time scale of months and years, I think motivation becomes an obvious factor. One way you can think of it is that you have to ask not merely whether this epistemic heuristic a good fit for a person’s environment, but also ask how likely the person is to consistently using the heuristic when it’s appropriate. Heuristics with a high effort-to-information ratio often wear a person out and they use them less and less.