This shapes up as a case study on the dangers of doing very speculative and abstract theory about medium-term planning. (Which might include examples like figuring out what kind of understanding is necessary to actually apply hypothetical future alignment theory in practice...)
The problem is that common sense doesn’t work or doesn’t exist in these situations, but it’s still possible to do actionable planning, and massage the plan into a specific enough form in time to meet reality, so that reality goes according to the plan that on the side of the present adapts to it, even as on the side of the medium-term future it devolves into theoretical epicycles with no common sense propping it up.
This doesn’t go bad when it’s not in contact with reality, because then reality isn’t hurrying it into a form that doesn’t fit the emerging intuition of what the theory wants to be. And so it has time to mature into its own thing, or fade away into obscurity, but in any case there is more sanity to it formed of internal integrity. Whereas with a theoretical medium-term plan reality continually butchers the plan, which warps the theory, and human intuition is not good enough to reconcile the desiderata in a sensible way fast enough.
On the one hand, I think this is borderline-unintelligible as currently phrased? On the other hand, I think you have a decent point underneath it all.
Let me know if I’m following, while I try to rephrase it.
When insulated from real-world or outer-world incentives, a project can build up a lot of internal-logic and inferential distance by building upon itself repeatedly.
The incentives of insulated projects can be almost artificially-simple? So one can basically Goodhart, or massage data and assessment-metrics, to an incredible degree. This is sometimes done unconsciously.
When such a project finally comes into contact with reality, this can topple things at the very bottom of the structure, which everything else was built upon.
So for some heavily-insulated, heavily-built, and not-very-well-grounded projects, finally coming into exposure with reality can trigger a lot of warping/worldview-collapse/fallout in the immediate term.
(Ugh, I think I ended up borderline-incoherent myself. I might revisit and clean it up later.)
I think it’s worth keeping in mind that “common social reality” is itself sometimes one of these unstable/ungrounded top-heavy many-epicycles self-reinforcing collapses-when-reality-hits structures.
I am beyond-sick of the fights about whether something is “erroneous personal reality vs social reality” or “personal reality vs erroneous social reality,” so I’m going to leave simulating that out as an exercise for the reader.
loud sigh
Jumping meta, and skipping to the end.
Almost every elaborate worldview is built on at least some fragile low-level components, and might also have a few robustly-grounded builds in there, if you’re lucky.
“Some generalizable truth can be extracted” is more likely to occur, if there were incentives and pressure to generate robust builds.*
* (...God, I got a sudden wave of sympathy for anyone who views Capitalists and Rationalists as some form of creepy scavengers. There is a hint of truth in that lens. I hope we’re more like vultures than dogs; vultures have a way better “nutrition to parasite” ratio.)
By pure evolutionary logic: whichever thing adhered closer to common properties of base-reality, and/or was better-trained to generalize or self-update, will usually hold up better when some of its circumstances change. This tends to be part of what boils up when worldview conflicts and cataclysms play out.
I do see “better survival of a worldview across a range of circumstances” as somewhat predictive of attributes that I consider good-to-have in a worldview.
I also think surviving worldviews aren’t always the ones that make people the happiest, or allow people to thrive? Sometimes that sucks.
(If anyone wants to get into “everything is all equally-ungrounded social reality?” No. That doesn’t actually follow, even from the true statement that “everything you perceive goes through a lens.” I threw some quick commentary on that side-branch here, but I mostly think it’s off-topic.)
I don’t know, a lot of this is from discussion of Kuhn, new paradigms/worldviews are not necessarily incentivized to say new things or make sense of new things, even though they do, they just frame them in a particular way. And when something doesn’t fit a paradigm, it’s ignored. This is good and inevitable for theorizing on human level, and doesn’t inform usefulness or correctness of what’s going on, as these things live inside the paradigm.
It’s about lifecycle of theory development, confronted with incentives of medium-term planning. Humans are not very intelligent, and the way we can do abstract theory requires developing a lot of tools that enable fluency with it, including the actual intuitive fluency that uses the tools to think more rigorously, which is what I call common sense.
My anchor is math, which is the kind of theory I’m familiar with, but the topic of the theory could be things like social structures, research methodologies, or human rationality. So when common sense has an opportunity to form, we have a “post-rigorous” stage where rigid principles (gears) that make the theory lawful can be wielded intuitively. Without getting to this stage, the theory is blind or (potentially) insane. It is blind without intuition or insane when intuition is unmoored from rigor. (It can be somewhat sane when pre-rigorous intuition is grounded in something else, even if by informal analogy.)
If left alone, a theory tends to sanity. It develops principles to organize its intuitions, and develops intuitions to wield its principles. Eventually you get something real that can be seen and shaped with purpose.
But when it’s not at that stage, forcing it to change will keep it unsettled longer. If the theory opines about how an organizational medium-term plan works, what it should be, yet it’s unsettled, you’ll get insane opinions about the plans that shape insane plans. And reality chasing the plan, forcing it to confront what actually happens at present, gives an incentive to keep changing the theory before it’s ready, keeping it in this state of limbo.
This shapes up as a case study on the dangers of doing very speculative and abstract theory about medium-term planning. (Which might include examples like figuring out what kind of understanding is necessary to actually apply hypothetical future alignment theory in practice...)
The problem is that common sense doesn’t work or doesn’t exist in these situations, but it’s still possible to do actionable planning, and massage the plan into a specific enough form in time to meet reality, so that reality goes according to the plan that on the side of the present adapts to it, even as on the side of the medium-term future it devolves into theoretical epicycles with no common sense propping it up.
This doesn’t go bad when it’s not in contact with reality, because then reality isn’t hurrying it into a form that doesn’t fit the emerging intuition of what the theory wants to be. And so it has time to mature into its own thing, or fade away into obscurity, but in any case there is more sanity to it formed of internal integrity. Whereas with a theoretical medium-term plan reality continually butchers the plan, which warps the theory, and human intuition is not good enough to reconcile the desiderata in a sensible way fast enough.
On the one hand, I think this is borderline-unintelligible as currently phrased? On the other hand, I think you have a decent point underneath it all.
Let me know if I’m following, while I try to rephrase it.
When insulated from real-world or outer-world incentives, a project can build up a lot of internal-logic and inferential distance by building upon itself repeatedly.
The incentives of insulated projects can be almost artificially-simple? So one can basically Goodhart, or massage data and assessment-metrics, to an incredible degree. This is sometimes done unconsciously.
When such a project finally comes into contact with reality, this can topple things at the very bottom of the structure, which everything else was built upon.
So for some heavily-insulated, heavily-built, and not-very-well-grounded projects, finally coming into exposure with reality can trigger a lot of warping/worldview-collapse/fallout in the immediate term.
Now to actually comment...
(Ugh, I think I ended up borderline-incoherent myself. I might revisit and clean it up later.)
I think it’s worth keeping in mind that “common social reality” is itself sometimes one of these unstable/ungrounded top-heavy many-epicycles self-reinforcing collapses-when-reality-hits structures.
I am beyond-sick of the fights about whether something is “erroneous personal reality vs social reality” or “personal reality vs erroneous social reality,” so I’m going to leave simulating that out as an exercise for the reader.
loud sigh
Jumping meta, and skipping to the end.
Almost every elaborate worldview is built on at least some fragile low-level components, and might also have a few robustly-grounded builds in there, if you’re lucky.
“Some generalizable truth can be extracted” is more likely to occur, if there were incentives and pressure to generate robust builds.*
* (...God, I got a sudden wave of sympathy for anyone who views Capitalists and Rationalists as some form of creepy scavengers. There is a hint of truth in that lens. I hope we’re more like vultures than dogs; vultures have a way better “nutrition to parasite” ratio.)
By pure evolutionary logic: whichever thing adhered closer to common properties of base-reality, and/or was better-trained to generalize or self-update, will usually hold up better when some of its circumstances change. This tends to be part of what boils up when worldview conflicts and cataclysms play out.
I do see “better survival of a worldview across a range of circumstances” as somewhat predictive of attributes that I consider good-to-have in a worldview.
I also think surviving worldviews aren’t always the ones that make people the happiest, or allow people to thrive? Sometimes that sucks.
(If anyone wants to get into “everything is all equally-ungrounded social reality?” No. That doesn’t actually follow, even from the true statement that “everything you perceive goes through a lens.” I threw some quick commentary on that side-branch here, but I mostly think it’s off-topic.)
I don’t know, a lot of this is from discussion of Kuhn, new paradigms/worldviews are not necessarily incentivized to say new things or make sense of new things, even though they do, they just frame them in a particular way. And when something doesn’t fit a paradigm, it’s ignored. This is good and inevitable for theorizing on human level, and doesn’t inform usefulness or correctness of what’s going on, as these things live inside the paradigm.
It’s about lifecycle of theory development, confronted with incentives of medium-term planning. Humans are not very intelligent, and the way we can do abstract theory requires developing a lot of tools that enable fluency with it, including the actual intuitive fluency that uses the tools to think more rigorously, which is what I call common sense.
My anchor is math, which is the kind of theory I’m familiar with, but the topic of the theory could be things like social structures, research methodologies, or human rationality. So when common sense has an opportunity to form, we have a “post-rigorous” stage where rigid principles (gears) that make the theory lawful can be wielded intuitively. Without getting to this stage, the theory is blind or (potentially) insane. It is blind without intuition or insane when intuition is unmoored from rigor. (It can be somewhat sane when pre-rigorous intuition is grounded in something else, even if by informal analogy.)
If left alone, a theory tends to sanity. It develops principles to organize its intuitions, and develops intuitions to wield its principles. Eventually you get something real that can be seen and shaped with purpose.
But when it’s not at that stage, forcing it to change will keep it unsettled longer. If the theory opines about how an organizational medium-term plan works, what it should be, yet it’s unsettled, you’ll get insane opinions about the plans that shape insane plans. And reality chasing the plan, forcing it to confront what actually happens at present, gives an incentive to keep changing the theory before it’s ready, keeping it in this state of limbo.