I have found it! This was the one:
https://www.lesswrong.com/posts/qvNrmTqywWqYY8rsP/solutions-to-problems-with-bayesianism
Seems to have seen better reception at: https://forum.effectivealtruism.org/posts/3z9acGc5sspAdKenr/solutions-to-problems-with-bayesianism
The winning search strategy was quite interesting as well I think:
I took the history of all LW articles I have roughly ever read, I had easy access to all such titles and URLs, but not article contents. I fed them one by one into a 7B LLM asking it to rate how likely based on the title alone the unseen article content could match what I described above, as vague as that memory may be. Then I looked at the highest ranking candidates, and they were a dud. Did the same thing with a 70B model, et voila, the solution was near the top indeed.
Now I just need to re-read it if it was worth dredging up, I guess when a problem starts to itch it’s hard to resist solving it.
I think this is a nice write-up, let me add some nuance in two directions:
Indeed these are quick-and-dirty heuristics that can be subpar, but you may or may not be surprised just how often decisions don’t reach even this bar. In my work, when we are about to make a decision, I sometimes explicitly have to ask: do we have even a single reason to pick the option that we were about to pick over one or more others? And I find myself saying that (one of) those other options actually have reason(s) for us to pick them—I didn’t bring up the question for nothing after all.
In these cases I could argue that we upgraded from no-reason deciding to at least any-reason deciding. (If we even did, in some contexts it’s not unheard of that the answer to the above is something along the lines of “I cannot name any reasons but I still want to pick the first option.”)
This is how we can cross from lower sophistication to the middle. However, there are perils of going ever higher: once we have identified at least one set of opposing reasons, we cross into a regime that can be immensely costly: how to weigh reasons against each other, especially when people disagree. And I’d argue that people in general are quite bad at doing this, hence why this can take up a lot of resources and have results with questionable arbitrariness.
Of course this has to all balance in how important the decision even is and how much effort, if any, should be extended towards it. And I think humans are quite bad at judging this too but we do approximate it somewhat at least with large variance.
Thank you for naming these patterns!