I agree with basically everything in the post, and especially that simple linear models are way undervalued. I’ve also come across cases where experts using literally 100x more data in there models get a worse outcome than other experts because they made a single bad assumption and didn’t sanity check it properly. And I’ve seen cases where someone builds a linear model on the reciprocal of the variable they should have used, or where they didn’t realize they were using a linear approximation of an exponential too far from the starting point. Modeling well is itself a skill that requires expertise and judgment. Other times, I see people build a simple linear model, which is built well, and then fail to notice or understand what it’s telling them.
There’s a Feynmann quote I love about talking simple models seriously:
As they’re telling me the conditions of the theorem, I construct something which fits all the conditions. You know, you have a set (one ball)—disjoint (two balls). Then the balls turn colors, grow hairs, or whatever, in my head as they put more conditions on. Finally they state the theorem, which is some dumb thing about the ball which isn’t true for my hairy green ball thing, so I say, ‘False!’
And a Wittgenstein quote about not thinking enough about what model predictions and observations imply:
“Tell me,” the great twentieth-century philosopher Ludwig Wittgenstein once asked a friend, “why do people always say it was natural for man to assume that the sun went around the Earth rather than that the Earth was rotating?” His friend replied, “Well, obviously because it just looks as though the Sun is going around the Earth.” Wittgenstein responded, “Well, what would it have looked like if it had looked as though the Earth was rotating?”
Literally last week I was at an event listening to an analyst from a major outlet that produces model-based reports that people pay a lot of money for. They were telling an audience of mostly VCs that their projections pretty much ignore the future impact of any technology that isn’t far enough along to have hard data. Like for energy, they have projections about nuclear, but exclude SMRs, and about hydrogen, but exclude synthetic hydrocarbons. Thankfully most of the room immediately understood (based on conversations I had later in the day) that this meant the model was guaranteed to be wrong in the most important cases, even though it looks like a strong, well-calibrated track record.
The solution to that, of course, is to put all the speculative possibilities in the model, weight them at zero for the modal case, and then do a sensitivity analysis. If your sensitivity analysis shows that simple linear models vary by multiple orders of magnitude in response to small changes in weights, well, that’s pretty important. But experts know if they publish models like that, most people will not read the reports carefully. They’ll skim, and cherry-pick, and misrepresent what you’re saying, and claim you’re trying to make yourself unfalsifiable. They’ll ignore the conditionality and probabilities of the different outcomes and just hear them all as “Well it could be any of these things.” I have definitely been subject to all of those, and at least once (when the error bars were >100x the most likely market size for a technology) chose not to publish the numerical outcomes of my model at all.
I agree with basically everything in the post, and especially that simple linear models are way undervalued. I’ve also come across cases where experts using literally 100x more data in there models get a worse outcome than other experts because they made a single bad assumption and didn’t sanity check it properly. And I’ve seen cases where someone builds a linear model on the reciprocal of the variable they should have used, or where they didn’t realize they were using a linear approximation of an exponential too far from the starting point. Modeling well is itself a skill that requires expertise and judgment. Other times, I see people build a simple linear model, which is built well, and then fail to notice or understand what it’s telling them.
There’s a Feynmann quote I love about talking simple models seriously:
And a Wittgenstein quote about not thinking enough about what model predictions and observations imply:
Literally last week I was at an event listening to an analyst from a major outlet that produces model-based reports that people pay a lot of money for. They were telling an audience of mostly VCs that their projections pretty much ignore the future impact of any technology that isn’t far enough along to have hard data. Like for energy, they have projections about nuclear, but exclude SMRs, and about hydrogen, but exclude synthetic hydrocarbons. Thankfully most of the room immediately understood (based on conversations I had later in the day) that this meant the model was guaranteed to be wrong in the most important cases, even though it looks like a strong, well-calibrated track record.
The solution to that, of course, is to put all the speculative possibilities in the model, weight them at zero for the modal case, and then do a sensitivity analysis. If your sensitivity analysis shows that simple linear models vary by multiple orders of magnitude in response to small changes in weights, well, that’s pretty important. But experts know if they publish models like that, most people will not read the reports carefully. They’ll skim, and cherry-pick, and misrepresent what you’re saying, and claim you’re trying to make yourself unfalsifiable. They’ll ignore the conditionality and probabilities of the different outcomes and just hear them all as “Well it could be any of these things.” I have definitely been subject to all of those, and at least once (when the error bars were >100x the most likely market size for a technology) chose not to publish the numerical outcomes of my model at all.