I likewise appreciate your prompt and generous response :-)
I think I see how you imagine a hypothetical example of “no net health from insurance” might work as a filter that “passes” Hanson’s claim.
In this case, I don’t think your example works super well and might almost cause more problems that not?
Differences of detail in different people’s examples might SUBTRACT from attention to key facts relevant to a larger claim because people might propose different examples that hint at different larger causal models.
Like, if I was going to give the strongest possible hypothetical example to illustrate the basic idea of “no net health from insurance” I’d offers something like:
EXAMPLE: Alice has some minor symptoms of something that would clear up by itself and because she has health insurance she visits a doctor. (“Doctor visits” is one of the few things that health insurance strongly and reliably causes in many people.) While there she gets a nosocomial infection that is antibiotic resistant, lowering her life expectancy. This is more common than many people think. Done.
This example is quite different from your example. In your example medical treatment is good, and the key difference is basically just “pre-pay” vs “post-pay”.
(Also, neither of our examples covers the issue where many innovative medical treatments often lower mortality due to the disease they aim at while, somehow (accidentally?) RAISING all cause mortality...)
In my mind, the substantive big picture claim rests ultimately on the sum of many positive and negative factors, each of which arguably deserves “an example of its own”. (Things that raise my confidence quite a lot is often hearing the person’s own best argument AGAINST their own conclusion, and then hearing an adequate argument against that critique. I trust the winning mind quite a bit more when someone is of two minds.)
No example is going to JUSTIFIABLY convince me, and the LACK of an example for one or all of the important factors wouldn’t prevent me from being justifiably convinced by other methods that don’t route through “specific examples”.
ALSO: For that matter, I DO NOT ACTUALLY KNOW if Robin Hanson is actually right about medical insurance’s net results, in the past or now. I vaguely suspect that he is right, but I’m not strongly confident. Real answers might require studies that haven’t been performed? In the meantime I have insurance because “what if I get sick?!” and because “don’t be a weirdo”.
---
I think my key crux here has something to do with the rhetorical standards and conversational norms that “should” apply to various conversations between different kinds of people.
I assumed that having examples “ready-to-hand” (or offered early in a written argument) was something that you would actually be strongly in favor of (and below I’ll offer a steelman in defense of), but then you said:
I wouldn’t insist that he has an example “ready to hand during debate”; it’s okay if he says “if you want an example, here’s where we can pull one up”.
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
If someone said that TO ME I would experience it as a sort of a rhetorical “fuck you”… but WHAT a fuck you! {/me kisses her fingers} Then I would pump them for author recommendations!
My personal goal is often just to find out how the OTHER person feels they do their best thinking, run that process under emulation if I can, and then try to ask good questions from inside their frames. If they have lots of examples there’s a certain virtue to that… but I can think of other good signs of systematically productive thought.
---
If I was going to run “example based discussion” under emulation to try to help you understand my position, I would offer the example of John Hattie’s “Visible Learning”.
It is literally a meta-meta-analysis of education.
It spends the first two chapters just setting up the methodology and responding preemptively to quibbles that will predictable come when motivated thinkers (like classroom teachers that the theory says are teaching suboptimally) try to hear what Hattie has to say.
Chapter 3 finally lays out an abstract architecture of principles for good teaching, by talking about six relevant factors and connecting them all (very very abstractly and loosely) to: tight OODA loops (though not under that name) and Popperian epistemology (explicitly).
I’ll fully grant that it can take me an hour to read 5 pages of this book, and I’m stopping a lot and trying to imagine what Hattie might be saying at each step. The key point for me is that he’s not filling the book with examples, but with abstract empirically authoritative statistical claims about a complex and multi-faceted domain. It doesn’t feel like bullshit, it feels like extremely condensed wisdom.
Because of academic citation norms, in some sense his claims ultimately ground out in studies that are arguably “nothing BUT examples”? He’s trying to condense >800 meta-analyses that cover >50k actual studies that cover >1M observed children.
I could imagine you arguing that this proves how useful examples are, because his book is based on over a million examples, but he hasn’t talked about an example ONCE so far. He talks about methods and subjectively observed tendencies in meta-analyses mostly, trying to prepare the reader with a schema in which later results can land.
Plausibly, anyone could follow Hattie’s citations back to an interesting meta-analysis, look at its references, track back to a likely study, look in their methods section, and find their questionnaires, track back to the methods paper validating an the questionnaire, then look in the supplementary materials to get specific questionnaire items… Then someone could create an imaginary kid in their head who answered that questionnaire some way (like in the study) and then imagine them getting the outcome (like in the study) and use that scenario as “the example”?
I’m not doing that as I read the book. I trust that I could do the above, “because scholarship” but I’m not doing it. When I ask myself why, it seems like it is because it would make reading the (valuable seeming) book EVEN SLOWER?
---
I keep looping back in my mind to the idea that a lot of this strongly depends on which people are talking and what kinds of communication norms are even relevant, and I’m trying to find a place where I think I strongly agree with “looking for examples”...
It makes sense to me that, if I were in the role of an angel investor, and someone wanted $200k from me, and offered 10% of their 2-month-old garage/hobby project, then asking for examples of various of their business claims would be a good way to move forward.
They might not be good at causal modeling, or good at stats, or good at scholarship, or super verbal, but if they have a “native faculty” for building stuff, and budgeting, and building things that are actually useful to actual people… then probably the KEY capacities would be detectable as a head full of examples to various key questions that could be strongly dispositive.
Like… a head full of enough good examples could be sufficient for a basically neurotypical person to build a valuable company, especially if (1) they were examples that addressed key tactical/strategic questions, and (2) no intervening bad examples were ALSO in their head?
(Like if they had terrible examples of startup governance running around in their heads, these might eventually interfere with important parts of being a functional founder down the road. Detecting the inability to give bad examples seems naively hard to me...)
As an investor, I’d be VERY interested in “pre-loaded ready-to-hand theories” that seem likely to actually work. Examples are kinda like “pre-loaded ready-to-hand theories”? Possession of these theories in this form would be a good sign in terms of the founder’s readiness to execute very fast, which is a virtue in startups.
A LACK of ready-to-hand examples would suggest that even a good and feasible idea whose premises were “merely scientifically true” might not happen very fast if an angel funded it and the founder had to instantly start executing on it full time.
I would not be offended if you want to tap out. I feel like we haven’t found a crux yet. I think examples and specificity are interesting and useful and important, but I merely have intuitions about why, roughly like “duh, of course you need data to train a model”, not any high church formal theory with a fancy name that I can link to in wikipedia :-P
I agree that this section of your comment is the most cruxy:
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
Yes. Then I would say, “Ok, I’ve never encountered a coherent generalization for which I couldn’t easily generate an example, so go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
Anyone talking about having a “causal model” is probably beyond the level that my specificity trick is going to demolish. The specificity trick I focus on in this post is for demolishing the coherence of the claims of the average untrained arguer, or occasionally catching oneself at thinking overly vaguely. That’s it.
″...go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
I think maybe we agree… verbosely… with different emphasis? :-)
At least I think we could communicate reasonably well. I feel like the danger, if any, would arise from playing example ping pong and having the serious disagreements arise from how we “cook (instantiate?)” examples into models, and “uncook (generalize?)” models into examples.
When people just say what their model “actually is”, I really like it.
When people only point to instances I feel like the instances often under-determine the hypothetical underlying idea and leave me still confused as to how to generate novel instances for myself that they would assent to as predictions consistent with the idea that they “meant to mean” with the instances.
I likewise appreciate your prompt and generous response :-)
I think I see how you imagine a hypothetical example of “no net health from insurance” might work as a filter that “passes” Hanson’s claim.
In this case, I don’t think your example works super well and might almost cause more problems that not?
Differences of detail in different people’s examples might SUBTRACT from attention to key facts relevant to a larger claim because people might propose different examples that hint at different larger causal models.
Like, if I was going to give the strongest possible hypothetical example to illustrate the basic idea of “no net health from insurance” I’d offers something like:
This example is quite different from your example. In your example medical treatment is good, and the key difference is basically just “pre-pay” vs “post-pay”.
(Also, neither of our examples covers the issue where many innovative medical treatments often lower mortality due to the disease they aim at while, somehow (accidentally?) RAISING all cause mortality...)
In my mind, the substantive big picture claim rests ultimately on the sum of many positive and negative factors, each of which arguably deserves “an example of its own”. (Things that raise my confidence quite a lot is often hearing the person’s own best argument AGAINST their own conclusion, and then hearing an adequate argument against that critique. I trust the winning mind quite a bit more when someone is of two minds.)
No example is going to JUSTIFIABLY convince me, and the LACK of an example for one or all of the important factors wouldn’t prevent me from being justifiably convinced by other methods that don’t route through “specific examples”.
ALSO: For that matter, I DO NOT ACTUALLY KNOW if Robin Hanson is actually right about medical insurance’s net results, in the past or now. I vaguely suspect that he is right, but I’m not strongly confident. Real answers might require studies that haven’t been performed? In the meantime I have insurance because “what if I get sick?!” and because “don’t be a weirdo”.
---
I think my key crux here has something to do with the rhetorical standards and conversational norms that “should” apply to various conversations between different kinds of people.
I assumed that having examples “ready-to-hand” (or offered early in a written argument) was something that you would actually be strongly in favor of (and below I’ll offer a steelman in defense of), but then you said:
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
If someone said that TO ME I would experience it as a sort of a rhetorical “fuck you”… but WHAT a fuck you! {/me kisses her fingers} Then I would pump them for author recommendations!
My personal goal is often just to find out how the OTHER person feels they do their best thinking, run that process under emulation if I can, and then try to ask good questions from inside their frames. If they have lots of examples there’s a certain virtue to that… but I can think of other good signs of systematically productive thought.
---
If I was going to run “example based discussion” under emulation to try to help you understand my position, I would offer the example of John Hattie’s “Visible Learning”.
It is literally a meta-meta-analysis of education.
It spends the first two chapters just setting up the methodology and responding preemptively to quibbles that will predictable come when motivated thinkers (like classroom teachers that the theory says are teaching suboptimally) try to hear what Hattie has to say.
Chapter 3 finally lays out an abstract architecture of principles for good teaching, by talking about six relevant factors and connecting them all (very very abstractly and loosely) to: tight OODA loops (though not under that name) and Popperian epistemology (explicitly).
I’ll fully grant that it can take me an hour to read 5 pages of this book, and I’m stopping a lot and trying to imagine what Hattie might be saying at each step. The key point for me is that he’s not filling the book with examples, but with abstract empirically authoritative statistical claims about a complex and multi-faceted domain. It doesn’t feel like bullshit, it feels like extremely condensed wisdom.
Because of academic citation norms, in some sense his claims ultimately ground out in studies that are arguably “nothing BUT examples”? He’s trying to condense >800 meta-analyses that cover >50k actual studies that cover >1M observed children.
I could imagine you arguing that this proves how useful examples are, because his book is based on over a million examples, but he hasn’t talked about an example ONCE so far. He talks about methods and subjectively observed tendencies in meta-analyses mostly, trying to prepare the reader with a schema in which later results can land.
Plausibly, anyone could follow Hattie’s citations back to an interesting meta-analysis, look at its references, track back to a likely study, look in their methods section, and find their questionnaires, track back to the methods paper validating an the questionnaire, then look in the supplementary materials to get specific questionnaire items… Then someone could create an imaginary kid in their head who answered that questionnaire some way (like in the study) and then imagine them getting the outcome (like in the study) and use that scenario as “the example”?
I’m not doing that as I read the book. I trust that I could do the above, “because scholarship” but I’m not doing it. When I ask myself why, it seems like it is because it would make reading the (valuable seeming) book EVEN SLOWER?
---
I keep looping back in my mind to the idea that a lot of this strongly depends on which people are talking and what kinds of communication norms are even relevant, and I’m trying to find a place where I think I strongly agree with “looking for examples”...
It makes sense to me that, if I were in the role of an angel investor, and someone wanted $200k from me, and offered 10% of their 2-month-old garage/hobby project, then asking for examples of various of their business claims would be a good way to move forward.
They might not be good at causal modeling, or good at stats, or good at scholarship, or super verbal, but if they have a “native faculty” for building stuff, and budgeting, and building things that are actually useful to actual people… then probably the KEY capacities would be detectable as a head full of examples to various key questions that could be strongly dispositive.
Like… a head full of enough good examples could be sufficient for a basically neurotypical person to build a valuable company, especially if (1) they were examples that addressed key tactical/strategic questions, and (2) no intervening bad examples were ALSO in their head?
(Like if they had terrible examples of startup governance running around in their heads, these might eventually interfere with important parts of being a functional founder down the road. Detecting the inability to give bad examples seems naively hard to me...)
As an investor, I’d be VERY interested in “pre-loaded ready-to-hand theories” that seem likely to actually work. Examples are kinda like “pre-loaded ready-to-hand theories”? Possession of these theories in this form would be a good sign in terms of the founder’s readiness to execute very fast, which is a virtue in startups.
A LACK of ready-to-hand examples would suggest that even a good and feasible idea whose premises were “merely scientifically true” might not happen very fast if an angel funded it and the founder had to instantly start executing on it full time.
I would not be offended if you want to tap out. I feel like we haven’t found a crux yet. I think examples and specificity are interesting and useful and important, but I merely have intuitions about why, roughly like “duh, of course you need data to train a model”, not any high church formal theory with a fancy name that I can link to in wikipedia :-P
I agree that this section of your comment is the most cruxy:
Yes. Then I would say, “Ok, I’ve never encountered a coherent generalization for which I couldn’t easily generate an example, so go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
Anyone talking about having a “causal model” is probably beyond the level that my specificity trick is going to demolish. The specificity trick I focus on in this post is for demolishing the coherence of the claims of the average untrained arguer, or occasionally catching oneself at thinking overly vaguely. That’s it.
I think maybe we agree… verbosely… with different emphasis? :-)
At least I think we could communicate reasonably well. I feel like the danger, if any, would arise from playing example ping pong and having the serious disagreements arise from how we “cook (instantiate?)” examples into models, and “uncook (generalize?)” models into examples.
When people just say what their model “actually is”, I really like it.
When people only point to instances I feel like the instances often under-determine the hypothetical underlying idea and leave me still confused as to how to generate novel instances for myself that they would assent to as predictions consistent with the idea that they “meant to mean” with the instances.
Maybe: intensive theories > extensive theories?
Indeed