I have a strong appreciation for the general point that “specificity is sometimes really great”, but I’m wondering if this point might miss the forest for the trees with some large portion of its actual audience?
If you buy that in some sense all debates are bravery debates then audience can matter a lot, and perhaps this point addresses central tendencies in “global english internet discourse” while failing to address central tendencies on LW?
There is a sense in which nearly all highly general statements are technically false, because they admit of at least some counter examples.
However any such statement might still be a useful in a structured argument of very high quality, perhaps as an illustration of a troubling central tendency, or a “lemma” in a multi-part probabalistic argument.
It might even be the case that the MEDIAN EXAMPLE of a real tendency is highly imperfect without that “demolishing” the point.
Suppose for example that someone has focused on a lot on higher level structural truths whose evidential basis was, say, a thorough exploration of many meta-analyses about a given subject.
“Mel the meta-meta-analyst” might be communicating summary claims that are important and generally true that “Sophia the specificity demander” might rhetorically “win against” in a way that does not structurally correspond to the central tendencies of the actual world.
Mel might know things about medical practice without ever having treated a patient or even talked to a single doctor or nurse. Mel might understand something about how classrooms work without being a teacher or ever having visited a classroom. Mel might know things about the behavior of congressional representatives without ever working as a congressional staffer. If forced to confabulate an exemplar patient, or exemplar classroom, or an exemplar political representative the details might be easy to challenge even as a claim about the central tendencies is correct.
Naively, I would think that for Mel to be justified in his claims (even WITHOUT having exemplars ready-to-hand during debate) Mel might need to be moderately scrupulous in his collection of meta-analytic data, and know enough about statistics to include and exclude studies or meta-analyses in appropriately weighed ways. Perhaps he would also need to be good at assessing the character of authors and scientists to be able to predict which ones are outright faking their data, or using incredibly sloppy data collection?
The core point here is that Sophia might not be lead to the truth SIMPLY by demanding specificity without regard to the nature of the claims of her interlocutor.
If Sophia thinks this tactic gives her “the POWER to DEMOLISH arguments” in full generality, that might not actually be true, and it might even lower the quality of her beliefs over time, especially if she mostly converses with smart people (worth learning from, in their area(s) of expertise) rather than idiots (nearly all of whose claims might perhaps be worth demolishing on average).
It is totally possible that some people are just confused and wrong (as, indeed, many people seem to be, on many topics… which is OK because ignorance is the default and there is more information in the world now than any human can integrate within a lifetime of study). In that case, demanding specificity to demolish confused and wrong arguments might genuinely and helpfully debug many low quality abstract claims.
However, I think there’s a lot to be said for first asking someone about the positive rigorous basis of any new claim, to see if the person who brought it up can articulate a constructive epistemic strategy.
If they have a constructive epistemic strategy that doesn’t rely on personal knowledge of specific details, that would be reasonable, because I think such things ARE possible.
A culturally local example might be Hanson’s general claim that medical insurance coverage does not appear to “cause health” on average. No single vivid patient generates this result. Vivid stories do exist here, but they don’t adequately justify the broader claim. Rather, the substantiation arises from tallying many outcomes in a variety of circumstances and empirically noticing relations between circumstances and tallies.
If I was asked to offer a single specific positive example of “general arguments being worthwhile” I might nominate Visible Learning by John Hattie as a fascinating and extremely abstract synthesis of >1M students participating in >50k studies of K-12 learning. In this case a core claim of the book is that mindless teaching happens sometimes, nearly all mindful attempts to improve things work a bit, and very rarely a large number of things “go right” and unusually large effect sizes can be observed. I’ve never seen one of these ideal classrooms I think, but the arguments that they have a collection of general characteristics seem solid so far.
Maybe I’ll change my mind by the end? I’m still in progress on this particular book, which makes it sort of “top of mind” for me, but the lack of specifics in the book present a readability challenge rather than an epistemic challenge ;-P
The book Made to Stick, by contrast, uses Stories that are Simple, Surprising, Emotional, Concrete, and Credible to argue that the best way to convince people of something is to tell them Stories that are Simple, Surprising, Emotional, Concrete, and Credible.
As near as I can tell, Made to Stick describes how to convince people of things whether or not the thing is true, which means that if these techniques work (and can in fact cause many false ideas to spread through speech communities with low epistemic hygiene, which the book arguably did not really “establish”) then a useful epistemic heuristic might be to give a small evidential PENALTY to all claims illustrated merely via vivid example.
I guess one thing I would like to say here at the end is that I mean this comment in a positive spirit. I upvoted this article and the previous one, and if the rest of the sequence has similar quality I will upvote those as well.
I’m generally IN FAVOR of writing imperfect things and then unpacking and discussing them. This is a better than median post in my opinion, and deserved discussion, rather than deserving to be ignored :-)
A culturally local example might be Hanson’s general claim that medical insurance coverage does not appear to “cause health” on average. No single vivid patient generates this result. Vivid stories do exist here, but they don’t adequately justify the broader claim. Rather, the substantiation arises from tallying many outcomes in a variety of circumstances and empirically noticing relations between circumstances and tallies.
I don’t see how this is an example of a time when my specificity test shouldn’t be used, because Robin Hanson would simply pass my specificity test. It’s safe to say that Robin has thought through at least one specific example of what the claim that “medical insurance doesn’t cause health” means. The sample dialogue with me and Robin would look like this:
Robin: Medical insurance coverage doesn’t cause health on average!
Liron: What’s a specific example (real or hypothetical) of someone who seeks medical insurance coverage because they think they’re improving their health outcome, but who actually would have had the same health outcome without insurance?
Robin: A 50-year-old man opts for a job that has insurance benefits over one that doesn’t, because he believes it will improve his expected health outcome. For the next 10 years, he’s healthy, so the insurance didn’t improve his outcome there, but he wouldn’t expect it to. Then at age 60, he gets a heart attack and goes to the hospital and gets double bypass surgery. But in the hypothetical where this same person hadn’t had health insurance, he would have had that same bypass surgery anyway, and then paid off the cost over the next few years. And then later he dies of old age. The data shows that this example I made up is a representative one—definitely not in every sense, but just in the sense that there’s no delta between health outcomes in counterfactual world-states where people either have or don’t have health insurance.
Liron: Okay, I guess your claim is coherent. I have no idea if it’s true or false, but I’m ready to start hearing your evidence, now that you’ve cleared this low bar of having a coherent claim.
Robin: It was such an easy exercise though, it seemed like a pointless formality.
Liron: Right. The exercise was designed to be the kind of basic roadblock that’s a pointless formality for Robin Hanson while being surprisingly difficult for the majority of humans.
Mel might know things about medical practice without ever having treated a patient or even talked to a single doctor or nurse. Mel might understand something about how classrooms work without being a teacher or ever having visited a classroom. Mel might know things about the behavior of congressional representatives without ever working as a congressional staffer. If forced to confabulate an exemplar patient, or exemplar classroom, or an exemplar political representative the details might be easy to challenge even as a claim about the central tendencies is correct.
Since I don’t think the Robin Hanson example was enough to change my mind, is there another example of a general claim Mel might make where we can agree he’s fundamentally right to make that claim but shouldn’t expect him to furnish a specific example of it?
If he’s been reading meta-analyses, couldn’t he just follow their references to analyses that contain specific object-level experiments that were done, and furnish those as his examples?
I wouldn’t insist that he has an example “ready to hand during debate”; it’s okay if he says “if you want an example, here’s where we can pull one up”. I agree we should be careful to make sure that the net effect of asking for examples is to raise the level of discourse, but I don’t think it’s a hard problem.
I appreciate your high-quality comment. What I’ve done in this comment, obviously, is activated my specificity powers. I won’t claim that I’ve demolished your argument, I just hope readers will agree that my approach was fair, respectful, productive, and not unnecessarily “adversarial”!
I likewise appreciate your prompt and generous response :-)
I think I see how you imagine a hypothetical example of “no net health from insurance” might work as a filter that “passes” Hanson’s claim.
In this case, I don’t think your example works super well and might almost cause more problems that not?
Differences of detail in different people’s examples might SUBTRACT from attention to key facts relevant to a larger claim because people might propose different examples that hint at different larger causal models.
Like, if I was going to give the strongest possible hypothetical example to illustrate the basic idea of “no net health from insurance” I’d offers something like:
EXAMPLE: Alice has some minor symptoms of something that would clear up by itself and because she has health insurance she visits a doctor. (“Doctor visits” is one of the few things that health insurance strongly and reliably causes in many people.) While there she gets a nosocomial infection that is antibiotic resistant, lowering her life expectancy. This is more common than many people think. Done.
This example is quite different from your example. In your example medical treatment is good, and the key difference is basically just “pre-pay” vs “post-pay”.
(Also, neither of our examples covers the issue where many innovative medical treatments often lower mortality due to the disease they aim at while, somehow (accidentally?) RAISING all cause mortality...)
In my mind, the substantive big picture claim rests ultimately on the sum of many positive and negative factors, each of which arguably deserves “an example of its own”. (Things that raise my confidence quite a lot is often hearing the person’s own best argument AGAINST their own conclusion, and then hearing an adequate argument against that critique. I trust the winning mind quite a bit more when someone is of two minds.)
No example is going to JUSTIFIABLY convince me, and the LACK of an example for one or all of the important factors wouldn’t prevent me from being justifiably convinced by other methods that don’t route through “specific examples”.
ALSO: For that matter, I DO NOT ACTUALLY KNOW if Robin Hanson is actually right about medical insurance’s net results, in the past or now. I vaguely suspect that he is right, but I’m not strongly confident. Real answers might require studies that haven’t been performed? In the meantime I have insurance because “what if I get sick?!” and because “don’t be a weirdo”.
---
I think my key crux here has something to do with the rhetorical standards and conversational norms that “should” apply to various conversations between different kinds of people.
I assumed that having examples “ready-to-hand” (or offered early in a written argument) was something that you would actually be strongly in favor of (and below I’ll offer a steelman in defense of), but then you said:
I wouldn’t insist that he has an example “ready to hand during debate”; it’s okay if he says “if you want an example, here’s where we can pull one up”.
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
If someone said that TO ME I would experience it as a sort of a rhetorical “fuck you”… but WHAT a fuck you! {/me kisses her fingers} Then I would pump them for author recommendations!
My personal goal is often just to find out how the OTHER person feels they do their best thinking, run that process under emulation if I can, and then try to ask good questions from inside their frames. If they have lots of examples there’s a certain virtue to that… but I can think of other good signs of systematically productive thought.
---
If I was going to run “example based discussion” under emulation to try to help you understand my position, I would offer the example of John Hattie’s “Visible Learning”.
It is literally a meta-meta-analysis of education.
It spends the first two chapters just setting up the methodology and responding preemptively to quibbles that will predictable come when motivated thinkers (like classroom teachers that the theory says are teaching suboptimally) try to hear what Hattie has to say.
Chapter 3 finally lays out an abstract architecture of principles for good teaching, by talking about six relevant factors and connecting them all (very very abstractly and loosely) to: tight OODA loops (though not under that name) and Popperian epistemology (explicitly).
I’ll fully grant that it can take me an hour to read 5 pages of this book, and I’m stopping a lot and trying to imagine what Hattie might be saying at each step. The key point for me is that he’s not filling the book with examples, but with abstract empirically authoritative statistical claims about a complex and multi-faceted domain. It doesn’t feel like bullshit, it feels like extremely condensed wisdom.
Because of academic citation norms, in some sense his claims ultimately ground out in studies that are arguably “nothing BUT examples”? He’s trying to condense >800 meta-analyses that cover >50k actual studies that cover >1M observed children.
I could imagine you arguing that this proves how useful examples are, because his book is based on over a million examples, but he hasn’t talked about an example ONCE so far. He talks about methods and subjectively observed tendencies in meta-analyses mostly, trying to prepare the reader with a schema in which later results can land.
Plausibly, anyone could follow Hattie’s citations back to an interesting meta-analysis, look at its references, track back to a likely study, look in their methods section, and find their questionnaires, track back to the methods paper validating an the questionnaire, then look in the supplementary materials to get specific questionnaire items… Then someone could create an imaginary kid in their head who answered that questionnaire some way (like in the study) and then imagine them getting the outcome (like in the study) and use that scenario as “the example”?
I’m not doing that as I read the book. I trust that I could do the above, “because scholarship” but I’m not doing it. When I ask myself why, it seems like it is because it would make reading the (valuable seeming) book EVEN SLOWER?
---
I keep looping back in my mind to the idea that a lot of this strongly depends on which people are talking and what kinds of communication norms are even relevant, and I’m trying to find a place where I think I strongly agree with “looking for examples”...
It makes sense to me that, if I were in the role of an angel investor, and someone wanted $200k from me, and offered 10% of their 2-month-old garage/hobby project, then asking for examples of various of their business claims would be a good way to move forward.
They might not be good at causal modeling, or good at stats, or good at scholarship, or super verbal, but if they have a “native faculty” for building stuff, and budgeting, and building things that are actually useful to actual people… then probably the KEY capacities would be detectable as a head full of examples to various key questions that could be strongly dispositive.
Like… a head full of enough good examples could be sufficient for a basically neurotypical person to build a valuable company, especially if (1) they were examples that addressed key tactical/strategic questions, and (2) no intervening bad examples were ALSO in their head?
(Like if they had terrible examples of startup governance running around in their heads, these might eventually interfere with important parts of being a functional founder down the road. Detecting the inability to give bad examples seems naively hard to me...)
As an investor, I’d be VERY interested in “pre-loaded ready-to-hand theories” that seem likely to actually work. Examples are kinda like “pre-loaded ready-to-hand theories”? Possession of these theories in this form would be a good sign in terms of the founder’s readiness to execute very fast, which is a virtue in startups.
A LACK of ready-to-hand examples would suggest that even a good and feasible idea whose premises were “merely scientifically true” might not happen very fast if an angel funded it and the founder had to instantly start executing on it full time.
I would not be offended if you want to tap out. I feel like we haven’t found a crux yet. I think examples and specificity are interesting and useful and important, but I merely have intuitions about why, roughly like “duh, of course you need data to train a model”, not any high church formal theory with a fancy name that I can link to in wikipedia :-P
I agree that this section of your comment is the most cruxy:
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
Yes. Then I would say, “Ok, I’ve never encountered a coherent generalization for which I couldn’t easily generate an example, so go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
Anyone talking about having a “causal model” is probably beyond the level that my specificity trick is going to demolish. The specificity trick I focus on in this post is for demolishing the coherence of the claims of the average untrained arguer, or occasionally catching oneself at thinking overly vaguely. That’s it.
″...go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
I think maybe we agree… verbosely… with different emphasis? :-)
At least I think we could communicate reasonably well. I feel like the danger, if any, would arise from playing example ping pong and having the serious disagreements arise from how we “cook (instantiate?)” examples into models, and “uncook (generalize?)” models into examples.
When people just say what their model “actually is”, I really like it.
When people only point to instances I feel like the instances often under-determine the hypothetical underlying idea and leave me still confused as to how to generate novel instances for myself that they would assent to as predictions consistent with the idea that they “meant to mean” with the instances.
I feel like the same scrutinity standard is not being applied. Guy with health insurance doesn’t check their health more often catching diseases earlier? Uncertainty doesn’t cause stress and workload on circulatory system? Why are these not holes that prevent it from being coherent? Why can’t Steve claim he has a friend that can be called that can exempilify exploitation?
If the bar is infact low Steve passed it upon positing McDonalds as relevant alternative and the argument went on to actually argue the argument. Or alternatively it requires opinion to have that Robin specification to be coherent and a reasonable arguer could try to hold it to be incoherent.
I feel like this is a case where epistemic status breaks symmetry. A white coat doctor and a witch doctor making the same claims requires the witch doctor to show more evidence to reach the same credibility levels. If argument truly screens off authority the problems needs to be in the argument. Steve is required to have the specification ready on hand during debate.
The difference is that we all have a much clearer picture of what Robin Hanson’s claim means than what Steve means, so Robin’s claim is sufficiently coherent and Steve’s isn’t. I agree there’s a gray area on what is “sufficiently coherent”, but I think we can agree there’s a significant difference on this coherence spectrum between Steve’s claim and Robin’s claim.
For example, any listener can reasonably infer from Robin’s claim that someone on medical insurance who gets cancer shouldn’t be expected to survive with a higher probability than someone without medical insurance. But reasonable listeners can have differing guesses about whether or not Steve’s claim would also describe a world where Uber institutes a $15/hr flat hourly wage for all drivers.
Why can’t Steve claim he has a friend that can be called that can exempilify exploitation?
Sure, then I’d just try to understand what specific property of the friend counts as exploitation. In Robin’s case, I already have good reasonable guesses about the operational definitions of “health”. Yes I can try to play “gotcha” and argue that Robin hasn’t nailed down his claim, and in some cases that might actually be the right thing to do—it’s up to me to determine what’s sufficiently coherent for my own understanding, and nail down the claim to that standard, before moving onto arguing about the truth-value of the claim.
If the bar is infact low Steve passed it upon positing McDonalds as relevant alternative and the argument went on to actually argue the argument
Ah, if Steve really meant “Yes Uber screws the employee out of $1/hr compared to McDonald’s”, then he would have passed the specificity bar. But the specific Steve I portray in the dialogue didn’t pass the bar because that’s not really the point he felt he wanted to make. The Steve that I portray found himself confused because his claim really was insufficiently specific, and therefore really was demolished, unexpectedly to him.
Well I am more familiar with settings where I have a duty to understand the world rather than the world having the duty to explain itself to me. I also hold that having unfamiliar things hit higher standards creates epistemic xenophobia. I would hold it important that one doesn’t assign falsehood to a claim they don’t understand. Althought it is also true that assigning truth to a claim one doesn’t understand is dangerous to relatively same caliber.
My go-to assumption would be that Steve understands something different with the word and might be running some sort of moon logic in his head. Rather than declare the “moon proof” to be invalid it’s more important that the translation between moon logic and my planet logic interfaces without confusion. Instead of using a word/concept I do know wrong he is using a word or concept I do not know.
“Coherent” usually points to a concept where a sentence is judged on it’s home logics terms. But as used here it’s clearly in the eye of the beholder. So it’s less “makes objective sense” and more a “makes sense to whom?”. The shared reality you create in a discussion or debate would be the arbiter but if the argument realies too much on those mechanics it doesn’t generalise to contextes outside of that.
I also just think there are a lot of Steves in the world who are holding on to belief-claims that lack specific referents, who could benefit from reading this post even if no one is arguing with them.
Related: it’s tempting to interpret Ignorance, a skilled practiceas pushing for the epistemological stance that specificity should overwhelm Mel the meta meta-analyst
There is a sense in which nearly all highly general statements are technically false, because they admit of at least some counter examples.
I think this is a common misunderstanding that people are having about what I’m saying. I’m not saying to hunt for a counterexample that demolishes a claim. I’m saying to ask the person making the claim for a single specific example that’s consistent with the general claim.
Imagine that a general claim has 900 examples and 100 counterexamples. Then I’m just asking to see one of the 900 examples :)
I have a strong appreciation for the general point that “specificity is sometimes really great”, but I’m wondering if this point might miss the forest for the trees with some large portion of its actual audience?
If you buy that in some sense all debates are bravery debates then audience can matter a lot, and perhaps this point addresses central tendencies in “global english internet discourse” while failing to address central tendencies on LW?
There is a sense in which nearly all highly general statements are technically false, because they admit of at least some counter examples.
However any such statement might still be a useful in a structured argument of very high quality, perhaps as an illustration of a troubling central tendency, or a “lemma” in a multi-part probabalistic argument.
It might even be the case that the MEDIAN EXAMPLE of a real tendency is highly imperfect without that “demolishing” the point.
Suppose for example that someone has focused on a lot on higher level structural truths whose evidential basis was, say, a thorough exploration of many meta-analyses about a given subject.
“Mel the meta-meta-analyst” might be communicating summary claims that are important and generally true that “Sophia the specificity demander” might rhetorically “win against” in a way that does not structurally correspond to the central tendencies of the actual world.
Mel might know things about medical practice without ever having treated a patient or even talked to a single doctor or nurse. Mel might understand something about how classrooms work without being a teacher or ever having visited a classroom. Mel might know things about the behavior of congressional representatives without ever working as a congressional staffer. If forced to confabulate an exemplar patient, or exemplar classroom, or an exemplar political representative the details might be easy to challenge even as a claim about the central tendencies is correct.
Naively, I would think that for Mel to be justified in his claims (even WITHOUT having exemplars ready-to-hand during debate) Mel might need to be moderately scrupulous in his collection of meta-analytic data, and know enough about statistics to include and exclude studies or meta-analyses in appropriately weighed ways. Perhaps he would also need to be good at assessing the character of authors and scientists to be able to predict which ones are outright faking their data, or using incredibly sloppy data collection?
The core point here is that Sophia might not be lead to the truth SIMPLY by demanding specificity without regard to the nature of the claims of her interlocutor.
If Sophia thinks this tactic gives her “the POWER to DEMOLISH arguments” in full generality, that might not actually be true, and it might even lower the quality of her beliefs over time, especially if she mostly converses with smart people (worth learning from, in their area(s) of expertise) rather than idiots (nearly all of whose claims might perhaps be worth demolishing on average).
It is totally possible that some people are just confused and wrong (as, indeed, many people seem to be, on many topics… which is OK because ignorance is the default and there is more information in the world now than any human can integrate within a lifetime of study). In that case, demanding specificity to demolish confused and wrong arguments might genuinely and helpfully debug many low quality abstract claims.
However, I think there’s a lot to be said for first asking someone about the positive rigorous basis of any new claim, to see if the person who brought it up can articulate a constructive epistemic strategy.
If they have a constructive epistemic strategy that doesn’t rely on personal knowledge of specific details, that would be reasonable, because I think such things ARE possible.
A culturally local example might be Hanson’s general claim that medical insurance coverage does not appear to “cause health” on average. No single vivid patient generates this result. Vivid stories do exist here, but they don’t adequately justify the broader claim. Rather, the substantiation arises from tallying many outcomes in a variety of circumstances and empirically noticing relations between circumstances and tallies.
If I was asked to offer a single specific positive example of “general arguments being worthwhile” I might nominate Visible Learning by John Hattie as a fascinating and extremely abstract synthesis of >1M students participating in >50k studies of K-12 learning. In this case a core claim of the book is that mindless teaching happens sometimes, nearly all mindful attempts to improve things work a bit, and very rarely a large number of things “go right” and unusually large effect sizes can be observed. I’ve never seen one of these ideal classrooms I think, but the arguments that they have a collection of general characteristics seem solid so far.
Maybe I’ll change my mind by the end? I’m still in progress on this particular book, which makes it sort of “top of mind” for me, but the lack of specifics in the book present a readability challenge rather than an epistemic challenge ;-P
The book Made to Stick, by contrast, uses Stories that are Simple, Surprising, Emotional, Concrete, and Credible to argue that the best way to convince people of something is to tell them Stories that are Simple, Surprising, Emotional, Concrete, and Credible.
As near as I can tell, Made to Stick describes how to convince people of things whether or not the thing is true, which means that if these techniques work (and can in fact cause many false ideas to spread through speech communities with low epistemic hygiene, which the book arguably did not really “establish”) then a useful epistemic heuristic might be to give a small evidential PENALTY to all claims illustrated merely via vivid example.
I guess one thing I would like to say here at the end is that I mean this comment in a positive spirit. I upvoted this article and the previous one, and if the rest of the sequence has similar quality I will upvote those as well.
I’m generally IN FAVOR of writing imperfect things and then unpacking and discussing them. This is a better than median post in my opinion, and deserved discussion, rather than deserving to be ignored :-)
I don’t see how this is an example of a time when my specificity test shouldn’t be used, because Robin Hanson would simply pass my specificity test. It’s safe to say that Robin has thought through at least one specific example of what the claim that “medical insurance doesn’t cause health” means. The sample dialogue with me and Robin would look like this:
Robin: Medical insurance coverage doesn’t cause health on average!
Liron: What’s a specific example (real or hypothetical) of someone who seeks medical insurance coverage because they think they’re improving their health outcome, but who actually would have had the same health outcome without insurance?
Robin: A 50-year-old man opts for a job that has insurance benefits over one that doesn’t, because he believes it will improve his expected health outcome. For the next 10 years, he’s healthy, so the insurance didn’t improve his outcome there, but he wouldn’t expect it to. Then at age 60, he gets a heart attack and goes to the hospital and gets double bypass surgery. But in the hypothetical where this same person hadn’t had health insurance, he would have had that same bypass surgery anyway, and then paid off the cost over the next few years. And then later he dies of old age. The data shows that this example I made up is a representative one—definitely not in every sense, but just in the sense that there’s no delta between health outcomes in counterfactual world-states where people either have or don’t have health insurance.
Liron: Okay, I guess your claim is coherent. I have no idea if it’s true or false, but I’m ready to start hearing your evidence, now that you’ve cleared this low bar of having a coherent claim.
Robin: It was such an easy exercise though, it seemed like a pointless formality.
Liron: Right. The exercise was designed to be the kind of basic roadblock that’s a pointless formality for Robin Hanson while being surprisingly difficult for the majority of humans.
Since I don’t think the Robin Hanson example was enough to change my mind, is there another example of a general claim Mel might make where we can agree he’s fundamentally right to make that claim but shouldn’t expect him to furnish a specific example of it?
If he’s been reading meta-analyses, couldn’t he just follow their references to analyses that contain specific object-level experiments that were done, and furnish those as his examples?
I wouldn’t insist that he has an example “ready to hand during debate”; it’s okay if he says “if you want an example, here’s where we can pull one up”. I agree we should be careful to make sure that the net effect of asking for examples is to raise the level of discourse, but I don’t think it’s a hard problem.
I appreciate your high-quality comment. What I’ve done in this comment, obviously, is activated my specificity powers. I won’t claim that I’ve demolished your argument, I just hope readers will agree that my approach was fair, respectful, productive, and not unnecessarily “adversarial”!
I likewise appreciate your prompt and generous response :-)
I think I see how you imagine a hypothetical example of “no net health from insurance” might work as a filter that “passes” Hanson’s claim.
In this case, I don’t think your example works super well and might almost cause more problems that not?
Differences of detail in different people’s examples might SUBTRACT from attention to key facts relevant to a larger claim because people might propose different examples that hint at different larger causal models.
Like, if I was going to give the strongest possible hypothetical example to illustrate the basic idea of “no net health from insurance” I’d offers something like:
This example is quite different from your example. In your example medical treatment is good, and the key difference is basically just “pre-pay” vs “post-pay”.
(Also, neither of our examples covers the issue where many innovative medical treatments often lower mortality due to the disease they aim at while, somehow (accidentally?) RAISING all cause mortality...)
In my mind, the substantive big picture claim rests ultimately on the sum of many positive and negative factors, each of which arguably deserves “an example of its own”. (Things that raise my confidence quite a lot is often hearing the person’s own best argument AGAINST their own conclusion, and then hearing an adequate argument against that critique. I trust the winning mind quite a bit more when someone is of two minds.)
No example is going to JUSTIFIABLY convince me, and the LACK of an example for one or all of the important factors wouldn’t prevent me from being justifiably convinced by other methods that don’t route through “specific examples”.
ALSO: For that matter, I DO NOT ACTUALLY KNOW if Robin Hanson is actually right about medical insurance’s net results, in the past or now. I vaguely suspect that he is right, but I’m not strongly confident. Real answers might require studies that haven’t been performed? In the meantime I have insurance because “what if I get sick?!” and because “don’t be a weirdo”.
---
I think my key crux here has something to do with the rhetorical standards and conversational norms that “should” apply to various conversations between different kinds of people.
I assumed that having examples “ready-to-hand” (or offered early in a written argument) was something that you would actually be strongly in favor of (and below I’ll offer a steelman in defense of), but then you said:
So for me it would ALSO BE OK to say “If you want an example I’m sorry. I can’t think of one right now. As a rule, I don’t think in terms of fictional stories. I put effort into thinking in terms of causal models and measurables and authors with axes to grind and bridging theories and studies that rule out causal models and what observations I’d expect from differently weighed ensembles of the models not yet ruled out… Maybe I can explain more of my current working causal model and tell you some authors that care about it, and you can look up their studies and try to find one from which you can invent stories if that helps you?”
If someone said that TO ME I would experience it as a sort of a rhetorical “fuck you”… but WHAT a fuck you! {/me kisses her fingers} Then I would pump them for author recommendations!
My personal goal is often just to find out how the OTHER person feels they do their best thinking, run that process under emulation if I can, and then try to ask good questions from inside their frames. If they have lots of examples there’s a certain virtue to that… but I can think of other good signs of systematically productive thought.
---
If I was going to run “example based discussion” under emulation to try to help you understand my position, I would offer the example of John Hattie’s “Visible Learning”.
It is literally a meta-meta-analysis of education.
It spends the first two chapters just setting up the methodology and responding preemptively to quibbles that will predictable come when motivated thinkers (like classroom teachers that the theory says are teaching suboptimally) try to hear what Hattie has to say.
Chapter 3 finally lays out an abstract architecture of principles for good teaching, by talking about six relevant factors and connecting them all (very very abstractly and loosely) to: tight OODA loops (though not under that name) and Popperian epistemology (explicitly).
I’ll fully grant that it can take me an hour to read 5 pages of this book, and I’m stopping a lot and trying to imagine what Hattie might be saying at each step. The key point for me is that he’s not filling the book with examples, but with abstract empirically authoritative statistical claims about a complex and multi-faceted domain. It doesn’t feel like bullshit, it feels like extremely condensed wisdom.
Because of academic citation norms, in some sense his claims ultimately ground out in studies that are arguably “nothing BUT examples”? He’s trying to condense >800 meta-analyses that cover >50k actual studies that cover >1M observed children.
I could imagine you arguing that this proves how useful examples are, because his book is based on over a million examples, but he hasn’t talked about an example ONCE so far. He talks about methods and subjectively observed tendencies in meta-analyses mostly, trying to prepare the reader with a schema in which later results can land.
Plausibly, anyone could follow Hattie’s citations back to an interesting meta-analysis, look at its references, track back to a likely study, look in their methods section, and find their questionnaires, track back to the methods paper validating an the questionnaire, then look in the supplementary materials to get specific questionnaire items… Then someone could create an imaginary kid in their head who answered that questionnaire some way (like in the study) and then imagine them getting the outcome (like in the study) and use that scenario as “the example”?
I’m not doing that as I read the book. I trust that I could do the above, “because scholarship” but I’m not doing it. When I ask myself why, it seems like it is because it would make reading the (valuable seeming) book EVEN SLOWER?
---
I keep looping back in my mind to the idea that a lot of this strongly depends on which people are talking and what kinds of communication norms are even relevant, and I’m trying to find a place where I think I strongly agree with “looking for examples”...
It makes sense to me that, if I were in the role of an angel investor, and someone wanted $200k from me, and offered 10% of their 2-month-old garage/hobby project, then asking for examples of various of their business claims would be a good way to move forward.
They might not be good at causal modeling, or good at stats, or good at scholarship, or super verbal, but if they have a “native faculty” for building stuff, and budgeting, and building things that are actually useful to actual people… then probably the KEY capacities would be detectable as a head full of examples to various key questions that could be strongly dispositive.
Like… a head full of enough good examples could be sufficient for a basically neurotypical person to build a valuable company, especially if (1) they were examples that addressed key tactical/strategic questions, and (2) no intervening bad examples were ALSO in their head?
(Like if they had terrible examples of startup governance running around in their heads, these might eventually interfere with important parts of being a functional founder down the road. Detecting the inability to give bad examples seems naively hard to me...)
As an investor, I’d be VERY interested in “pre-loaded ready-to-hand theories” that seem likely to actually work. Examples are kinda like “pre-loaded ready-to-hand theories”? Possession of these theories in this form would be a good sign in terms of the founder’s readiness to execute very fast, which is a virtue in startups.
A LACK of ready-to-hand examples would suggest that even a good and feasible idea whose premises were “merely scientifically true” might not happen very fast if an angel funded it and the founder had to instantly start executing on it full time.
I would not be offended if you want to tap out. I feel like we haven’t found a crux yet. I think examples and specificity are interesting and useful and important, but I merely have intuitions about why, roughly like “duh, of course you need data to train a model”, not any high church formal theory with a fancy name that I can link to in wikipedia :-P
I agree that this section of your comment is the most cruxy:
Yes. Then I would say, “Ok, I’ve never encountered a coherent generalization for which I couldn’t easily generate an example, so go ahead and tell me your causal model and I’ll probably cook up an obvious example to satisfy myself in the first minute of your explanation.”
Anyone talking about having a “causal model” is probably beyond the level that my specificity trick is going to demolish. The specificity trick I focus on in this post is for demolishing the coherence of the claims of the average untrained arguer, or occasionally catching oneself at thinking overly vaguely. That’s it.
I think maybe we agree… verbosely… with different emphasis? :-)
At least I think we could communicate reasonably well. I feel like the danger, if any, would arise from playing example ping pong and having the serious disagreements arise from how we “cook (instantiate?)” examples into models, and “uncook (generalize?)” models into examples.
When people just say what their model “actually is”, I really like it.
When people only point to instances I feel like the instances often under-determine the hypothetical underlying idea and leave me still confused as to how to generate novel instances for myself that they would assent to as predictions consistent with the idea that they “meant to mean” with the instances.
Maybe: intensive theories > extensive theories?
Indeed
I feel like the same scrutinity standard is not being applied. Guy with health insurance doesn’t check their health more often catching diseases earlier? Uncertainty doesn’t cause stress and workload on circulatory system? Why are these not holes that prevent it from being coherent? Why can’t Steve claim he has a friend that can be called that can exempilify exploitation?
If the bar is infact low Steve passed it upon positing McDonalds as relevant alternative and the argument went on to actually argue the argument. Or alternatively it requires opinion to have that Robin specification to be coherent and a reasonable arguer could try to hold it to be incoherent.
I feel like this is a case where epistemic status breaks symmetry. A white coat doctor and a witch doctor making the same claims requires the witch doctor to show more evidence to reach the same credibility levels. If argument truly screens off authority the problems needs to be in the argument. Steve is required to have the specification ready on hand during debate.
The difference is that we all have a much clearer picture of what Robin Hanson’s claim means than what Steve means, so Robin’s claim is sufficiently coherent and Steve’s isn’t. I agree there’s a gray area on what is “sufficiently coherent”, but I think we can agree there’s a significant difference on this coherence spectrum between Steve’s claim and Robin’s claim.
For example, any listener can reasonably infer from Robin’s claim that someone on medical insurance who gets cancer shouldn’t be expected to survive with a higher probability than someone without medical insurance. But reasonable listeners can have differing guesses about whether or not Steve’s claim would also describe a world where Uber institutes a $15/hr flat hourly wage for all drivers.
Sure, then I’d just try to understand what specific property of the friend counts as exploitation. In Robin’s case, I already have good reasonable guesses about the operational definitions of “health”. Yes I can try to play “gotcha” and argue that Robin hasn’t nailed down his claim, and in some cases that might actually be the right thing to do—it’s up to me to determine what’s sufficiently coherent for my own understanding, and nail down the claim to that standard, before moving onto arguing about the truth-value of the claim.
Ah, if Steve really meant “Yes Uber screws the employee out of $1/hr compared to McDonald’s”, then he would have passed the specificity bar. But the specific Steve I portray in the dialogue didn’t pass the bar because that’s not really the point he felt he wanted to make. The Steve that I portray found himself confused because his claim really was insufficiently specific, and therefore really was demolished, unexpectedly to him.
Well I am more familiar with settings where I have a duty to understand the world rather than the world having the duty to explain itself to me. I also hold that having unfamiliar things hit higher standards creates epistemic xenophobia. I would hold it important that one doesn’t assign falsehood to a claim they don’t understand. Althought it is also true that assigning truth to a claim one doesn’t understand is dangerous to relatively same caliber.
My go-to assumption would be that Steve understands something different with the word and might be running some sort of moon logic in his head. Rather than declare the “moon proof” to be invalid it’s more important that the translation between moon logic and my planet logic interfaces without confusion. Instead of using a word/concept I do know wrong he is using a word or concept I do not know.
“Coherent” usually points to a concept where a sentence is judged on it’s home logics terms. But as used here it’s clearly in the eye of the beholder. So it’s less “makes objective sense” and more a “makes sense to whom?”. The shared reality you create in a discussion or debate would be the arbiter but if the argument realies too much on those mechanics it doesn’t generalise to contextes outside of that.
Sure, makes sense.
I also just think there are a lot of Steves in the world who are holding on to belief-claims that lack specific referents, who could benefit from reading this post even if no one is arguing with them.
Related: it’s tempting to interpret Ignorance, a skilled practice as pushing for the epistemological stance that specificity should overwhelm Mel the meta meta-analyst
I think this is a common misunderstanding that people are having about what I’m saying. I’m not saying to hunt for a counterexample that demolishes a claim. I’m saying to ask the person making the claim for a single specific example that’s consistent with the general claim.
Imagine that a general claim has 900 examples and 100 counterexamples. Then I’m just asking to see one of the 900 examples :)