Deep skepticism of the sleep literature is fine, even if you rely on some sleep research yourself, but it’s insufficient to respond to the objection of hypocrisy of relying on the sleep literature with “well, I’m really careful about which studies I use”. You need to explain why the studies you use somehow avoid the methodological problems that cause you to reject other studies. If you don’t, it seems like you are just cherry-picking supporting studies because they support you.
I have a tentative guess on why he’s doing that, based on Scott Alexander’s post about trapped priors.
I’ll give an example of the basic problem outlined in the post myself, to spare you from having to read all of it before understanding my comment. Suppose that a physicist spends two hours trying to convince you that the Earth is flat. Would you see that as strong evidence that the Earth is flat? Personally, I’d see that as extremely weak evidence. Instead of updating much that the Earth is flat, the conversation would instead make me seriously consider the following more-plausible-to-me hypotheses:
The physicist really enjoys pulling very elaborate pranks on people.
It’s April fools or something similar and I for some reason just haven’t realized it yet.
The physicist currently has some sort of untreated psychosis.
Any physics department the physicist has studied or worked on is extremely terrible and ought not to be trusted in the future.
I am hallucinating, or otherwise have perceptions of the world that don’t track reality. Perhaps I’m in a dream, perhaps I have extremely-early-onset Alzheimer’s, perhaps I’m on a potent perception-altering drug.
Similarly, if I saw a lot of studies claiming to show something absurd like that prohibited-by-the-laws-of-physics “psychic” phenomena are real, I won’t need to read their methodology to conclude that there’s something wrong with them. And if a group of people claims to have such psychic powers, I won’t think twice before dismissing their personal experience as unreliable. And, at the same time, I’ll accept, without batting an eye, studies and anecdotes claiming that such powers are not possible.
So dismissing arguments from experts, studies, and personal anecdotes as horribly flawed and no more than weak evidence — even before trying to assess their quality — is perfectly reasonable and Bayesian if they’re claiming something that you think is absurd. But clearly, if you have that attitude towards a belief of yours that does not reflect the territory, that will be problematic.
Guzey’s attitude towards sleep research and anecdotes seems compatible with him having a prior that it’s outright absurd that sleep restriction could be harmful. And, like, it’s not an epistemological sin to be born with that prior. But it does mean that what he sees as very weak evidence against his theses won’t necessarily be very weak evidence for people with different priors, if he can’t explain why he thinks that sleep restriction being harmful is absurd.
(I’m not claiming that that is what is going on here. I just thought that the idea of trapped priors was probably relevant.)
Again, re: the default claim “sufficient sleep is good and necessary for proper functioning for normal people on the scale of a few days to a week”, this is not a formal logic claim about all cases, it is a claim about trends and norms. And I am not convinced that the default claim is limited to the scale of a few days to a week, but is instead about sleep in general.
My main objection to Guzey’s response would be a bit different than yours here. I’d point out that I haven’t argued anywhere that “sufficient sleep” is required for “proper functioning,” and that the meta-analyses I quoted don’t purport to show that either. Rather, they report the magnitude and direction of the effect of experimental sleep restriction on e.g. cognitive ability across a variety of different studies, and the magnitude of the effect they show, although substantial, is a far cry from being enough to outright incapacitate you from most everyday tasks. Guzey seems to be responding to a claim like “people do worse on math exams when they’re tipsy compared to when they’re sober” with “but have you talked to tipsy people? They act almost entirely normally!”
If a physicist were to spend two hours trying to explain to me how they knew that the earth was flat, I’d expect to come away from that conversation with a better understanding of the physical world or the social construction of physics knowledge, which would better help me navigate my life, even if I ended up wronger on the bottom-line answer—because that’s how epistemically persuasive explanations work, they have to show an ability to win bets either more often or with less computational cost than alternative hypotheses.
I feel sort of weird about the ‘trapped prior’ point, because I think it’s more reasonably pointed at academic fields than individual people? Like, it is not that surprising for an academic field to have ‘core beliefs’ that everyone who disagrees with is ‘not in the field’, given the forces that people in the field can exert on each other. One fun example of this is PhilGoetz’s post Too Good To Be True, wherein he points to a claim that out of 60 studies studying vaccines and autism, none of them find any link, and then observes that by standard frequentist analysis, ~3 of them should have been significant at the p=0.05 level, and getting none of them significant at that level is pretty unlikely without suppression. And it’s obvious why there would be suppression; no one wants to give ammunition to the enemy.
Separately, it feels like it doesn’t really distinguish ‘justified priors’ from ‘unjustified priors’. If you tell me that a psychology experiment found evidence of psychic effects, I will basically just not believe it. But hopefully if you transported my mind to a universe where psychics were real, I would believe the corresponding studies in those universes—because I had seen things like dowsing being used by oil companies in that universe. This is, from the perspective of a new study, ‘my prior’, but that prior is built out of all of the evidence that I’ve seen before. It’s kind of fair to call my position on psychics a “trapped prior” but it feels more fair to call it “a mountain of evidence”.
I have a tentative guess on why he’s doing that, based on Scott Alexander’s post about trapped priors.
I’ll give an example of the basic problem outlined in the post myself, to spare you from having to read all of it before understanding my comment. Suppose that a physicist spends two hours trying to convince you that the Earth is flat. Would you see that as strong evidence that the Earth is flat? Personally, I’d see that as extremely weak evidence. Instead of updating much that the Earth is flat, the conversation would instead make me seriously consider the following more-plausible-to-me hypotheses:
The physicist really enjoys pulling very elaborate pranks on people.
It’s April fools or something similar and I for some reason just haven’t realized it yet.
The physicist currently has some sort of untreated psychosis.
Any physics department the physicist has studied or worked on is extremely terrible and ought not to be trusted in the future.
I am hallucinating, or otherwise have perceptions of the world that don’t track reality. Perhaps I’m in a dream, perhaps I have extremely-early-onset Alzheimer’s, perhaps I’m on a potent perception-altering drug.
Similarly, if I saw a lot of studies claiming to show something absurd like that prohibited-by-the-laws-of-physics “psychic” phenomena are real, I won’t need to read their methodology to conclude that there’s something wrong with them. And if a group of people claims to have such psychic powers, I won’t think twice before dismissing their personal experience as unreliable. And, at the same time, I’ll accept, without batting an eye, studies and anecdotes claiming that such powers are not possible.
So dismissing arguments from experts, studies, and personal anecdotes as horribly flawed and no more than weak evidence — even before trying to assess their quality — is perfectly reasonable and Bayesian if they’re claiming something that you think is absurd. But clearly, if you have that attitude towards a belief of yours that does not reflect the territory, that will be problematic.
Guzey’s attitude towards sleep research and anecdotes seems compatible with him having a prior that it’s outright absurd that sleep restriction could be harmful. And, like, it’s not an epistemological sin to be born with that prior. But it does mean that what he sees as very weak evidence against his theses won’t necessarily be very weak evidence for people with different priors, if he can’t explain why he thinks that sleep restriction being harmful is absurd.
(I’m not claiming that that is what is going on here. I just thought that the idea of trapped priors was probably relevant.)
My main objection to Guzey’s response would be a bit different than yours here. I’d point out that I haven’t argued anywhere that “sufficient sleep” is required for “proper functioning,” and that the meta-analyses I quoted don’t purport to show that either. Rather, they report the magnitude and direction of the effect of experimental sleep restriction on e.g. cognitive ability across a variety of different studies, and the magnitude of the effect they show, although substantial, is a far cry from being enough to outright incapacitate you from most everyday tasks. Guzey seems to be responding to a claim like “people do worse on math exams when they’re tipsy compared to when they’re sober” with “but have you talked to tipsy people? They act almost entirely normally!”
If a physicist were to spend two hours trying to explain to me how they knew that the earth was flat, I’d expect to come away from that conversation with a better understanding of the physical world or the social construction of physics knowledge, which would better help me navigate my life, even if I ended up wronger on the bottom-line answer—because that’s how epistemically persuasive explanations work, they have to show an ability to win bets either more often or with less computational cost than alternative hypotheses.
I feel sort of weird about the ‘trapped prior’ point, because I think it’s more reasonably pointed at academic fields than individual people? Like, it is not that surprising for an academic field to have ‘core beliefs’ that everyone who disagrees with is ‘not in the field’, given the forces that people in the field can exert on each other. One fun example of this is PhilGoetz’s post Too Good To Be True, wherein he points to a claim that out of 60 studies studying vaccines and autism, none of them find any link, and then observes that by standard frequentist analysis, ~3 of them should have been significant at the p=0.05 level, and getting none of them significant at that level is pretty unlikely without suppression. And it’s obvious why there would be suppression; no one wants to give ammunition to the enemy.
Separately, it feels like it doesn’t really distinguish ‘justified priors’ from ‘unjustified priors’. If you tell me that a psychology experiment found evidence of psychic effects, I will basically just not believe it. But hopefully if you transported my mind to a universe where psychics were real, I would believe the corresponding studies in those universes—because I had seen things like dowsing being used by oil companies in that universe. This is, from the perspective of a new study, ‘my prior’, but that prior is built out of all of the evidence that I’ve seen before. It’s kind of fair to call my position on psychics a “trapped prior” but it feels more fair to call it “a mountain of evidence”.