[W]hat they do is to keep all the data points that support the theory, and throw out the rest, calling it a “divergence problem”, and further, claim the remaining points as additional substantiation of the theory.
And yet the field completely lacks journals with articles criticizing this.
Would you clarify this? That seems on its face to be a very strong, which is to say improbable, claim.
I wasn’t saying journals don’t mention the divergence problem, if that’s what you thought. I was saying they don’t criticize the practice of stripping all the data you don’t like from a dataset and then calling the remaining points further substantiation of your theory. It’s this “trick” that is regarded as commonplace in climatology and thus “no big deal”.
I was saying they don’t criticize the practice of stripping all the data you don’t like from a dataset and then calling the remaining points further substantiation of your theory.
There seem to be two kinds of criticism that it’s important to distinguish. On the one hand, there is the following domain-invariant criticism: “It’s wrong to strip data with no motivation other than you don’t like it.” The difficulty with making this criticism is that you have to justify your claim to be able to read the data-stripper’s mind. You need to show that this really was their only motivation. However, although you might have sufficient Bayesian evidence to justify this claim, you probably don’t have enough scientific evidence to convince a journal editor.
On the other hand, there are domain-specific criticisms: “It’s wrong to strip this specific data, and here are domain-specific reasons why it’s wrong: X, Y, and Z.” (E.g., X might be a domain-specific argument that the data probably wasn’t due to measurement error.) It seems much easier to justify this latter kind of criticism at the standards required for a scientific journal.
These considerations are independent of the domain under consideration. I would expect them to operate in other domains besides climate science. For example, I would expect it to be uncommon to find astronomers accusing each other in peer reviewed journals of throwing out data just because they don’t like it, even though I expect that it probably happens just as often as in climatology.
It’s just easier to avoid getting into psychological motivations for throwing data out if you have a theoretic argument for why the data shouldn’t have been thrown out. This seems sufficient to me to explain your observation.
It’s this “trick” that is regarded as commonplace in climatology and thus “no big deal”.
In that case, you should be able to find climatologists openly admitting to throwing out data just because they don’t like it. But the “just because” part rules out all the alleged examples that I’ve seen, including those from the CRU e-mails.
There seem to be two kinds of criticism that it’s important to distinguish. On the one hand, there is the following domain-invariant criticism: “It’s wrong to strip data with no motivation other than you don’t like it.”
This wasn’t my claim. They may very well have a reason for excluding that data, and were well-intentioned in doing so. It’s just that they don’t understand that when you filter a data set so that it only retains points consistent with theory T, you can’t turn around and use it as evidence of T. And no one ever points this out.
It’s not that they recognize themselves as throwing out data points because they don’t like them; it’s that “well of course these points are wrong—they don’t match the theory!”
In that case, you should be able to find climatologists openly admitting to throwing out data just because they don’t like it. But the “just because” part rules out all the alleged examples that I’ve seen, including those from the CRU e-mails.
Really? You gave me the impression before you hadn’t read them, based on your reaction to the term “divergence problem”. But if you read them, you know that this is what happened: Scientist 1 notices that data set A shows cooling after time t1. Scientist 2 says, don’t worry, just delete the part after t1, but otherwise continue to use the data set; this is a standard technique. (A brilliant idea, even—i.e. “trick”)
It would be one thing if they said, “Clip out points x304 thru x509 because of data-specific problem P related to that span, then check for conformance with theory T.” But here, it was, “Clip out data on the basis of it being inconsistent with T (hopefully we’ll have a reason later), and then cite it as proof of T.” (The remainder was included in a chart attempting to substantiate T.)
Weren’t they filtering out proxy data because it was inconsistent with the (more reliable) data, not with the theory? The divergence problem is that the tree ring proxy diverges from the actual measured temperatures after 1960. The tree ring data show a pretty good fit with the measured temperatures from 1850 or so to 1960, so it seems like they do serve as a decent proxy for temperature, which raises the questions of 1) what to do with the tree ring data to estimate historical temperatures and 2) why this divergence in trends is happening.
The initial response to question 1 was to exclude the post-1960 data, essentially assuming that something weird happened to the trees after 1960 which didn’t affect the rest of the data set. That is problematic, especially since they didn’t have an answer to question 2, but it’s not as bad as what you’re describing. There’s no need to even consider any theory T. And now there’s been a bunch of research into why the divergence happens and what it implies about the proxy estimates, as well as efforts to find other proxies that don’t behave in this weird way.
Again, the problem is not that they threw out a portion of the series. The problem is throwing out a portion of the series and also using the remainder as further substantiation. Yes, the fact that it doesn’t match more reliable measures is a reason to conclude it’s invalid during one particular period; but having decided this, it cannot count as an additional supporting data point.
If the inference flows from the other measures to the tree ring data, it cannot flow back as reinforcement for the other measures.
But if they’re fitting the tree ring data to another data set and not to the theory, then they don’t have the straightforward circularity problem where the data are being tailored to the theory and then used as confirmation of that theory.
I’m starting to think that there’s a bigger inferential gap between us than I realized. I don’t see how tree ring data has been used “as reinforcement for the other measures,” and now I’m wondering what you mean by it being used to further substantiate the theory, and even what the theory is. Maybe it’s not worth continuing off on this tangent here?
Let me try one last time, with as little jargon as possible. Here is what I am claiming happened, and what its implications are:
Most proxies for temperature follow a temperature vs. time pattern of P1.
Some don’t. They adhere to a different pattern, P2, which is just P1 for a while, and then something different.
Scientists present a claim C1: the past history of temperature is that of P1.
Scientists present data substantating C1. Their data is the proxies following P1.
The scientists provide further data to substantiate C1. That data is the proxies following P2, but with the data that are different from P1 trimmed off.
So scientists were using P2, filtered for its agreement with P1, to prove C1.
That is not kosher.
That method was used in major reports.
That method went uncriticized for years after certainty of C1 was claimed.
That merits an epic facepalm regarding the basic reasoning skills of this field.
Does this exposition differe from what you thought I was arguing before?
Then I guess I just disagree with you. Scientists’ belief about the temperature pattern (P1) from 1850 to the present isn’t based on proxies—it’s based on measurements of the temperature which are much more reliable than any proxy. The best Bayesian estimate of the temperature since 1850 gives almost all of the weight to the measurements and very little weight to any other source of evidence (that is especially true over the past 50 years when measurements have been more rigorous, and that is the time period when P1 and P2 differ).
The tree ring proxy was filtered based on its agreement with the temperature measurements, and then used to estimate temperatures prior to 1850, when we don’t have measurements. If you want to think of it as substantiating something, it helped confirm the estimates made with other proxy data sets (other tree rings, ice cores, etc.), and it was not filtered based on its agreement with those other proxies. So I don’t think that the research has the kind of obvious flaw that you’re describing here.
I do think that the divergence problem raises questions which I haven’t seen answered adequately, but I’ve assumed that those questions were dealt with in the climate literature. The biggest issue I have is with using the tree ring proxy to support the claim that the temperatures of the past few decades are unprecedented (in the context of the past 1500 years or so) when that proxy hasn’t tracked the high temperatures over the past few decades. I thought you might have been referring to that with your “further substantiation” comment, and that either you knew enough about the literature to correct my mistaken assumption that it dealt with this problem, or you were overclaiming by that nobody in the field was concerned about this and we could at least get glimpses of the literature that dealt with it. (And I have gotten those glimpses over the past couple days—Wikipedia cites a paper that raises the possibility that tree rings don’t track temperatures above a certain threshold, and the paper I linked shows that they are trying to use proxies that don’t diverge.)
Would you clarify this? That seems on its face to be a very strong, which is to say improbable, claim.
The first hit on Google scholar for climate “divergence problem” turns up this: On the ‘Divergence Problem’ in Northern Forests: A review of the tree-ring evidence and possible causes from the journal Global and Planetary Change. From a cursory glance at the abstract, it seems to fit the bill.
I wasn’t saying journals don’t mention the divergence problem, if that’s what you thought. I was saying they don’t criticize the practice of stripping all the data you don’t like from a dataset and then calling the remaining points further substantiation of your theory. It’s this “trick” that is regarded as commonplace in climatology and thus “no big deal”.
There seem to be two kinds of criticism that it’s important to distinguish. On the one hand, there is the following domain-invariant criticism: “It’s wrong to strip data with no motivation other than you don’t like it.” The difficulty with making this criticism is that you have to justify your claim to be able to read the data-stripper’s mind. You need to show that this really was their only motivation. However, although you might have sufficient Bayesian evidence to justify this claim, you probably don’t have enough scientific evidence to convince a journal editor.
On the other hand, there are domain-specific criticisms: “It’s wrong to strip this specific data, and here are domain-specific reasons why it’s wrong: X, Y, and Z.” (E.g., X might be a domain-specific argument that the data probably wasn’t due to measurement error.) It seems much easier to justify this latter kind of criticism at the standards required for a scientific journal.
These considerations are independent of the domain under consideration. I would expect them to operate in other domains besides climate science. For example, I would expect it to be uncommon to find astronomers accusing each other in peer reviewed journals of throwing out data just because they don’t like it, even though I expect that it probably happens just as often as in climatology.
It’s just easier to avoid getting into psychological motivations for throwing data out if you have a theoretic argument for why the data shouldn’t have been thrown out. This seems sufficient to me to explain your observation.
In that case, you should be able to find climatologists openly admitting to throwing out data just because they don’t like it. But the “just because” part rules out all the alleged examples that I’ve seen, including those from the CRU e-mails.
This wasn’t my claim. They may very well have a reason for excluding that data, and were well-intentioned in doing so. It’s just that they don’t understand that when you filter a data set so that it only retains points consistent with theory T, you can’t turn around and use it as evidence of T. And no one ever points this out.
It’s not that they recognize themselves as throwing out data points because they don’t like them; it’s that “well of course these points are wrong—they don’t match the theory!”
Really? You gave me the impression before you hadn’t read them, based on your reaction to the term “divergence problem”. But if you read them, you know that this is what happened: Scientist 1 notices that data set A shows cooling after time t1. Scientist 2 says, don’t worry, just delete the part after t1, but otherwise continue to use the data set; this is a standard technique. (A brilliant idea, even—i.e. “trick”)
It would be one thing if they said, “Clip out points x304 thru x509 because of data-specific problem P related to that span, then check for conformance with theory T.” But here, it was, “Clip out data on the basis of it being inconsistent with T (hopefully we’ll have a reason later), and then cite it as proof of T.” (The remainder was included in a chart attempting to substantiate T.)
Weren’t they filtering out proxy data because it was inconsistent with the (more reliable) data, not with the theory? The divergence problem is that the tree ring proxy diverges from the actual measured temperatures after 1960. The tree ring data show a pretty good fit with the measured temperatures from 1850 or so to 1960, so it seems like they do serve as a decent proxy for temperature, which raises the questions of 1) what to do with the tree ring data to estimate historical temperatures and 2) why this divergence in trends is happening.
The initial response to question 1 was to exclude the post-1960 data, essentially assuming that something weird happened to the trees after 1960 which didn’t affect the rest of the data set. That is problematic, especially since they didn’t have an answer to question 2, but it’s not as bad as what you’re describing. There’s no need to even consider any theory T. And now there’s been a bunch of research into why the divergence happens and what it implies about the proxy estimates, as well as efforts to find other proxies that don’t behave in this weird way.
Again, the problem is not that they threw out a portion of the series. The problem is throwing out a portion of the series and also using the remainder as further substantiation. Yes, the fact that it doesn’t match more reliable measures is a reason to conclude it’s invalid during one particular period; but having decided this, it cannot count as an additional supporting data point.
If the inference flows from the other measures to the tree ring data, it cannot flow back as reinforcement for the other measures.
But if they’re fitting the tree ring data to another data set and not to the theory, then they don’t have the straightforward circularity problem where the data are being tailored to the theory and then used as confirmation of that theory.
I’m starting to think that there’s a bigger inferential gap between us than I realized. I don’t see how tree ring data has been used “as reinforcement for the other measures,” and now I’m wondering what you mean by it being used to further substantiate the theory, and even what the theory is. Maybe it’s not worth continuing off on this tangent here?
Let me try one last time, with as little jargon as possible. Here is what I am claiming happened, and what its implications are:
Most proxies for temperature follow a temperature vs. time pattern of P1.
Some don’t. They adhere to a different pattern, P2, which is just P1 for a while, and then something different.
Scientists present a claim C1: the past history of temperature is that of P1.
Scientists present data substantating C1. Their data is the proxies following P1.
The scientists provide further data to substantiate C1. That data is the proxies following P2, but with the data that are different from P1 trimmed off.
So scientists were using P2, filtered for its agreement with P1, to prove C1.
That is not kosher.
That method was used in major reports.
That method went uncriticized for years after certainty of C1 was claimed.
That merits an epic facepalm regarding the basic reasoning skills of this field.
Does this exposition differe from what you thought I was arguing before?
Then I guess I just disagree with you. Scientists’ belief about the temperature pattern (P1) from 1850 to the present isn’t based on proxies—it’s based on measurements of the temperature which are much more reliable than any proxy. The best Bayesian estimate of the temperature since 1850 gives almost all of the weight to the measurements and very little weight to any other source of evidence (that is especially true over the past 50 years when measurements have been more rigorous, and that is the time period when P1 and P2 differ).
The tree ring proxy was filtered based on its agreement with the temperature measurements, and then used to estimate temperatures prior to 1850, when we don’t have measurements. If you want to think of it as substantiating something, it helped confirm the estimates made with other proxy data sets (other tree rings, ice cores, etc.), and it was not filtered based on its agreement with those other proxies. So I don’t think that the research has the kind of obvious flaw that you’re describing here.
I do think that the divergence problem raises questions which I haven’t seen answered adequately, but I’ve assumed that those questions were dealt with in the climate literature. The biggest issue I have is with using the tree ring proxy to support the claim that the temperatures of the past few decades are unprecedented (in the context of the past 1500 years or so) when that proxy hasn’t tracked the high temperatures over the past few decades. I thought you might have been referring to that with your “further substantiation” comment, and that either you knew enough about the literature to correct my mistaken assumption that it dealt with this problem, or you were overclaiming by that nobody in the field was concerned about this and we could at least get glimpses of the literature that dealt with it. (And I have gotten those glimpses over the past couple days—Wikipedia cites a paper that raises the possibility that tree rings don’t track temperatures above a certain threshold, and the paper I linked shows that they are trying to use proxies that don’t diverge.)