I did #4 and #1. Here is what I wrote for each section of #4 (note: this will spoil your ability to do the exercise if you read it).
1. How do you explain these effects?
Seems like a trick question. Like, I have models of the world that feel like they might predict effects 2 and 3, and I can sort of wrangle explanations for 1 and 4, but my split-second reaction is “I’m not sure these are real effects, probably none replicate (though number 2 sounds like it might just be a restatement of a claim I already believe)”.
2. How would you have gone about uncovering them?
As I think about trying to determine whether someone did their diet for ethical reasons, I immediately feel highly skeptical of the result. I think that the things people will tick-box as ‘because I care about animals’ does not necessarily refer to a deep underlying structure of the world that is ‘ethics’, and can refer to one of many things (e.g. exposure to effective guilt-based marketing, reflections on ethical philosophy, the ownership of a dog/cat/pet from an early age, etc). But I guess that just doing a simple questionnaire isn’t of literally zero value.
Loyalty two feels like a thing I could design a better measure for, but I worry this is tangled up with me believing it’s true, and thus illusion-of-transparency assuming people mean the same thing as me if they check-box ‘loyalty’.
Number 3 seems totally testable and straightforward.
Number 4 seems broadly testable. Creativity could be done with that “list the uses of a brick” test, or some other fun ones.
I notice this makes me more skeptical about the first two ‘results’ and more trusting of the last two ‘results’.
3. These are all reversed, and the actual findings were the opposite of what I said. How do you explain the opposite, correct effects?
Ah, the classic ‘I reversed the experimental findings trick’. Well, I guess I did fine on it this time. Oh look, I just managed to think of an explanation for number 2, which is that a more discerning audience of less loyal customers increases adversarial pressures among service providers, raising the prices. Interesting. I think I mostly am noticing how modern psychological research methodology can be quite terrible, and that such a questionnaire without incorporating a thoughtful model of the environment will often be useless. Model-free empirical questions can be overdetermined by the implicit model.
4. Actually, none of these results could be replicated. Why and how were non-null effects detected in the first place? Answers using your designs from (2) are preferable.
Okay. Science is awful.
---
More general thoughts: This helped me notice how relying on assuming a simple empirical psychological claim like this shouldn’t be used as evidence about anything. That pattern-matches to radical skepticism, but that’s not what I mean. I think I’m mostly saying context-free/theory-free claims are meaningless in psychology/sociology, or something like that.
And #1.
The only thing I can come up with is that the graph doesn’t prove causality in any particular way. (it did take me like 3 whole minutes to come up with noticing correlation isn’t causation—I was primarily looking for things like axis labelled in unhelpful ways or something). I can tell a story where these are uncorrelated and everyone is dumb. I can tell a story where decreasing wages is the *explanation* for why debt is growing—it was previously in equilibrium, but now is getting paid off much more slowly. I can tell a story of active prevention, whereby because wages are going down, the government is making students pay less and store more of it as debt so they still have a good quality of life immediately after college.
Again, I’m noticing how simple context-free/theory-free claims do not determine an interpretation.
While the post promised answers in the comments, there were no comments, neither on the post or on the linked Washington Post article, so I’m not sure what the expected take-away was.
Hm, not sure what happened to the Washington Post comments. Sorry about that. Here’s my guess as to what I was thinking:
The axes are comparing an average (median income) to a total (student loan debt). This is generally a recipe for uninformative comparisons. Worse, the average is both per person and per year. So by itself this tells you little about the debt burden shouldered by a typical member of a generation. For example, you could easily see growth in total debt while individual debt burden fell, depending on the growth in degrees awarded and the typical time to pay off debt. If you wanted to make claims about how debt burdens individuals, as the blurb does, you’d have to look at what’s happening with the typical debt of recent graduates.
But of course you can’t stop there and say, “Ah, Peter Thiel is trying to mislead me, I’m going to disbelieve what I see as his point.” Recent-graduate debt has been increasing, just not as much as the graph suggests. And maybe total student loan debt is a significant number in its own right?
(I don’t know if I had intended the above as “the answer”; more likely, I just wanted people thinking about it more thoroughly than some of the commentary I had seen at the time. You also make good points.)
Thanks for trying these out. I don’t think I ever heard in detail from anyone who did (beyond “this was neat”). If I were writing them today I’d be less coy about it.
I did #4 and #1. Here is what I wrote for each section of #4 (note: this will spoil your ability to do the exercise if you read it).
1. How do you explain these effects?
Seems like a trick question. Like, I have models of the world that feel like they might predict effects 2 and 3, and I can sort of wrangle explanations for 1 and 4, but my split-second reaction is “I’m not sure these are real effects, probably none replicate (though number 2 sounds like it might just be a restatement of a claim I already believe)”.
2. How would you have gone about uncovering them?
As I think about trying to determine whether someone did their diet for ethical reasons, I immediately feel highly skeptical of the result. I think that the things people will tick-box as ‘because I care about animals’ does not necessarily refer to a deep underlying structure of the world that is ‘ethics’, and can refer to one of many things (e.g. exposure to effective guilt-based marketing, reflections on ethical philosophy, the ownership of a dog/cat/pet from an early age, etc). But I guess that just doing a simple questionnaire isn’t of literally zero value.
Loyalty two feels like a thing I could design a better measure for, but I worry this is tangled up with me believing it’s true, and thus illusion-of-transparency assuming people mean the same thing as me if they check-box ‘loyalty’.
Number 3 seems totally testable and straightforward.
Number 4 seems broadly testable. Creativity could be done with that “list the uses of a brick” test, or some other fun ones.
I notice this makes me more skeptical about the first two ‘results’ and more trusting of the last two ‘results’.
3. These are all reversed, and the actual findings were the opposite of what I said. How do you explain the opposite, correct effects?
Ah, the classic ‘I reversed the experimental findings trick’. Well, I guess I did fine on it this time. Oh look, I just managed to think of an explanation for number 2, which is that a more discerning audience of less loyal customers increases adversarial pressures among service providers, raising the prices. Interesting. I think I mostly am noticing how modern psychological research methodology can be quite terrible, and that such a questionnaire without incorporating a thoughtful model of the environment will often be useless. Model-free empirical questions can be overdetermined by the implicit model.
4. Actually, none of these results could be replicated. Why and how were non-null effects detected in the first place? Answers using your designs from (2) are preferable.
Okay. Science is awful.
---
More general thoughts: This helped me notice how relying on assuming a simple empirical psychological claim like this shouldn’t be used as evidence about anything. That pattern-matches to radical skepticism, but that’s not what I mean. I think I’m mostly saying context-free/theory-free claims are meaningless in psychology/sociology, or something like that.
And #1.
The only thing I can come up with is that the graph doesn’t prove causality in any particular way. (it did take me like 3 whole minutes to come up with noticing correlation isn’t causation—I was primarily looking for things like axis labelled in unhelpful ways or something). I can tell a story where these are uncorrelated and everyone is dumb. I can tell a story where decreasing wages is the *explanation* for why debt is growing—it was previously in equilibrium, but now is getting paid off much more slowly. I can tell a story of active prevention, whereby because wages are going down, the government is making students pay less and store more of it as debt so they still have a good quality of life immediately after college.
Again, I’m noticing how simple context-free/theory-free claims do not determine an interpretation.
While the post promised answers in the comments, there were no comments, neither on the post or on the linked Washington Post article, so I’m not sure what the expected take-away was.
Hm, not sure what happened to the Washington Post comments. Sorry about that. Here’s my guess as to what I was thinking:
The axes are comparing an average (median income) to a total (student loan debt). This is generally a recipe for uninformative comparisons. Worse, the average is both per person and per year. So by itself this tells you little about the debt burden shouldered by a typical member of a generation. For example, you could easily see growth in total debt while individual debt burden fell, depending on the growth in degrees awarded and the typical time to pay off debt. If you wanted to make claims about how debt burdens individuals, as the blurb does, you’d have to look at what’s happening with the typical debt of recent graduates.
But of course you can’t stop there and say, “Ah, Peter Thiel is trying to mislead me, I’m going to disbelieve what I see as his point.” Recent-graduate debt has been increasing, just not as much as the graph suggests. And maybe total student loan debt is a significant number in its own right?
(I don’t know if I had intended the above as “the answer”; more likely, I just wanted people thinking about it more thoroughly than some of the commentary I had seen at the time. You also make good points.)
Thanks for trying these out. I don’t think I ever heard in detail from anyone who did (beyond “this was neat”). If I were writing them today I’d be less coy about it.