I didn’t catch the deliberately-introduced error; instead I thought the bit about a recession in the 1830s was likely false. I took “the Industrial Revolution” to include the “Second Industrial Revolution” (as, indeed, the article seems to) and it seems to me that electricity was important for the progress of that, so I’m not entirely convinced that what was introduced to the article was exactly a falsehood.
Price gouging:
It seemed utterly unbelievable to me that a gathering of economists would disapprove wholeheartedly of “price gouging” and, lo, I was correct. (I feel like this one was pretty easy.)
World economy:
A few things seemed dubious and nothing seemed obviously wrong. There’s a statement that economists usually use PPP to translate currencies and not official exchange rates, but then a table that (if I understand it right) uses both of those. (But this is kinda reconcilable: usually PPP is used for comparisons, and it’s much better, but no harm in including nominal figures as well.) There’s a claim that it’s usual to consider only “human economic activity” and I don’t know what that means—what’s the alternative? E.g., if it doesn’t include making things with automatic machines then this is obviously false; if it’s contrasting with aliens or something then it’s so obviously true as not to be worth saying; probably it means something else but I don’t know what, and looking it up feels a bit cheaty. This feels odd but my epistemic state is more “I don’t understand something here” than “I know what this means and it’s wrong”. (Oh, I guess maybe it means “human economic activity as opposed to other human activities not classified as economic, like parents taking care of their own children”, in which case it’s not content-free and is probably correct.) There’s a claim that before the Industrial Revolution “global output was dominated by China and India” and that India was the largest economy circa 1CE which seems iffy—bigger than the Roman Empire? -- but, I dunno, those both had large populations and China at least was pretty advanced in some ways. There’s a claim that the African Union has an economy bigger than $2T/year; I’m not sure just who’s in the African Union but most of Africa is pretty poor and I’d have thought that if e.g. Nigeria and/or South Africa is at ~$1T then there ought to be a bigger list of $2T individual countries. The first of these feels like a nitpick; for the second, it feels like maybe I just don’t understand a technical term; the third and fourth feel like memorization more than understanding. The thing that seems most likely wrong is the claim about China and India, though I wouldn’t be astonished to find it’s completely true. … The actual fabrication turns out to be none of those things. I did wonder about errors introduced in the table that was falsified, but didn’t consider the thing that was actually done. It’s a fair cop.
Cell (biology):
Surely 1-5nm is waaaay too small for even a small prokaryotic cell. The 100nm given as biggest typical size for a eukaryotic cell feels way too small too. Are these really microns rather than nanometres, perhaps? Of course some eukaryotic cells get much larger than 100um—consider a hen’s egg, for instance. But 100um feels like a plausible “seldom much bigger than this” figure. (Yup, this is spot on, and indeed these were originally microns.)
Fundamental theorems of welfare economics:
This is not a field I’m aware of knowing anything about, but the two theorems sound familiar (though I hadn’t associated them with the specific title here). The only thing that seems obviously fishy is that the article says the theorems don’t ensure existence of the equilibria, but also says that the second theorem guarantees that “any Pareto optimum can be supported as a competitive equilibrium for some initial set of endowments”, which sure sounds like ensuring the existence of equilibria to me. There are a couple of other things that look strange but I think they’re more likely just badly-written-article strange: “The production economy is more general and entails additional assumptions” (surely to be more general is to require fewer assumptions, but maybe what’s meant is that in the broader setting you need extra assumptions that automatically hold in the narrower setting or something), and “The assumptions are all based on the standard graduate microeconomics textbook” which is just a really weird sentence. There are lots of technicalities wherein errors could be hiding, but I’m no economist and wouldn’t notice. So, my guess is that it’s the “don’t ensure existence” thing. -- Nope, all those are apparently correct, and actually there are some further technical conditions that I didn’t know about. This seems very much in the “memorization” rather than “understanding” category to me; e.g., for all I know “perfect information” is implied by “rationality” as those terms are used in economics. Maybe given infinite intelligence I could have figured out that there needed to be some sort of condition about externalities, but: bah.
List of causes of death by rate:
This says that neurological disorders are the biggest single cause of death. Ahead of both cardiovascular and respiratory problems? Really? Can’t possibly be right. And, indeed, it isn’t. On the one hand, this is “memorization”; on the other, surely this is pretty well known.
Natural selection:
Surely that description of “microevolution” and “macroevolution” can’t be right; the distinction is about the scale of the changes, not the nature of the environmental pressures that cause them. … Huh, apparently that isn’t the issue; the actual error is one I should have spotted if I’d been reading more carefully and hadn’t seen something else I was sure was wrong. I have to say I still think those sentences about microevolution and macroevolution are wrong. Aren’t they? (I tend to avoid those terms, which I mostly associate with creationists who argue that one is real and the other not, so maybe I’m misunderstanding them, but e.g. the Wikipedia pages for them don’t seem to match what’s claimed in this article.)
Working time:
Surely the decreases in working hours shown in that table are too dramatic. Unless, I guess, they’re the result of e.g. increasing numbers of part-time workers, rather than changes to typical full-time hours or something? (If so, then right next to the table there should be a load of caveats about this.) But this claims that, for instance, in France in 1950 the average worker worked nearly 3000 hours per year, versus about half that now. And can it really be true that Greece has longer-working workers than any other EU country? It’s not that any of the individual figures is completely unbelievable, but in the aggregate something seems really off. I’m sure there has been a decrease in working hours, but it surely can’t be this big. (I may have been encouraged toward this conclusion by noticing that the directory containing this example had what looked like data-processing code, suggesting that something quantitative has been changed in a nontrivial way.) Anyway, cheaty or not, I was spot on with this.
So, I got exactly half of them right (and alternated between right and wrong!). For two of the wrong ones I’m prepared to argue that the thing I thought I’d spotted is wrong even though it wasn’t deliberately introduced as a test; for another, I’m maybe prepared to argue that the deliberate error introduced isn’t exactly an error.
Thanks for writing this, it was interesting to read a participant’s thoughts!
Responses, spoilered:
Industrial revolution: I think if you re-read the article and look at all the modifications made, you will agree that there definitely are false claims. (The original answer sheet referred to a version that had fewer modifications than the final edited article; I have fixed this.)
Price gouging: I do think this is pretty clear if one understand economics, but indeed, the public has very different views from economists here, so I thought it makes for a good change. (This wasn’t obvious to all of my friends, at least.)
World economy: I received some (fair) complaints about there being too much stuff in the article. In any case, it might be good for one to seriously think about what the world economy looks like.
Cell: Yep, one of the easier ones I’d say.
Fundamental theorems of welfare economics: Yeah, I don’t think modification was successful. (But let me defend myself: I wanted to try some “lies of omission”, and something where an omission is unarguably wrong is in the context of mathematical theorems with missing assumptions. Well, I thought for math it’s difficult to find an example that is not too easy nor too hard, and decided to go for economics instead. And asymmetric information and externalities really are of practical importance.)
List of causes by death rate: Leans towards memorization, yes. I’d guess it’s not obvious to everyone, though, and I think one can do non-zero inference here.
Natural selection: (I think this is one of the weaker modifications.)
I tried them all. My notes are full of spoilers.
Industrial Revolution (_2):
I didn’t catch the deliberately-introduced error; instead I thought the bit about a recession in the 1830s was likely false. I took “the Industrial Revolution” to include the “Second Industrial Revolution” (as, indeed, the article seems to) and it seems to me that electricity was important for the progress of that, so I’m not entirely convinced that what was introduced to the article was exactly a falsehood.
Price gouging:
It seemed utterly unbelievable to me that a gathering of economists would disapprove wholeheartedly of “price gouging” and, lo, I was correct. (I feel like this one was pretty easy.)
World economy:
A few things seemed dubious and nothing seemed obviously wrong. There’s a statement that economists usually use PPP to translate currencies and not official exchange rates, but then a table that (if I understand it right) uses both of those. (But this is kinda reconcilable: usually PPP is used for comparisons, and it’s much better, but no harm in including nominal figures as well.) There’s a claim that it’s usual to consider only “human economic activity” and I don’t know what that means—what’s the alternative? E.g., if it doesn’t include making things with automatic machines then this is obviously false; if it’s contrasting with aliens or something then it’s so obviously true as not to be worth saying; probably it means something else but I don’t know what, and looking it up feels a bit cheaty. This feels odd but my epistemic state is more “I don’t understand something here” than “I know what this means and it’s wrong”. (Oh, I guess maybe it means “human economic activity as opposed to other human activities not classified as economic, like parents taking care of their own children”, in which case it’s not content-free and is probably correct.) There’s a claim that before the Industrial Revolution “global output was dominated by China and India” and that India was the largest economy circa 1CE which seems iffy—bigger than the Roman Empire? -- but, I dunno, those both had large populations and China at least was pretty advanced in some ways. There’s a claim that the African Union has an economy bigger than $2T/year; I’m not sure just who’s in the African Union but most of Africa is pretty poor and I’d have thought that if e.g. Nigeria and/or South Africa is at ~$1T then there ought to be a bigger list of $2T individual countries. The first of these feels like a nitpick; for the second, it feels like maybe I just don’t understand a technical term; the third and fourth feel like memorization more than understanding. The thing that seems most likely wrong is the claim about China and India, though I wouldn’t be astonished to find it’s completely true. … The actual fabrication turns out to be none of those things. I did wonder about errors introduced in the table that was falsified, but didn’t consider the thing that was actually done. It’s a fair cop.
Cell (biology):
Surely 1-5nm is waaaay too small for even a small prokaryotic cell. The 100nm given as biggest typical size for a eukaryotic cell feels way too small too. Are these really microns rather than nanometres, perhaps? Of course some eukaryotic cells get much larger than 100um—consider a hen’s egg, for instance. But 100um feels like a plausible “seldom much bigger than this” figure. (Yup, this is spot on, and indeed these were originally microns.)
Fundamental theorems of welfare economics:
This is not a field I’m aware of knowing anything about, but the two theorems sound familiar (though I hadn’t associated them with the specific title here). The only thing that seems obviously fishy is that the article says the theorems don’t ensure existence of the equilibria, but also says that the second theorem guarantees that “any Pareto optimum can be supported as a competitive equilibrium for some initial set of endowments”, which sure sounds like ensuring the existence of equilibria to me. There are a couple of other things that look strange but I think they’re more likely just badly-written-article strange: “The production economy is more general and entails additional assumptions” (surely to be more general is to require fewer assumptions, but maybe what’s meant is that in the broader setting you need extra assumptions that automatically hold in the narrower setting or something), and “The assumptions are all based on the standard graduate microeconomics textbook” which is just a really weird sentence. There are lots of technicalities wherein errors could be hiding, but I’m no economist and wouldn’t notice. So, my guess is that it’s the “don’t ensure existence” thing. -- Nope, all those are apparently correct, and actually there are some further technical conditions that I didn’t know about. This seems very much in the “memorization” rather than “understanding” category to me; e.g., for all I know “perfect information” is implied by “rationality” as those terms are used in economics. Maybe given infinite intelligence I could have figured out that there needed to be some sort of condition about externalities, but: bah.
List of causes of death by rate:
This says that neurological disorders are the biggest single cause of death. Ahead of both cardiovascular and respiratory problems? Really? Can’t possibly be right. And, indeed, it isn’t. On the one hand, this is “memorization”; on the other, surely this is pretty well known.
Natural selection:
Surely that description of “microevolution” and “macroevolution” can’t be right; the distinction is about the scale of the changes, not the nature of the environmental pressures that cause them. … Huh, apparently that isn’t the issue; the actual error is one I should have spotted if I’d been reading more carefully and hadn’t seen something else I was sure was wrong. I have to say I still think those sentences about microevolution and macroevolution are wrong. Aren’t they? (I tend to avoid those terms, which I mostly associate with creationists who argue that one is real and the other not, so maybe I’m misunderstanding them, but e.g. the Wikipedia pages for them don’t seem to match what’s claimed in this article.)
Working time:
Surely the decreases in working hours shown in that table are too dramatic. Unless, I guess, they’re the result of e.g. increasing numbers of part-time workers, rather than changes to typical full-time hours or something? (If so, then right next to the table there should be a load of caveats about this.) But this claims that, for instance, in France in 1950 the average worker worked nearly 3000 hours per year, versus about half that now. And can it really be true that Greece has longer-working workers than any other EU country? It’s not that any of the individual figures is completely unbelievable, but in the aggregate something seems really off. I’m sure there has been a decrease in working hours, but it surely can’t be this big. (I may have been encouraged toward this conclusion by noticing that the directory containing this example had what looked like data-processing code, suggesting that something quantitative has been changed in a nontrivial way.) Anyway, cheaty or not, I was spot on with this.
So, I got exactly half of them right (and alternated between right and wrong!). For two of the wrong ones I’m prepared to argue that the thing I thought I’d spotted is wrong even though it wasn’t deliberately introduced as a test; for another, I’m maybe prepared to argue that the deliberate error introduced isn’t exactly an error.
Thanks for writing this, it was interesting to read a participant’s thoughts!
Responses, spoilered:
Industrial revolution: I think if you re-read the article and look at all the modifications made, you will agree that there definitely are false claims. (The original answer sheet referred to a version that had fewer modifications than the final edited article; I have fixed this.)
Price gouging: I do think this is pretty clear if one understand economics, but indeed, the public has very different views from economists here, so I thought it makes for a good change. (This wasn’t obvious to all of my friends, at least.)
World economy: I received some (fair) complaints about there being too much stuff in the article. In any case, it might be good for one to seriously think about what the world economy looks like.
Cell: Yep, one of the easier ones I’d say.
Fundamental theorems of welfare economics: Yeah, I don’t think modification was successful. (But let me defend myself: I wanted to try some “lies of omission”, and something where an omission is unarguably wrong is in the context of mathematical theorems with missing assumptions. Well, I thought for math it’s difficult to find an example that is not too easy nor too hard, and decided to go for economics instead. And asymmetric information and externalities really are of practical importance.)
List of causes by death rate: Leans towards memorization, yes. I’d guess it’s not obvious to everyone, though, and I think one can do non-zero inference here.
Natural selection: (I think this is one of the weaker modifications.)
Working time: Deleted the spoiler files, thanks.