Isn’t this why we do independent verification? As far as I know, in Physics a single experiment almost never causes people to re-examine a theory. It’s quite likely that there are systemic errors that the experimentalists didn’t think of. It’s only after numerous independent verifications that people start looking at theories again.
Or did I miss some subtlety in the argument?
Ref: Michelson–Morley experiment (repeated several times and by several people before being accepted), and more recently the superluminal neutrino result which is currently being called an “anomaly” pending verification at other facilities[1].
It’s quite likely that there are systemic errors that the experimentalists didn’t think of.
This would be an example of negligence on the part of the experimentalist, though perhaps excusable negligence.
Eliezer’s point is that an experimentalist might get results contrary to theory (i.e., extremely unlikely, given that the theory is true) even if the experimentalist thought of every possible source of systematic error. After all, extremely unlikely outcomes do happen (albeit rarely, of course). Therefore, the theorist should be able to “defy the data” without implying that the experimentalist made any kind of oversight.
Yes, this. It simply shouldn’t be necessary—ever—to loudly defy a single result. An I replicated result should not be seen as a result at all, but merely a step in the experimental process. Sadly, that’s not how most people treat results.
Isn’t this why we do independent verification? As far as I know, in Physics a single experiment almost never causes people to re-examine a theory. It’s quite likely that there are systemic errors that the experimentalists didn’t think of. It’s only after numerous independent verifications that people start looking at theories again.
Or did I miss some subtlety in the argument?
Ref: Michelson–Morley experiment (repeated several times and by several people before being accepted), and more recently the superluminal neutrino result which is currently being called an “anomaly” pending verification at other facilities[1].
http://blog.vixra.org/2011/09/19/can-neutrinos-be-superluminal/
This would be an example of negligence on the part of the experimentalist, though perhaps excusable negligence.
Eliezer’s point is that an experimentalist might get results contrary to theory (i.e., extremely unlikely, given that the theory is true) even if the experimentalist thought of every possible source of systematic error. After all, extremely unlikely outcomes do happen (albeit rarely, of course). Therefore, the theorist should be able to “defy the data” without implying that the experimentalist made any kind of oversight.
Yes, this. It simply shouldn’t be necessary—ever—to loudly defy a single result. An I replicated result should not be seen as a result at all, but merely a step in the experimental process. Sadly, that’s not how most people treat results.