(not really sure what that means in this context).
I am just saying that confounding could make your effect weaker (if there is cancellation of paths), or stronger (if there is some sort of interaction with the treatment), or weaker sometimes and stronger other times. You just don’t know. Confounding doesn’t just increase the variance of your effect estimate, it creates bias in the estimate. That is, if you add up some confounded bits to your estimate, you are adding up garbage.
Wouldn’t this be covered by randomization?
No. The werewolf example is a clear case of the copies not being exchangeable. Different versions of you could react to (randomized!) treatment differently, and you won’t know how without more assumptions. For instance, if you were a woman, you would have a different hormonal composition due to the monthly cycle, etc. etc. etc.
From the sound of it, you’re largely making the theoretician’s objection: “but there are a billion ways your
simple design could go wrong!”
Look, what I am saying is not very complicated. I am not asking you to become a mathematician. You are looking for causal effects. That’s great! It is not my goal to discourage you! Just report your assumptions. All of them. Say you assume monotonicity, exchangeability of copies, etc. If you don’t know what assumptions you need to make, maybe read up on them. Reporting assumptions is good science, right? It’s standard practice in the stats literature.
No, see. The burden of proof is not on me. If you make an assumption, the burden of proof that it holds (or at the very least the burden of reporting) is on you. Causal mechanisms in general are not monotonic...Just report your assumptions. All of them. Say you assume monotonicity, exchangeability of copies, etc. If you don’t know what assumptions you need to make, maybe read up on them.
This is an example of what I mean by you are taking a wildly impractical theoretical approach. Have you ever seen an experiment in which every assumption is reported with a proof? No, because such a paper would not be an experiment but an exercise in pure mathematics or statistics and no one would ever get anything done if they tried to actually apply your suggestions since they would spend all their time reading up on various statistical frameworks and going ‘well, I guess I should specify this and that assumption but wait don’t I also assume independence of who’s the current Justice of the Supreme Court?’ etc
But don’t just assume some random thing you came up with after reading some slice of the literature that happened to catch your fancy will give you the effect you want.
I hate to break it to you, but that’s pretty much how it works. People read a slice of the literature, apply simple common models, which yield reasonable answers, and only start delving into the foundations and examining closely the methods if someone makes a good case that a hidden assumption or a method’s limitation is important. This should not dismay you any more than a philosopher of science should be dismayed that scientists spend their days in the lab and he is only consulted to deal with borderline cases like Intelligent Design.
Reporting assumptions is standard practice. For example in causal inference literature the mantra is often “we assume SUTVA (stable unit treatment value assumption), and conditional ignorability.” You can’t prove them all (in fact many are untestable). Reporting is still a good idea (for sensitivity analysis, replication, arguing about their reasonableness, etc.)
Exchangeability of copies and monotonicity are pretty important. People always report monotonicity (because you get identification when you could not before). But anyways, I shouldn’t be the one to have to tell you this.
Also, it’s not some, it’s all assumptions needed to get your answer from the data. Even if exchangeability holds for you, it might not hold for someone else who might want to try your design. If you don’t write down what you assume, how should they know if your design will carry over?
Anyways, this is just the Scruffy AI mistake all over again. Actually it’s worse than that. The scientific attitude is to try to falsify, e.g. look for reasons your model might fail. You are assuming as a default that your model is reasonable, and not even leaving a paper trail.
I am just saying that confounding could make your effect weaker (if there is cancellation of paths), or stronger (if there is some sort of interaction with the treatment), or weaker sometimes and stronger other times. You just don’t know. Confounding doesn’t just increase the variance of your effect estimate, it creates bias in the estimate. That is, if you add up some confounded bits to your estimate, you are adding up garbage.
No. The werewolf example is a clear case of the copies not being exchangeable. Different versions of you could react to (randomized!) treatment differently, and you won’t know how without more assumptions. For instance, if you were a woman, you would have a different hormonal composition due to the monthly cycle, etc. etc. etc.
Look, what I am saying is not very complicated. I am not asking you to become a mathematician. You are looking for causal effects. That’s great! It is not my goal to discourage you! Just report your assumptions. All of them. Say you assume monotonicity, exchangeability of copies, etc. If you don’t know what assumptions you need to make, maybe read up on them. Reporting assumptions is good science, right? It’s standard practice in the stats literature.
This is an example of what I mean by you are taking a wildly impractical theoretical approach. Have you ever seen an experiment in which every assumption is reported with a proof? No, because such a paper would not be an experiment but an exercise in pure mathematics or statistics and no one would ever get anything done if they tried to actually apply your suggestions since they would spend all their time reading up on various statistical frameworks and going ‘well, I guess I should specify this and that assumption but wait don’t I also assume independence of who’s the current Justice of the Supreme Court?’ etc
I hate to break it to you, but that’s pretty much how it works. People read a slice of the literature, apply simple common models, which yield reasonable answers, and only start delving into the foundations and examining closely the methods if someone makes a good case that a hidden assumption or a method’s limitation is important. This should not dismay you any more than a philosopher of science should be dismayed that scientists spend their days in the lab and he is only consulted to deal with borderline cases like Intelligent Design.
Reporting assumptions is standard practice. For example in causal inference literature the mantra is often “we assume SUTVA (stable unit treatment value assumption), and conditional ignorability.” You can’t prove them all (in fact many are untestable). Reporting is still a good idea (for sensitivity analysis, replication, arguing about their reasonableness, etc.)
That’s reporting some assumptions, and presumably ones who have earned their being specifically singled out.
Exchangeability of copies and monotonicity are pretty important. People always report monotonicity (because you get identification when you could not before). But anyways, I shouldn’t be the one to have to tell you this.
Also, it’s not some, it’s all assumptions needed to get your answer from the data. Even if exchangeability holds for you, it might not hold for someone else who might want to try your design. If you don’t write down what you assume, how should they know if your design will carry over?
Anyways, this is just the Scruffy AI mistake all over again. Actually it’s worse than that. The scientific attitude is to try to falsify, e.g. look for reasons your model might fail. You are assuming as a default that your model is reasonable, and not even leaving a paper trail.