It would be interesting to make Null experiment, which will consist only of two control groups, so we will know what is the medium difference between two equal groups. It would also interesting to add two control groups in each experiment, as we will see how strong is the effect.
For example if we have difference between main and control in 10 per cent, it could looks like strong result. But if we have second control group, and it has 7 per cent difference from first control group, our result is not so strong after all.
I think that it is clear that can’t do it just splitting existing control group in two parts, as such action could be done in many different ways and researcher could choose most favorable, and also because there could be some interactions inside control group, and also because smaler statistic power.
I think that it is clear that can’t do it just splitting existing control group in two parts, as such action could be done in many different ways and researcher could choose most favorable, and also because there could be some interactions inside control group, and also because smaler statistic power.
You can. Cross-validation, the bootstrap, permutation tests—these rely on that sort of procedure. They generate an empirical distribution of differences between groups or effect sizes which replace the assumption of being two normal distributions etc. It would be better to do those with both the experimental and control data, though.
It would be interesting to make Null experiment, which will consist only of two control groups, so we will know what is the medium difference between two equal groups. It would also interesting to add two control groups in each experiment, as we will see how strong is the effect.
For example if we have difference between main and control in 10 per cent, it could looks like strong result. But if we have second control group, and it has 7 per cent difference from first control group, our result is not so strong after all.
I think that it is clear that can’t do it just splitting existing control group in two parts, as such action could be done in many different ways and researcher could choose most favorable, and also because there could be some interactions inside control group, and also because smaler statistic power.
You can. Cross-validation, the bootstrap, permutation tests—these rely on that sort of procedure. They generate an empirical distribution of differences between groups or effect sizes which replace the assumption of being two normal distributions etc. It would be better to do those with both the experimental and control data, though.