I think the usage in the cited book is bad and unorthodox. E.g. one can still study storms experimentally—though nobody can completely control a storm.
I think we are talking past each other. I agree that those are experiments in a broad and colloquial use of the term. They aren’t “controlled” experiments: which is a term that I was wanting to clarify (since I know a little bit about it). This means that they do not allow you to randomly assign treatments to experimental units which generally means that the risk of bias is greater (hence the statistical analysis must be done with care and the conclusions drawn should face greater scrutiny).
Pick up any textbook on statistical design or statistical analysis of experiments and the framework I gave will be what’s in there for “controlled experimentation”. There are other types of experiments. But these suffer from the problem that it can be difficult to sort out hidden causes. Suppose we want to know if the presence of A causes C (say eating meat causes heart disease). In an observational study we find units having trait A and those not (so find meat-eaters and vegetarians) and we then wait to observe response C. If we observe a response C in experimental units possessing trait A, its hard to know if A causes C or if there is some third trait B (present in some of the units) which causes both A and C.
In the case of a controlled experiment, A is now a treatment and not a trait of the units (so in this case you would randomly assign a carnivorous or vegetarian diet to people), thus we can randomly assign A to the units (and assume the randomization means that not every unit having hidden trait B will be given treatment A). In this case we might observe that A and C have no relation, whereas in the observational study we might. (For instance people who choose to be vegetarian may be more focused on health)
An example of how econometricians have dealt with “selection bias” or the fact that observation studies fail to have certain nice properties of controlled experiments is here
Natural experiments are experiments too. See:
http://en.wikipedia.org/wiki/Natural_experiment
http://en.wikipedia.org/wiki/Experiment
http://dictionary.reference.com/browse/experiment
I think the usage in the cited book is bad and unorthodox. E.g. one can still study storms experimentally—though nobody can completely control a storm.
I think we are talking past each other. I agree that those are experiments in a broad and colloquial use of the term. They aren’t “controlled” experiments: which is a term that I was wanting to clarify (since I know a little bit about it). This means that they do not allow you to randomly assign treatments to experimental units which generally means that the risk of bias is greater (hence the statistical analysis must be done with care and the conclusions drawn should face greater scrutiny).
Pick up any textbook on statistical design or statistical analysis of experiments and the framework I gave will be what’s in there for “controlled experimentation”. There are other types of experiments. But these suffer from the problem that it can be difficult to sort out hidden causes. Suppose we want to know if the presence of A causes C (say eating meat causes heart disease). In an observational study we find units having trait A and those not (so find meat-eaters and vegetarians) and we then wait to observe response C. If we observe a response C in experimental units possessing trait A, its hard to know if A causes C or if there is some third trait B (present in some of the units) which causes both A and C.
In the case of a controlled experiment, A is now a treatment and not a trait of the units (so in this case you would randomly assign a carnivorous or vegetarian diet to people), thus we can randomly assign A to the units (and assume the randomization means that not every unit having hidden trait B will be given treatment A). In this case we might observe that A and C have no relation, whereas in the observational study we might. (For instance people who choose to be vegetarian may be more focused on health)
An example of how econometricians have dealt with “selection bias” or the fact that observation studies fail to have certain nice properties of controlled experiments is here