Much of the uncertainty in estimating the “success” of these programs lies in not knowing to what degree each of the different social indicators measuring said “success” have already been corrupted, in line with Campbell’s law.
Seeing the phrase “randomized controlled trials” over and over was at least a nice start. Too much social policy debate seems to compare Group A with Policy A and Group B with Policy not-A as if “A vs not-A” wasn’t merely one out of thousands of major economic and cultural difference between the two self-selected groups.
Agreed on RCTs being the gold standard, the top tier of the evidence hierarchy (other than reviews of multiple RCTs).
However, if you choose a biased (corrupted) indicator, all your perfect measurement methodology can only minimize additional biases. A top of the line flood-proof house that you erect on top of a swamp will still go down. (Substitute with weakest-link analogy of your choice.)
Much of the uncertainty in estimating the “success” of these programs lies in not knowing to what degree each of the different social indicators measuring said “success” have already been corrupted, in line with Campbell’s law.
Seeing the phrase “randomized controlled trials” over and over was at least a nice start. Too much social policy debate seems to compare Group A with Policy A and Group B with Policy not-A as if “A vs not-A” wasn’t merely one out of thousands of major economic and cultural difference between the two self-selected groups.
Agreed on RCTs being the gold standard, the top tier of the evidence hierarchy (other than reviews of multiple RCTs).
However, if you choose a biased (corrupted) indicator, all your perfect measurement methodology can only minimize additional biases. A top of the line flood-proof house that you erect on top of a swamp will still go down. (Substitute with weakest-link analogy of your choice.)