For instance, he says that a teacher in the top quartile increases their class’ performance “by over 10 percent in a single year”. I can believe that, but maybe all it means is that a top-quartile teacher increases their class’ scores by 11% a year, while a bottom-quartile teacher increases their class’ scores by 9% a year. That would hardly refute the idea that teacher effects are small on average!
I understand him to be speaking about them increasing 10% more than non-top quartile teachers.
Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.
It’d be immediately possible in the US: eliminate de facto immigration barriers for foreign doctors.
The article doesn’t only describe immigration barriers but also barriers of credentialism. The non-US degree often isn’t enough to work in the US as a doctor because it’s quality is in doubt.
If there would be hard evidence that those doctors perform as well as US doctors the ability to seek rents via credentialism will be reduced.
Maybe. I did a bit more searching on Google Scholar but didn’t uncover a more recent review article with relevant statistics. (I did find one study of a thousand students in HSGI schools in “school year 2013-2014”, which found that 12% of the variance in GPA was at the teacher level.)
I looked at that study. It seems their best predictors were e (i) Fall algebra EOC (End Of Course) scores,
(ii) English language learner (ELL) status, (iii) Black student status, and (iv) Hispanic student status.
Of course you do a better prediction of the student performance for a standardized test when you look at the last standardized test they took than when you look at whether or not they had a very good teacher for a single school year.
The 12% variance under that setting might be compatible with the claims that Gates makes. 12% variance per year might compound over multiple years to bridge the gap between the US and Asia in two years and US Black White gap in 4 years.
I understand him to be speaking about them increasing 10% more than non-top quartile teachers.
OK, thanks for clarifying. That sounds like a more impressive effect. At the same time, it’s probably still consistent with teacher quality explaining only 10% of the variance in student performance.
I’ll do back-of-envelope arithmetic to demonstrate. The median top-quartile teacher is at the 88th percentile. The median non-top quartile teacher is at the 38th. Suppose, just to allow me to arrive at concrete numbers, teacher quality has a normal distribution. Then the median top-quartile teacher is 1.48 standard deviations better than the median non-top quartile teacher. Now, an R^2 of 10% implies a correlation of sqrt(10%) = 0.23 between teacher quality and pupil performance, so the difference in pupil performance between the median non-top quartile teacher and the median top-quartile teacher is 1.48 * 0.23 = 0.34 standard deviations. That’s a statistically detectable effect, and one that could well translate into 10% higher test scores after a year with the better teachers.
Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.
Plausible. If I remember correctly the black/white difference is about 1 standard deviation, so if my estimated effect size of 0.34 SD for good vs. less good teachers is accurate and can be built on year by year, it’s enough to close the black/white difference in 3 years. I don’t know the US-Asia difference but probably the same kind of logic applies.
Agreed, medical error is a real & substantial issue. I am just dubious about the ability of some proposals to inexpensively reduce fatal medical error. (But I am optimistic about others. Checklists seem promising.)
The article doesn’t only describe immigration barriers but also barriers of credentialism.
The way I would put it is that the credentialism barriers are the immigration barriers. AFAIK the explicit immigration barriers for foreign doctors looking to enter the US and practice in the US aren’t the bottleneck; the requirement that the doctor do a US residence programme, or a degree from a US school, is a much stronger de facto bar to immigrating.
I am just dubious about the ability of some proposals to inexpensively reduce fatal medical error. (But I am optimistic about others. Checklists seem promising.)
In the present system there aren’t strong economic incentives to reduce medical error. If you consider Checklists to be promising, then the lack of any economic incentives to use their virtues might be part of the reason why they don’t get adopted.
The incentive system of doing procedures that can be billed because they are included in a list of billable procedures and doing them in a defensive way that survives a lawsuit is bad. It means that money is wasted for procedures that cost a lot of money and provide little benefit. It also means that policies such as checklists (if we grant them to work) don’t get incentivised.
The whole system is unable to incentivise cheap solutions. Scott’s post about the inability of a hospital to prescribe Melatonin to it’s patients is illustrative:
This is why the story of Ramelteon scares me so much – not because it’s a bad drug, because it isn’t. But because one of the most basic and useful human hormones got completely excluded from medicine just because it didn’t have a drug company to push it. And the only way it managed to worm its way back in was to have a pharmaceutial company spend a decade and several hundred million dollars to tweak its chemical structure very slightly, patent it, and market it as a hot new drug at a 2000% markup.
The way I would put it is that the credentialism barriers are the immigration barriers.
From a political perspective immigration and credentialism are two different subjects, you have to convince different constituencies to create change.
In the present system there aren’t strong economic incentives to reduce medical error. [etc.]
I think this is broadly correct, certainly in the case of the US medical system.
From a political perspective immigration and credentialism are two different subjects, you have to convince different constituencies to create change.
Yes, from the standpoint of effecting political change, one might have to treat them as two different subjects, even though w.r.t. doctors in the US the two greatly overlap.
I understand him to be speaking about them increasing 10% more than non-top quartile teachers.
Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.
It’s worth noting that Medical Error is the third leading cause of death in the US http://www.bmj.com/content/353/bmj.i2139
The article doesn’t only describe immigration barriers but also barriers of credentialism. The non-US degree often isn’t enough to work in the US as a doctor because it’s quality is in doubt. If there would be hard evidence that those doctors perform as well as US doctors the ability to seek rents via credentialism will be reduced.
I looked at that study. It seems their best predictors were e (i) Fall algebra EOC (End Of Course) scores, (ii) English language learner (ELL) status, (iii) Black student status, and (iv) Hispanic student status.
Of course you do a better prediction of the student performance for a standardized test when you look at the last standardized test they took than when you look at whether or not they had a very good teacher for a single school year. The 12% variance under that setting might be compatible with the claims that Gates makes. 12% variance per year might compound over multiple years to bridge the gap between the US and Asia in two years and US Black White gap in 4 years.
OK, thanks for clarifying. That sounds like a more impressive effect. At the same time, it’s probably still consistent with teacher quality explaining only 10% of the variance in student performance.
I’ll do back-of-envelope arithmetic to demonstrate. The median top-quartile teacher is at the 88th percentile. The median non-top quartile teacher is at the 38th. Suppose, just to allow me to arrive at concrete numbers, teacher quality has a normal distribution. Then the median top-quartile teacher is 1.48 standard deviations better than the median non-top quartile teacher. Now, an R^2 of 10% implies a correlation of sqrt(10%) = 0.23 between teacher quality and pupil performance, so the difference in pupil performance between the median non-top quartile teacher and the median top-quartile teacher is 1.48 * 0.23 = 0.34 standard deviations. That’s a statistically detectable effect, and one that could well translate into 10% higher test scores after a year with the better teachers.
Plausible. If I remember correctly the black/white difference is about 1 standard deviation, so if my estimated effect size of 0.34 SD for good vs. less good teachers is accurate and can be built on year by year, it’s enough to close the black/white difference in 3 years. I don’t know the US-Asia difference but probably the same kind of logic applies.
Agreed, medical error is a real & substantial issue. I am just dubious about the ability of some proposals to inexpensively reduce fatal medical error. (But I am optimistic about others. Checklists seem promising.)
The way I would put it is that the credentialism barriers are the immigration barriers. AFAIK the explicit immigration barriers for foreign doctors looking to enter the US and practice in the US aren’t the bottleneck; the requirement that the doctor do a US residence programme, or a degree from a US school, is a much stronger de facto bar to immigrating.
I agree with your last paragraph.
In the present system there aren’t strong economic incentives to reduce medical error. If you consider Checklists to be promising, then the lack of any economic incentives to use their virtues might be part of the reason why they don’t get adopted.
The incentive system of doing procedures that can be billed because they are included in a list of billable procedures and doing them in a defensive way that survives a lawsuit is bad. It means that money is wasted for procedures that cost a lot of money and provide little benefit. It also means that policies such as checklists (if we grant them to work) don’t get incentivised.
The whole system is unable to incentivise cheap solutions. Scott’s post about the inability of a hospital to prescribe Melatonin to it’s patients is illustrative:
This is why the story of Ramelteon scares me so much – not because it’s a bad drug, because it isn’t. But because one of the most basic and useful human hormones got completely excluded from medicine just because it didn’t have a drug company to push it. And the only way it managed to worm its way back in was to have a pharmaceutial company spend a decade and several hundred million dollars to tweak its chemical structure very slightly, patent it, and market it as a hot new drug at a 2000% markup.From a political perspective immigration and credentialism are two different subjects, you have to convince different constituencies to create change.
I think this is broadly correct, certainly in the case of the US medical system.
Yes, from the standpoint of effecting political change, one might have to treat them as two different subjects, even though w.r.t. doctors in the US the two greatly overlap.