I’m a PhD student in Yoshua’s lab. I’ve spoken with him about this issue several times, and he has moved on this issue,
Thank you!
satt
Karma: 4,126
I understand him to be speaking about them increasing 10% more than non-top quartile teachers.
OK, thanks for clarifying. That sounds like a more impressive effect. At the same time, it’s probably still consistent with teacher quality explaining only 10% of the variance in student performance.
I’ll do back-of-envelope arithmetic to demonstrate. The median top-quartile teacher is at the 88th percentile. The median non-top quartile teacher is at the 38th. Suppose, just to allow me to arrive at concrete numbers, teacher quality has a normal distribution. Then the median top-quartile teacher is 1.48 standard deviations better than the median non-top quartile teacher. Now, an R^2 of 10% implies a correlation of sqrt(10%) = 0.23 between teacher quality and pupil performance, so the difference in pupil performance between the median non-top quartile teacher and the median top-quartile teacher is 1.48 * 0.23 = 0.34 standard deviations. That’s a statistically detectable effect, and one that could well translate into 10% higher test scores after a year with the better teachers.
Eg. enough to circumvent the US-Asia difference in two years and also enough to circumvent the Black-White difference in four years as suggested in the answer to the Stackexchange question.
Plausible. If I remember correctly the black/white difference is about 1 standard deviation, so if my estimated effect size of 0.34 SD for good vs. less good teachers is accurate and can be built on year by year, it’s enough to close the black/white difference in 3 years. I don’t know the US-Asia difference but probably the same kind of logic applies.
It’s worth noting that Medical Error is the third leading cause of death in the US http://www.bmj.com/content/353/bmj.i2139
Agreed, medical error is a real & substantial issue. I am just dubious about the ability of some proposals to inexpensively reduce fatal medical error. (But I am optimistic about others. Checklists seem promising.)
The article doesn’t only describe immigration barriers but also barriers of credentialism.
The way I would put it is that the credentialism barriers are the immigration barriers. AFAIK the explicit immigration barriers for foreign doctors looking to enter the US and practice in the US aren’t the bottleneck; the requirement that the doctor do a US residence programme, or a degree from a US school, is a much stronger de facto bar to immigrating.
I agree with your last paragraph.
an RSL in Sydney
To save others from Googling, the RSL appears to be Australia’s Returned and Services League, which licenses clubs as meeting venues, originally for veterans but now also for guests.
Education is a good example of a field where teachers are payed for useless training instead of being payed for producing outcomes for students.
That doesn’t surprise me. Training and degrees are easily observable; outcomes (or, rather, how much of an outcome is attributable to each teacher) are not. It’s harder to pay people based on something less observable.
Given that citing a TED talk as a response to a scientific paper is a bit bad form, I added a question on skeptics to verify the claim.
Thanks!
But I’m not sure it matters. What Bill Gates says in that quotation might well be consistent with what I wrote — it’s hard to be sure because he’s quite vague.
For instance, he says that a teacher in the top quartile increases their class’ performance “by over 10 percent in a single year”. I can believe that, but maybe all it means is that a top-quartile teacher increases their class’ scores by 11% a year, while a bottom-quartile teacher increases their class’ scores by 9% a year. That would hardly refute the idea that teacher effects are small on average!
I think part of the problem of the cited study is likely that it’s done on data from before no-child-left-behind and the efforts of the Gates Foundation.
Maybe. I did a bit more searching on Google Scholar but didn’t uncover a more recent review article with relevant statistics. (I did find one study of a thousand students in HSGI schools in “school year 2013-2014”, which found that 12% of the variance in GPA was at the teacher level.)
However, while the study I cited uses only pre-2004 data, I don’t see much reason to think a newer study would reveal a big increase in teacher-level variance in outcomes. The low proportion of variance attributable to teachers has been true, as far as I know, for as long as people have investigated it (at least 46 years). I’m doubtful that an act which demands high-stakes testing, improvements in average state-wide test scores, and state-wide standards for teachers has changed that, or that a charity has changed that.
It’s also simply possible to find groups of teachers where there’s little variance in teaching skill, that doesn’t mean that differences don’t exist on a larger scale.
It’s possible, but I don’t see evidence that the paper I cited has this flaw. Its new analysis was based on about 100 Tennessee schools included in Project STAR, and the earlier analyses it summarized (table 1) used “samples of poor or minority students”, “nationally representative samples of students”, and “a large sample of public school students in Texas”.
There are treatment that require high skill to administer like Brain surgery and treatments that require less skill like handing over a pill. For the high skill tasks I do think that variance in patient outcomes would decrease.
Fair point — my thought experiment only addresses diagnosis and treatment recommendation, not treatment administration. I think there would be more doctor-level variation among the latter...although not much more in absolute terms.
But even for simply taking a pill a doctor can spend five minutes to hand over the pill or he can spend an hour to talk through the issue with the patient and make sure that the patient has TAP’s to actually take the pills according to the schedule.
Currently there’s no economic reason to spend that hour.
Probably true in most places.
Even if currently everybody spends five minutes for such a patient you would suddenly get variance if you would start to pay some doctors by the outcome instead of simply paying them per visit.
Agreed that you’d get more variance, but I suspect it wouldn’t be much more (subject to this hypothetical’s exact details).
I don’t think that large sample sizes are everything. I think you can learn a lot by looking very carefully at the details of single cases.
This is certainly true. I’ve seen too many stories of people with rare genetic conditions and other diseases successfully working out aetiologies to think otherwise. But those are unusual cases where people tended to invest lots of effort into figuring their cases out. I don’t see them as signs that we can improve the quality-to-cost ratio of healthcare in general by looking very carefully at the details of each case.
Additionally the person who’s the best in city X at treating disease Y for class of patients Z might get more than 40 patients of class Z with Y because everybody wants to be treated by the best.
Sure. But the ratios in my example are more important than the exact numbers.
If all those expensively trained doctors achieve the same outcomes there’s also a question of why we limit their supply as strongly as we are doing it at the present by forcing to them to have the expensive training. In that case the way to go would be to get people with cheaper training to compete in the market of the highly trained doctors.
A good question, and a good answer to the question!
In the absence of performance tracking this won’t be possible because the expensively trained doctors have more prestige.
It’d be immediately possible in the US: eliminate de facto immigration barriers for foreign doctors. Those barriers are, I’d guess, lower in other developed countries (hence why doctors in the UK, Germany, Canada, etc. earn less than US doctors) but I expect doctors’ salaries there could also be reduced a bit by further relaxing immigration restrictions for foreign doctors.
Another option is for patients to go to the cheaper doctors: medical tourism.
For example, if your utility function is log (X), then you will accept the first bet
Not even that. You start with $1 (utility = 0) and can choose between
walking away with $1 (utility = 0), and
accepting a lottery with a 50% chance of leaving you with $0 (utility = −∞) and a 50% chance of having $3 (utility = log(3)).
The first bet’s expected utility is then −∞, and you walk away with the $1.
Why don’t we pay doctors in the present system based on their skills? We can’t measure their skills in the present paradigm, because we can’t easily compare the outcomes of different doctors. Hard patients get send to doctors with good reputations and as a result every doctor has an excuse for getting bad outcomes. In the status quo he can just assert that his patients were hard.
That is one difficulty, but I expect a bigger and more fundamental difficulty is just that there’s lots of random noise in how patients respond to medical treatments.
Thought experiment: suppose every doctor were replaced by identical computers all running the same treatment-recommending software. How much does the variance in patient outcomes decrease? My gut says not very much. If it’s right, most variance isn’t doctor-level, it’s going to be higher-level (at the level of a disease or a hospital/clinic) or lower-level (patient-level).
To me the most obvious analogy is teaching. A standard finding in education research is that classroom/teacher-level variation is only a small part of the variation in educational outcomes. (Doing a quick Google...tables 1 & 5 of this highly-cited paper suggest it’s typically ~ 10% of the variance.) Education, like healthcare, is very expensive, mostly carried out for laypeople by trained specialists, and is generally considered (excepting people promoting their own one-size-fits-all solutions) a really knotty & complex thing to do, so I take the analogy seriously.
This matters because it means individual practitioners are going to have a hard time beating the EBM approach at estimating treatment effects, because the statistical win of assessing treatments at the finer-grained level of the practitioner is going to be more than cancelled out by the statistical loss of each practitioner having a smaller sample to refer to.
Imagine going from a multi-centre study of 40 specialists treating 1,600 people, to each specialist knowing about only their 40 patients. Each specialist then has only 1/40th the information they would’ve had, and that’s going to negate the slight gain of eliminating the effect of different specialists. (The specialists could of course tell each other about their results, but then one’s basically back to the large-scale, expensive EBM-style approach, and the agile, startuppy USP is lost.)
Allowing for doctor-level effects in analysis of treatments could help things, but I predict it would be a small improvement, and an improvement produced by extending the EBM approach, rather than building a parallel track to it.
Following from your quotes above, you could focus your search on systems for which the accuracy of predictions has been poor.
Mmm, reading the post it seems like it’s driving towards “unpredictable” as an operational definition of “complex”. And, reflecting a bit, I reckon that’s not too far from how people actually tend to use the idea of a “complex system”.
The title primed me to upvote the OP, because I think novel, potentially overlooked arguments against promising policies are valuable. However, reading the post from start to finish, I don’t think the core critique is communicated clearly. It seems to go from
the UBI making the “assumption that the state needs its citizens” “more wrong”, to
the UBI intensifying the divide between “contributors and noncontributors”, to
the UBI worsening, “in the long run”, the difficulties of integrating “the economically unnecessary parts of the population into society”, to, in the end
the UBI triggering “unforeseen consequences”, “a whole class of problems that arise out of the changed relationships between citizens, states and economies”
and these four critiques are not the same, and not interchangeable. Moreover, the point where the post ends up (critique 4) is not novel or different to those I’ve seen before. It’s the generic warning of unintended consequences that gets levelled against every public policy proposal ever, basic income included.
While I’m here, I’m uneasy with the handling of the side points as well. Paragraph 2 conflates Saudi Arabia’s population with its citizenry, and these aren’t the same thing. Yes, that sounds like pedantry, and for a lot of countries it would be pedantry, but Saudi Arabia might be literally the worst big country for which to elide the population-citizenry distinction. And the treatment of existing UBI critiques in the last paragraph is unduly generous. The claim that a UBI would “be impossible to get rid of if it is a failure” is very strong, and the notion that a UBI is mutually exclusive of “gradual approaches” puzzles me, since a UBI could certainly be introduced gradually (whether by gradually increasing the UBI allowance from zero, or by introducing the “U” in “UBI” slowly by incrementally expanding the range of people included).
I haven’t downvoted the post either, because I do like the idea of using Discussion as a forum for polishing half-baked arguments about topics popular on LW, and don’t (at the moment) want to discourage that activity.
I’d also like to see downvotes & upvotes shown separately, but want to keep their anonymity.
The big upside of a downvote is that it lets you quickly signal that a comment’s bad without having its poster follow you around afterwards or draw you into an unproductively time-consuming argument. This can of course be abused, but in LW’s one big case of downvote abuse (Eugine_Nier) it didn’t take long to see who was behind it anyway.
Thinking along basically the same lines, I tried to access the actual paper via its DOI link and got redirected to a “Production in progress” page. So we have what looks suspiciously like an embargo!
Might want to clarify whether you literally mean school by “school”, or school & university. In context I’m guessing the first?
I’d be reluctant to go as far as “usually”, but yes, publish-or-perish norms are playing a role here too.
But academics write for other academics, and journalists don’t and can’t. (They’ve tried. They can’t. Remember Vox?)
Would that be Vox, Vox, or Vox?
Edit, 5 minutes later: a bit more seriously, I’m not sure I’d agree that “academics write for other academics” holds as a strong generalization. Many academics focus on writing for academics, but many don’t. I think the (relatively) low level of information flow from academia to general audiences is at least as much a demand-side phenomenon as a supply-side one.
In a lot of cases, yes, although some of the shocked are presumably guilty merely of e.g. following Sam Wang’s forecasts (which gave Clinton 50:1 or better odds) rather than Silver’s.
Not for the first time, a draft paper by Nassim Taleb makes me think, “mathematically, the paper looks fine, and congratulations on knowing calculus, but isn’t this waffling past the issue people are actually interested in?”
Many people who trusted poll-based forecasts were indeed shocked on election day, but not so much because the months-long sequence of forecasts wobbled back & forth too much. They were shocked more because the final forecasts failed to predict the final outcome. And the problem there was the difficulty of polling representative samples of people who’d actually vote, a very different problem to the one Taleb identifies.
I also have a specific question to tie this back to a rationality based framework: When you read Silver (or your preferred reputable election forecaster, I like Andrew Gelman) post their forecasts prior to the election, do you accept them as equal or better than any estimate you could come up with? Or do you do a mental adjustment or discounting based on some factor you think they’ve left out?
In the specific case of the US election, I did use Silver’s forecasts as a guideline, but I also considered a model based on fundamentals (income growth and dead American soldiers), not polls: Doug Hibbs’s Bread & Peace model. Hibbs predicted a 53%-54% split in Clinton’s favour of the Dem/Rep vote. (That 53%-54% prediction might sound pretty poor with hindsight, but the expected error attached to the prediction was 2%, and Clinton has 51%, so it’s within statistical bounds.)
Whether it’s prediction market variations
I didn’t really pay attention to prediction markets. For elections & referendums I don’t believe prediction markets add anything significant beyond polls & fundamentals.
or adjustments based on perceiving changes in nationalism or politician specific skills
I didn’t consciously adjust on that basis. As far as I know, ordinary, retrospective economic voting played a big role in explaining even “the Extraordinary Election of Adolf Hitler”, so I figured I wouldn’t bother putting my thumb on the scale because of vigorous nationalism or charisma.
Upvoted your post as encouragement to post more stuff like this. I shrugged my shoulders at Taleb’s paper, but the topic’s fascinating and I’d like to see more stuff like Taleb’s paper, even if the paper itself is unimpressive.
This comment brought to you by sarahconstantin’s and AnnaSalamon’s Your Wannabe Rationalist Community Blog Needs You! posts. Without them I’d likely have carried on my 11-month streak of not posting.
I don’t think it’s going to be very difficult to discover Eugene’s new account once he makes it. The real difficulty is making it not worth his while to keep coming back.
Seconded.
I don’t count myself as either a rationalist or a community member here,
FWIW, I think of anyone who posts here regularly as a Wronger! (I know, I know, you disagree with other people here about how to do causal inference and about the insightfulness/worthiness of academics — but disagreeing with the rest of the gang on some specific topic is pretty common, I reckon, and not nearly enough to get you kicked out of the treehouse.)
I think you guys should find a way to throw the nrx out, and let them start their own community. I think they are going to do more harm than good in the long run.
This I disagree with. The only neoreactionaries I remember being obnoxious enough here to raise a real stink are Eugine_Nier and Jim, and Jim hasn’t posted here since 2012. That’s too thin a basis for kicking out a particular political group, especially since Eugine_Nier being here has had some benefit. (I have occasionally seen them shake people out of misconceptions.) It’s just that Eugine_Nier’s abuse of the voting system outweighed/outweighs that benefit. (That wasn’t Eugine_Nier’s only downside, but it was the big one.)
I don’t think HungryHobo meant it to go anywhere in particular. It looks like a dummy link with the spoiler in its
titleattribute so the spoiler appears when the link’s hovered over.
LW used to be politically neutral; I’m not sure it is so anymore.
I reckon LW’s more politically neutral (in kilobug’s apparent sense) now than it was in 2014, but that that’s mainly attributable to the fall in traffic. (I’d guess LW’s less politically-neutral-in-the-apparent-kilobug-sense than I felt it was in 2013.)
(I also think that as I understand the term “politically neutral”, LW was never politically neutral, because the idea of a politically neutral institution is probably incoherent in the first place. I upvoted kilobug anyway because I think they mean something else by “politically neutral”, and in any case I agree with the rest of their comment.)
And the current wave of populist, nationalistic, libertarian/individualist ideology which seems to be very popular in the USA is being represented in the general atmosphere of LW.
LW’s always had a big, big libertarian/individualist streak. In the first survey back in ’09, a plurality (45%) called themselves libertarians. That’s actually been diluted over the years as more conventional left-wingers have drifted in.
Although if we go looking for descendants rather than ancestors...
Irène Joliot-Curie (12 September 1897 – 17 March 1956) was a French scientist, the daughter of Marie Curie and Pierre Curie and the wife of Frédéric Joliot-Curie. Jointly with her husband, Joliot-Curie was awarded the Nobel Prize for Chemistry in 1935 for their discovery of artificial radioactivity. This made the Curies the family with the most Nobel laureates to date.[1] Both children of the Joliot-Curies, Hélène and Pierre, are also esteemed scientists.[2]
Assuming that
the US does in fact hold a nationwide presidential election in 2020,
the Democratic & Republican parties get > 90% of the votes in 2020, and
US military fatalities in new, unprovoked, foreign wars are minimal (< 3000),
I predict:
a Trump landslide with probability 20%, assuming 2% weighted growth over Trump’s term
a Trump landslide with probability 70%, assuming 3% weighted growth over Trump’s term
a Trump landslide with probability 95%, assuming 4% weighted growth over Trump’s term
using your definition of landslide, and defining “weighted growth” as annualized growth in quarterly, inflation-adjusted, disposable, personal income per capita, with recent growth weighted more heavily with a discount factor λ of 0.9. (See Hibbs for more details.) This is a clear prediction.
I’m more doubtful of chatter about rising political polarization than I am about fundamentals-based models of voting, and those models highlight the economy & war as the factors that most matter. As such I reckon sufficient economic prosperity could in fact produce a landslide for Trump (and virtually any incumbent, really).