Update: The editors of the Journal of Clinical Epidemiology have now rejected my second letter to the editor, and thus helped prove Eliezer’s point about four layers of conversation.
Anders_H
Karma: 1,297
The New Riddle of Induction: Neutral and Relative Perspectives on Color
Why do you think two senior biostats guys would disagree with you if it was obviously wrong? I have worked with enough academics to know that they are far far from infallible, but curious on your analysis of this question.
Good question. I think a lot of this is due to a cultural difference between those of us who have been trained in the modern counterfactual causal framework, and an old generation of methodologists who felt the old framework worked well enough for them and never bothered to learn about counterfactuals.
I wrote this on my personal blog; I was reluctant to post this to Less Wrong since it is not obviously relevant to the core interests of LW users. However, I concluded that some of you may find it interesting as an example of how the academic publishing system is broken. It is relevant to Eliezer’s recent Facebook comments about building an intellectual edifice.
Odds ratios and conditional risk ratios
I wrote this on my personal blog; I was reluctant to post this to Less Wrong since it is not obviously relevant to the core interests of LW users. However, I concluded that some of you may find it interesting as an example of how the academic publishing system is broken. It is relevant to Eliezer’s recent Facebook comments about building an intellectual edifice.
VortexLeague: Can you be a little more specific about what kind of help you need?
A very short, general introduction to Less Wrong is available at http://lesswrong.com/about/
Essentially, Less Wrong is a reddit-type forum for discussing how we can make our beliefs more accurate.
Thank you for the link, that is a very good presentation and it is good to see that ML people are thinking about these things.
There certainly are ML algorithms that are designed to make the second kind of predictions, but generally they only work if you have a correct causal model
It is possible that there are some ML algorithms that try to discover the causal model from the data. For example, /u/IlyaShpitser works on these kinds of methods. However, these methods only work to the extent that they are able to discover the correct causal model, so it seems disingenious to claim that we can ignore causality and focus on “prediction”.
I skimmed this paper and plan to read it in more detail tomorrow. My first thought is that it is fundamentally confused. I believe the confusion comes from the fact that the word “prediction” is used with two separate meanings: Are you interested in predicting Y given an observed value of X (Pr[Y | X=x]), or are you interested in predicting Y given an intervention on X (i.e. Pr[Y|do(X=x)]).
The first of these may be useful for certain purposes. but If you intend to use the research for decision making and optimization (i.e. you want to intervene to set the value of X , in order to optimize Y), then you really need the second type of predictive ability, in which case you need to extract causal information from the data. This is only possible if you have a randomized trial, or if you have a correct causal model.
You can use the word “prediction” to refer to the second type of research objective, but this is not the kind of prediction that machine learning algorithms are designed to do.
In the conclusions, the authors write:
“By contrast, a minority of statisticians (and most machine learning researchers) belong to the “algorithmic modeling culture,” in which the data are assumed to be the result of some unknown and possibly unknowable process, and the primary goal is to find an algorithm that results in the same outputs as this process given the same inputs. ”
The definition of “algorithmic modelling culture” is somewhat circular, as it just moves the ambiguity surrounding “prediction” to the word “input”. If by “input” they mean that the algorithm observes the value of an independent variable and makes a prediction for the dependent variable, then you are talking about a true prediction model, which may be useful for certain purposes (diagnosis, prognosis, etc) but which is unusable if you are interested in optimizing the outcome.
If you instead claim that the “input” can also include observations about interventions on a variable, then your predictions will certainly fail unless the algorithm was trained in a dataset where someone actually intervened on X (i.e. someone did a randomized controlled trial), or unless you have a correct causal model.
Machine learning algorithms are not magic, they do not solve the problem of confounding unless they have a correct causal model. The fact that these algorithms are good at predicting stuff in observational datasets does not tell you anything useful for the purposes of deciding what the optimal value of the independent variable is.
In general, this paper is a very good example to illustrate why I keep insisting that machine learning people need to urgently read up on Pearl, Robins or Van der Laan. The field is in danger of falling into the same failure mode as epidemiology, i.e. essentially ignoring the problem of confounding. In the case of machine learning, this may be more insidious because the research is dressed up in fancy math and therefore looks superficially more impressive.
Thanks for catching that, I stand corrected.
The rational choice depends on your utility function. Your utility function is unlikely to be linear with money. For example, if your utility function is log (X), then you will accept the first bet, be indifferent to the second bet, and reject the third bet. Any risk-averse utility function (i.e. any monotonically increasing function with negative second derivative) reaches a point where the agent stops playing the game.
A VNM-rational agent with a linear utility function over money will indeed always take this bet. From this, we can infer that linear utility functions do not represent the utility of humans.
(EDIT: The comments by Satt and AlexMennen are both correct, and I thank them for the corrections. I note that they do not affect the main point, which is that rational agents with standard utility functions over money will eventually stop playing this game)
Because I didn’t perceive a significant disruption to the event, I was mentally bucketing you with people I know who severely dislike children and would secretly (or not so secretly) prefer that they not attend events like this at all; or that they should do so only if able to remain silent (which in practice means not at all.) I suspect Anders_H had the same reaction I did.
Just to be clear, I did not attend Solstice this year, and I was mentally reacting to a similar complaint that was made after last year’s Solstice event. At last year’s event, I did not perceive the child to be at all noteworthy as a disturbance. From reading this thread, it seems that the situation may well have been different this year, and that my reaction might have been different if I had been there. I probably should not have commented without being more familiar with what happened at this year’s event.
I also note that my thinking around this may very well be biased, as I used to live in a group house with this child.
While I understand that some people may feel this way, I very much hope that this sentiment is rare. The presence of young children at the event only adds to the sense of belonging to a community, which is an important part of what we are trying to “borrow” from religions.
I’d like each user to have their own sub domain (I.e such that my top level posts can be accessed either from Anders_h.lesswrong.com or from LW discussion). If possible it would be great if users could customize the design of their sub domain, such that posts look different when accessed from LW discussion.
Given that this was posted to LW, you’d think this link would be about a different equation..
Is Caviar a Risk Factor For Being a Millionaire?
The one-year embargo on my doctoral thesis has been lifted, it is now available at https://dash.harvard.edu/bitstream/handle/1/23205172/HUITFELDT-DISSERTATION-2015.pdf?sequence=1 . To the best of my knowledge, this is the first thesis to include a Litany of Tarski in the introduction.
Upvoted. I’m not sure how to phrase this without sounding sycophantic, but here is an attempt: Sarah’s blog posts and comments were always top quality, but the last couple of posts seem like the beginning of something important, almost comparable to when Scott moved from squid314 to Slatestarcodex.
Today, I uploaded a sequence of three working papers to my website at https://andershuitfeldt.net/working-papers/
This is an ambitious project that aims to change fundamental things about how epidemiologists and statisticians think about choice of effect measure, effect modification and external validity. A link to an earlier version of this manuscript was posted to Less Wrong half a year ago, the manuscript has since been split into three parts and improved significantly. This work was also presented in poster form at EA Global last month.
I want to give a heads up before you follow the link above: Compared to most methodology papers, the mathematics in these manuscripts is definitely unsophisticated, almost trivial. I do however believe that the arguments support the conclusions, and that those conclusions have important implications for applied statistics and epidemiology.
I would very much appreciate any feedback. I invoke “Crocker’s Rules” (see http://sl4.org/crocker.html) for all communication regarding these papers. Briefly, this means that I ask you, as a favor, to please communicate any disagreement as bluntly and directly as possible, without regards to social conventions or to how such directness may affect my personal state of mind.
I have made a standing offer to give a bottle of Johnnie Walker Blue Label to anyone who finds a flaw in the argument that invalidates the paper, and a bottle of 10-year old Single Scotch Malt to anyone who finds a significant but fixable error; or makes a suggestion that substantially improves the manuscript.
If you prefer giving anonymous feedback, this can be done through the link http://www.admonymous.com/effectmeasurepaper .
I am not sure I fully understand this comment, or why you believe my argument is circular. It is possible that you are right, but I would very much appreciate a more thorough explanation.
In particular, I am not “concluding” that humans were produced by an evolutionary process; but rather using it as background knowledge. Moreover, this statement seems uncontroversial enough that I can bring it in as a premise without having to argue for it.
Since “humans were produced by an evolutionary process” is a premise and not a conclusion, I don’t understand what you mean by circular reasoning.