A request for the “argument from authority” fallacy. Freethinkers discuss ideas directly on their merits, not on their author’s job description. A rationalist doesn’t ignore any evidence, of course (even authors’ job descriptions), but try to weight them accurately, okay?
“Chances are, if you’re a statistics/data science practitioner with a few years of experience applying different techniques to different problems and different data sets, and you have some general intuitions about which techniques apply better to which domains, you’re probably doing this in a Bayesian way.”
Freethinkers who might know a little bit about statistics/data science would, presumably by your lights, stop reading here.
I have seen a lot of this type of overconfident stuff out of LW over the years. That is, in Bayesian terms, I already had a prior, and it already updated in a “read a textbook and stop blogging” direction.
In general, there is no good way for me to know what your prior is on the subject I’m going to write, unless I already knew you really well. But it’s unreasonable to expect me to know what all of my audiences’ priors are and try to write something that agrees with them. I don’t think it’s possible. I’m not optimizing for the argmax of the less wrong commenters anyway. I want to write something that’s probably wrong, so I can update from it if I need to. Your kind of responses, and saying “read a textbook and stop blogging” seems to go against the spirit of free thought and debate anyway. Personally, I think the tendency of people to respond in that way is probably why communities like these dissipate after a while. I wrote a post about this subject as well.
Why write about technical stuff you don’t really know very well? How did you expect that to go?
I find this attitude of “lets blog about stuff that’s probably wrong with the aim of updating later when people correct you” kind of a weird way to go. Why not just go and read directly about what you want to learn about?
Your way helps you, of course, but generates negative externalities for everybody else (who have to either spend time correcting you if they know better, or get misled by you if they don’t).
Wasn’t scholarship (e.g. reading stuff) one of the virtues?
At what point do you consider yourself to have read enough? At what point do you decide that you’ve read enough textbooks and now it’s ok to blog?
Also, I don’t see how this creates negative externalities on people. That kind of assumes a bizarre situation where people are either forced to read, or forced to respond to things. Apply that reasoning to basically the entire internet, social media, every day discussions with people, and you basically have to quarantine yourself from most of the world to avoid that risk. Or you conclude that all speech has to be meticulously curated so that there is very low risk of misleading, offending, upsetting, or otherwise wasting someone’s time.
“At what point do you consider yourself to have read enough?”
How about a single statistics class at a university. At that point one might appreciate the set of things one might not know about yet. In reality, though, I feel that if you want to blog about technical topics, you should be an expert on said technical topics. If you are not an expert, it seems you should listen, not talk.
“Also, I don’t see how this creates negative externalities on people.”
Conditional on you being wrong, you should expect no negative externalities only if you expect the activity of blogging to be akin to pissing into the wind—you don’t expect folks to take you seriously anyways. If you are not an expert yet, you should not be very confident about avoiding being wrong on technical topics.
If folks do take you seriously, they either get the wrong idea, or have to spend energy correcting you, or leave it alone, and let you mislead others.
How about a single statistics class at a university.
Not sure why you would say this (assuming I haven’t even done that) and then immediately admit that you expect something much higher. What that level of expertise is I’m not sure, but probably having a Ph.D in statistics?
I have an undergraduate degree in math / physics, and I’ve been working at a data science job for 3 years, while spending most of my free time studying these subjects. I wouldn’t call myself an expert, but at least personally, I think I’ve reached a point where I can feasibly have discussions with people about statistics / ML, and not say things that are totally far off from where at least a certain mode of experts are on the subject.
Of course, the topic I was discussing is actually somewhere on the border of statistics, mathematics, and philosophy, and my guess is there are few academic programs that focus specifically on that overlapping region. That makes it very unlikely for anyone on this site to be at the level of expertise you demand. And if the subject is really that esoteric, it also makes it more unlikely that someone would somehow damagingly misuse what they read here. There are no infohazards (as far as I know) in my post, and there really aren’t any concrete suggestions for actions to take, either.
Maybe there is a cultural/generational difference here.
I have seen very little on Bayes out of LW over the years I agree with—take it as a datapoint if you wish. Most of it is somewhere between at least somewhat wrong and not even wrong.
Hanson had a post somewhere on how folks should practice holding strong opinions and arguing for them, but not taking the whole thing very seriously. Maybe that’s what you are doing.
LessWrong has tended towards skepticism (though not outright rejection) of academic credentials ( consider Eliezer’s “argument trumps authority” discussions in the Sequences). However, this site is more or less a place for somewhat informal intellectual discussion. It is not an authoritative information repository, and as far as I can tell, does not claim to be. Anyone who participates in discussions here is probably well aware of this fact, and is fully expected to be able to consider the arguments here, not take them at face value.
If you disagree with some of the core ideas around this community (like Bayesian epistemology), as well as what you perceive to be the “negative externalities” of the tendency towards informal / non-expert discussion, then to me it seems likely that you disagree with certain aspects of the culture here. But you seem to have chosen to oppose those aspects, rather than simply choosing not to participate.
I don’t really have time to “oppose” in the sense you mean, as that’s a full time job. But for the record this aspect of LW culture is busted, I think.
“somewhat informal intellectual discussion”
All I am saying is, if you are going to talk about technical topics, either: (a) know what you are talking about, or (b) if you don’t or aren’t sure, maybe read more and talk less, or at least put disclaimers somewhere. That’s at least a better standard than what [university freshmen everywhere] are doing.
If you think you know what you are talking about, but then someone corrects you on something basic, heavily update towards (b).
I try to adhere to this also, actually—on technical stuff I don’t know super well. Which is a lot of stuff.
The kind of meaningless trash talk MrMind is engaged in above, I find super obnoxious.
All I am saying is, if you are going to talk about technical topics, either: (a) know what you are talking about, or (b) if you don’t or aren’t sure, maybe read more and talk less, or at least put disclaimers somewhere. That’s at least a better standard than what [university freshmen everywhere] are doing.
But this is a philosophical position you’re taking. You’re not just explaining to us what common decency and protocol should dictate—you’re arguing for a particular conception of discourse norms you believe should be adopted. And probably, in this community, a minority position at that. But, the way that you have stated this comes across like you think your position is obvious, to the point where it’s not really worth arguing for. To me, it doesn’t seem so obvious. Moreover, if it’s not obvious, and if you were to follow your own guidelines fully, you might decide to leave that argument up to the professional, fully credentialed philosophers.
Anyway, what you are bringing up is worth arguing about in my opinion. LW may be credential-agnostic, but it also would be beneficial to have some way of knowing which arguments carry the most weight, and what information is deemed the most reliable—while also allowing people of all levels of expertise to discuss it freely. Such a problem is very difficult, but I think following your principle of “only experts talk, non-experts listen” is sort of extreme and not really appropriate outside of classrooms and lecture halls.
I am saying there is a very easy explanation on why the stats community moved on and LW is still talking about this: LW’s thinking on this is “freshman level.”
I don’t think “know what you are talking about” is controversial, but perhaps I am just old.
I think it’s ok for non-experts to talk, I just think they need to signal stuff appropriately. Wikipedia has a similar problem with non-expert and expert talk being confused, which is why it’s not seen as a reliable source on technical topics.
Being “credential-agnostic” is sort of being a bad Bayesian—you should condition on all available evidence if you aren’t certain of claims (and you shouldn’t be if you aren’t an expert). Argument only screens authority under complete certainty.
Non-experts may not know the boundary of their own knowledge, and may also have trouble knowing where the boundaries of the knowledge of others are as well.
In fact, I think that quite frequently even experts have trouble knowing the extent of their own expertise. You can find countless examples of academics weighing in on matters they aren’t really qualified for. I think this is a particularly acute problem in the philosophy of science. This is a problem I had a lot when I read books by authors of pop-sci / pop-philosophy. They sure seem like experts to the non-initiated. I attribute this mainly to them becoming disconnected from academia and living in a bubble containing mostly just them and their fans, who don’t offer much in the way of substantive disagreement. But this is one of the reasons I value discussion so highly.
When I began writing this post, I did not honestly perceive my level of knowledge to be at the “freshman” level. As I’ve mentioned before, many of the points are re-hashes of stuff from people like Jaynes, and although I might have missed some of his subtle points, is there any good way for me to know that he represents a minority or obsolete position without being deeply familiar with the aspects of that field, as someone with decades of experience would?
The simplest solution is just to read until I have that level of experience with the topics as measured by actual time spent on it, but I feel like that would come at the very high cost of not being able to participate in online discussions, which are valuable. But even then, I probably would still not know where my limits are until I bump into opposing views, which would need to occur through discussion.
You can find countless examples of academics weighing in on matters they aren’t really qualified for.
Yes, absolutely. See also SMBC’s “send in the bishops, they can move diagonally” (chess masters on the Iraq war).
is there any good way for me to know that he represents a minority or obsolete position.
I don’t know if Jaynes represents a minority position (there are a lot of Bayesian statisticians). It’s more like the field moved on from this argument to more interesting arguments. Basically smart Bayesians and frequentists mostly understood each other’s arguments, and considered them mostly valid.
This is the type of B vs F argument people have these days (I linked this here before):
If you really want the gory details, you can also read the Robins/Ritov paper. But it’s a hard paper.
Full disclosure: Robins was my former boss, and I am probably predisposed to liking his stuff.
Re: “what’s a good way to know”: I would say ask experts. Stat profs love talking about this stuff, you can email your local one, and try to go for coffee or something.
Re: “freshman level,” this was perhaps uncharitable phrasing. I just perceive, perhaps incorrectly, a lot of LW discussions as the type of discussion that takes place in dorms everywhere.
This is the type of B vs F argument people have these days (I linked this here before):
I skimmed this a bit, and it seems like the argument went several rounds but was never actually resolved in the end? See Chris Sim’s last comment here which Robins and Wasserman apparently never responded to. Also, besides this type of highly technical discussion, can you point us to some texts that explains the overall history and current state of the F vs B debate in the professional stats community? I’d like to understand how and why they moved on from the kinds of discussion that LW is still having.
There is a recent bookComputer Age Statistical Inference by Efron and Hastie (who are well-respected statisticians). They start by distinguishing three kinds of statistics—frequentist (by which they mean Neyman and Pearson with some reliance on Fisher); Bayesian which everybody here knows well; and Fisherian by which they mean mostly maximum likelihood and derivatives. They say that Fisher, though the was dismissive of the Bayesian approach, didn’t fully embrace the frequentism either and blazed his own path somewhere in the middle.
We can ask Chris and Larry (I can if/when I see them).
My take on the way this argument got resolved is that Chris and Larry/Jamie agree on the math—namely that to “solve” the example using B methods we need to have a prior that depends on pi. The possible source of disagreement is interpretational.
Larry and Jamie think that this is Bayesians doing “frequentist pursuit”, that is using B machinery to mimic a fundamentally F behavior. As they say, there is nothing wrong with this, but the B here seems extraneous. Chris probably doesn’t see it that way, he probably thinks this is the natural way to do this problem in a B way.
The weird thing about (what I think) Chris’ position here is that this example violates the “likelihood principle” some Bayesians like. The likelihood principle states that all information lives in the likelihood. Of course here the example is set up in such a way that the assignment probably pi(X) is (a) not a part of the likelihood and (b) is highly informative. The natural way for a Bayesian to deal with this is to stick pi(X) in the prior. This is formally ok, but kind of weird and unnatural.
How weird and unnatural it is is a matter of interpretation, I suppose.
This example is very simple, there are much more complicated versions of this. For example, what if we don’t know pi(X), but have to model it? Does pi(X) still go into the prior? That way lie dragons...
I guess my point is, these types of highly technical discussions are the discussions that professionals have if B vs F comes up. If this is too technical, may I ask why even get into this? Maybe this level of technicality is the natural point of technicality for this argument in this, the year of our Lord 2017? This is kind of my point, if you aren’t a professional, why are you even talking about this?
It’s a good question about a history text on B vs F. Let me ask around.
edit: re: dragons, I guess what I mean is, it seems most things in life can be phrased in F or B ways. But there are a lot of phenomena for which the B phrasing, though it exists, isn’t really very clarifying. These might include identification and model misspecification issues. In such cases the B phrasing just feels like carrying around ideological baggage.
My philosophy is inherently multiparadigm—you use the style of kung fu that yields the most benefit or the most clarity for the problem. Sometimes that’s B and sometimes that’s F and sometimes that’s something else. I guess in your language that would be “instrumental rationality in data analysis.”
I don’t think that having a conversation with someone who’s wrong is necessarily bad for myself. Arguing against someone who’s wrong can help me to clarify my own thoughts on a topic.
CFAR supports the notion that one of the best ways to learn is to teach. Mixing reading textbooks passively with active argument is good for learning a subject well.
What did you expect with “Very partisan / opinionated”? I don’t think that’s how the average academic expert would preface his professional position if academics would be in the habit of stating the epistemic status.
I was hoping that “very partisan” would signal that I recognize there are a sizable chunk of people with very different views on the subject, and that recognition indicates some kind of epistemic humility. I was wrong about that, and in the future I’ll try to indicate that more explicitly.
Ilya is a student and coauthor of Judea Pearl, whose work on causality and Bayes nets was cited by Eliezer many times. He’s an expert at the stuff that LW is amateuring in.
Psychologists are not statisticians, though. Generally they are relatively naive users of stats methods (as are a lot of other applied folks, e.g. doctors that publish, cognitive scientists, social scientists, epidemiologists, etc.) Ideally, methods folks and applied folks collaborate, but this does not always happen.
You can fish for positive findings with B methods just fine, the issue isn’t F vs B, the issue is bad publication incentives.
There is also a little bit of “there is a huge replication crisis on, long story short, we should read this random dude’s blog (with apologies to the OP).”
Pearl is, apparently, only half Bayesian.
I am wrong a lot—I can point you to some errors in my papers if you want.
The replication crysis is decomposable into many pieces, two of which are surely bad incentives and relative inexperience of the “applied folks”. Another though is, that’s the main point, that frequentist methods are a set of ad-hoc, poorly explained, poorly understood heuristics. No wonder that they are used improperly. On the other hand, I’ve seen the crysis explained mostly by Bayesian statisticians, so I’m possibly in a bubble. If you can point me to a frequentist explanation I would be glad to pop it.
I am wrong a lot—I can point you to some errors in my papers if you want.
Apparently though, cousin_it thinks you cannot be criticized or argued against...
“Another though[t] is, that’s the main point, that frequentist methods are a set of ad-hoc, poorly explained, poorly understood heuristics.”
I don’t think so. This is what LW repeatedly gets wrong, and I am kind of tired of talking about it. How are you so confident re: what frequentist methods really are about, if you aren’t a statistician? This is incredibly bizarre to me.
Rather than argue about it constantly, which I am very very tired of doing (see above “negative externalities”), I can point you to Larry Wasserman’s book “All of Statistics.” It’s a nice frequentist book. Start there, perhaps. Larry is very smart, one of the smartest statisticians alive, I think.
Apparently though, cousin_it thinks you cannot be criticized or argued against...
My culture thrives on peer review, as much as we grumble about it. Emphasis on “peer,” of course.
You should probably be a bit more charitable to cousin_it, he’s very smart too.
what frequentist methods really are about, if you aren’t a statistician?
I was under the impression that it was sufficient to read statistics books. Apparently though, you need also to be anointed by another statistician to even talk about the subject.
My culture thrives on peer review, as much as we grumble about it. Emphasis on “peer,” of course.
You seem to imply that no statistician has ever criticized frequentist methods. LW is just parroting what others, more expert men already said.
You should probably be a bit more charitable to cousin_it, he’s very smart too.
Isn’t it, as long as you’re making an incorrect statement, irrelevant how intelligent you are? Jaynes was wrong about quantum mechanics. Einstein was wrong about the unified field. Everybody can be wrong, no matter how respected or intelligent they are.
“I was under the impression that it was sufficient to read statistics books.”
Ok, what have you read?
I am not the “blogging police,” I am just saying, based on past experience, that when people who aren’t statisticians talk about these issues, the result is very low quality. So low that it would have been better to stay silent. Statistics is a very mathematical field. These types of arguments are akin to “should we think about mathematics topologically or algebraically?”
“You seem to imply that no statistician has ever criticized frequentist methods.”
The pretty standard Bayesian curriculum: De Finetti, Jaynes-Bretthorst, Sivia.
See “Tom Knight and the LISP machine”:
I love Lisp koans much more than I love Lisp… Anyway, it’s still a question of knowing a subject, not being part of a cabal.
Sure is, but how certain are you it’s incorrect? If uncertain, intelligence is useful information you should Bayes Theorem in.
Well, I prefer evidence to signalling: if the problems is only my tediousness, refusing to accept a settled argument, someone can simply point me to a paper, a blog post or a book saying “here, this shows clearly that the replication crysis happened for this reason, not because of the opaqueness of frequentist methods”. I am willing to update. I have done it in the past many times, I’m confident I can do this time too.
Here, all this “He is very intelligent! No, you are very intelligent!” is… sad.
I guess the natural question is—what about standard Frequentist curriculum? Lots of stuff is neither B or F in stats (for example the book my group and I are going through now).
“it’s still a question of knowing a subject”
Indeed. That’s exactly the point.
The most common way I see “fishing” manifest with Bayesian methods is changing the prior until you get the signal you want. In fact, the “clarity” of Bayesian machinery is even aiding and abetting this type of practice.
You say you are willing to update—don’t you find it weird that basically the only place people still talk about B vs F is here on LW? Professional statisticians moved on from this argument decades ago.
The charitable view is LW likes arguing about unsettled philosophy, but aren’t up to speed on what real philosophical arguments are in the field. (In my field, for example, one argument is about testability, and how much should causal models assume). The uncharitable view is LW is addicted to online wankery.
Let me retrace the steps of this conversation, so that we have at least a direction to move towards. The OP argued that we keep a careful eye so that we don’t drift from Bayesianism as the only correct mathematical form of inference. You try to silence him saying that if he is not a statistician, he should not talk about that. I point out that those who routinely use frequentists statistics are commonly fucking it up (the disaster about the RDA of vitamin D is another easily mockable mistakes of frequentist statisticians). The conversation then degenerates on dick-size measuring, only with IQ or academic credentials.
So, let me regroup what I believe to be true, so that specific parts of what I believe to be true can be attacked (but if it’s just: “you don’t have the credentials to talk about that” or “other intelligent people think differently”, please refrain).
1 the only correct foundation for inference and probability is Bayesian 2 Bayesian probability has a broader applicability than frequentist probability 3 basic frequentist statistics can be and should be reformulated from a Bayesian point of view 4 frequentist statistics is taught badly and applied even worse 5 point 4 bears a no small responsability in famous scientific mistakes 6 nor Bayesian or frequentist statiscs bound dishonest scientists 7 advanced statistics has much more in common with functional analysis and measure theory, so that whether it’s expressed in one or the other form is less important 8 LW has the merit of insisting on Bayes because frequentist statiscs, being the academic tradition, has a higher status, and no amount mistakes derived from it seems able to make a dent in its reputation 9 Bayes theorem is the basis of the first formally defined artificial intelligence
I hope this list can keep the discussion productive.
“The conversation then degenerates on dick-size measuring.”
“I hope this list can keep the discussion productive.”
Alright then, Bayes away!
Generic advice for others: the growth mindset for stats (which is a very hard mathematical subject) is to be more like a grad student, e.g. work very very hard and read a lot, and maybe even try to publish. Leave arguing about philosophy to undergrads.
The Wikipedia page for replication crisis doesn’t mention frequentism or Bayesianism. The main reasons are more like the file drawer effect, publish or perish, etc. Of course an honest Bayesian wouldn’t be vulnerable to those, but neither would an honest frequentist.
If I wanted to tell people what I thought they ought to do, I’d have written about decision theory instead. Depending on your decision theory, it might tell you to do something non Bayesian, because you might not have a Bayesian technique right in front of you, but maybe you have a good heuristic that you know from experience works well. All I’m saying is that, probably, your reasoning approximates Bayesian reasoning, even when the “methods” you are using don’t look Bayesian. The way you model those methods as a whole probably does though.
Even if I were writing about decision theory, I don’t really see why making an argument for a particular way of thinking is equivalent to “telling people what to do”, though. Everything that gets written on Less wrong are either arguments or proposals, never commands. Eliezer isnt a statistician either, and yet here we are on his site dedicated to trying to figure out the right way to think. Besides that, I’m pretty sure there are tons of low hanging fruit in my essay that you could easily argue against, without going directly to argument from authority.
I certainly agree with you that Eliezer isn’t a statistician. I may disagree with you on the implications of this.
“All I’m saying is that, probably, your reasoning approximates Bayesian reasoning, even when the “methods” you are using don’t look Bayesian.”
If by “my reasoning” you mean me as a human using my brain, I don’t really see in what sense this is true. I do lots of things with my brain that aren’t Bayesian. If by “my reasoning” you mean stuff I do with data as a statistician, that’s simply false. For example, stuff I do with influence functions has no Bayesian analogue at all.
edit: there is probably some way I could set up some semi-parametric influence function stuff in a Bayesian way—I am not sure.
Funny thing though. If Ilya ever used an argument from authority on me, I’d thank him and start thinking hard about where I went wrong. You’ve read the sequences, right? Remember the praise for Judea Pearl? Well, Ilya is his student and coauthor. If he disagrees with you, it’s strong evidence.
Are you a statistician?
If yes: what’s your favorite paper you wrote on Bayes?
If not: why are you telling experts what to do?
-1
A request for the “argument from authority” fallacy. Freethinkers discuss ideas directly on their merits, not on their author’s job description. A rationalist doesn’t ignore any evidence, of course (even authors’ job descriptions), but try to weight them accurately, okay?
Will do, thank you.
edit:
“Chances are, if you’re a statistics/data science practitioner with a few years of experience applying different techniques to different problems and different data sets, and you have some general intuitions about which techniques apply better to which domains, you’re probably doing this in a Bayesian way.”
Freethinkers who might know a little bit about statistics/data science would, presumably by your lights, stop reading here.
I have seen a lot of this type of overconfident stuff out of LW over the years. That is, in Bayesian terms, I already had a prior, and it already updated in a “read a textbook and stop blogging” direction.
In general, there is no good way for me to know what your prior is on the subject I’m going to write, unless I already knew you really well. But it’s unreasonable to expect me to know what all of my audiences’ priors are and try to write something that agrees with them. I don’t think it’s possible. I’m not optimizing for the argmax of the less wrong commenters anyway. I want to write something that’s probably wrong, so I can update from it if I need to. Your kind of responses, and saying “read a textbook and stop blogging” seems to go against the spirit of free thought and debate anyway. Personally, I think the tendency of people to respond in that way is probably why communities like these dissipate after a while. I wrote a post about this subject as well.
Why write about technical stuff you don’t really know very well? How did you expect that to go?
I find this attitude of “lets blog about stuff that’s probably wrong with the aim of updating later when people correct you” kind of a weird way to go. Why not just go and read directly about what you want to learn about?
Your way helps you, of course, but generates negative externalities for everybody else (who have to either spend time correcting you if they know better, or get misled by you if they don’t).
Wasn’t scholarship (e.g. reading stuff) one of the virtues?
Scholarship was mentioned as one of the virtues, but EY didn’t put it into practice much. People tend to learn by imitation.
At what point do you consider yourself to have read enough? At what point do you decide that you’ve read enough textbooks and now it’s ok to blog?
Also, I don’t see how this creates negative externalities on people. That kind of assumes a bizarre situation where people are either forced to read, or forced to respond to things. Apply that reasoning to basically the entire internet, social media, every day discussions with people, and you basically have to quarantine yourself from most of the world to avoid that risk. Or you conclude that all speech has to be meticulously curated so that there is very low risk of misleading, offending, upsetting, or otherwise wasting someone’s time.
“At what point do you consider yourself to have read enough?”
How about a single statistics class at a university. At that point one might appreciate the set of things one might not know about yet. In reality, though, I feel that if you want to blog about technical topics, you should be an expert on said technical topics. If you are not an expert, it seems you should listen, not talk.
“Also, I don’t see how this creates negative externalities on people.”
Conditional on you being wrong, you should expect no negative externalities only if you expect the activity of blogging to be akin to pissing into the wind—you don’t expect folks to take you seriously anyways. If you are not an expert yet, you should not be very confident about avoiding being wrong on technical topics.
If folks do take you seriously, they either get the wrong idea, or have to spend energy correcting you, or leave it alone, and let you mislead others.
Not sure why you would say this (assuming I haven’t even done that) and then immediately admit that you expect something much higher. What that level of expertise is I’m not sure, but probably having a Ph.D in statistics?
I have an undergraduate degree in math / physics, and I’ve been working at a data science job for 3 years, while spending most of my free time studying these subjects. I wouldn’t call myself an expert, but at least personally, I think I’ve reached a point where I can feasibly have discussions with people about statistics / ML, and not say things that are totally far off from where at least a certain mode of experts are on the subject.
Of course, the topic I was discussing is actually somewhere on the border of statistics, mathematics, and philosophy, and my guess is there are few academic programs that focus specifically on that overlapping region. That makes it very unlikely for anyone on this site to be at the level of expertise you demand. And if the subject is really that esoteric, it also makes it more unlikely that someone would somehow damagingly misuse what they read here. There are no infohazards (as far as I know) in my post, and there really aren’t any concrete suggestions for actions to take, either.
Maybe there is a cultural/generational difference here.
I have seen very little on Bayes out of LW over the years I agree with—take it as a datapoint if you wish. Most of it is somewhere between at least somewhat wrong and not even wrong.
Hanson had a post somewhere on how folks should practice holding strong opinions and arguing for them, but not taking the whole thing very seriously. Maybe that’s what you are doing.
There may indeed be a cultural difference here.
LessWrong has tended towards skepticism (though not outright rejection) of academic credentials ( consider Eliezer’s “argument trumps authority” discussions in the Sequences). However, this site is more or less a place for somewhat informal intellectual discussion. It is not an authoritative information repository, and as far as I can tell, does not claim to be. Anyone who participates in discussions here is probably well aware of this fact, and is fully expected to be able to consider the arguments here, not take them at face value.
If you disagree with some of the core ideas around this community (like Bayesian epistemology), as well as what you perceive to be the “negative externalities” of the tendency towards informal / non-expert discussion, then to me it seems likely that you disagree with certain aspects of the culture here. But you seem to have chosen to oppose those aspects, rather than simply choosing not to participate.
I don’t really have time to “oppose” in the sense you mean, as that’s a full time job. But for the record this aspect of LW culture is busted, I think.
“somewhat informal intellectual discussion”
All I am saying is, if you are going to talk about technical topics, either: (a) know what you are talking about, or (b) if you don’t or aren’t sure, maybe read more and talk less, or at least put disclaimers somewhere. That’s at least a better standard than what [university freshmen everywhere] are doing.
If you think you know what you are talking about, but then someone corrects you on something basic, heavily update towards (b).
I try to adhere to this also, actually—on technical stuff I don’t know super well. Which is a lot of stuff.
The kind of meaningless trash talk MrMind is engaged in above, I find super obnoxious.
But this is a philosophical position you’re taking. You’re not just explaining to us what common decency and protocol should dictate—you’re arguing for a particular conception of discourse norms you believe should be adopted. And probably, in this community, a minority position at that. But, the way that you have stated this comes across like you think your position is obvious, to the point where it’s not really worth arguing for. To me, it doesn’t seem so obvious. Moreover, if it’s not obvious, and if you were to follow your own guidelines fully, you might decide to leave that argument up to the professional, fully credentialed philosophers.
Anyway, what you are bringing up is worth arguing about in my opinion. LW may be credential-agnostic, but it also would be beneficial to have some way of knowing which arguments carry the most weight, and what information is deemed the most reliable—while also allowing people of all levels of expertise to discuss it freely. Such a problem is very difficult, but I think following your principle of “only experts talk, non-experts listen” is sort of extreme and not really appropriate outside of classrooms and lecture halls.
I am saying there is a very easy explanation on why the stats community moved on and LW is still talking about this: LW’s thinking on this is “freshman level.”
I don’t think “know what you are talking about” is controversial, but perhaps I am just old.
I think it’s ok for non-experts to talk, I just think they need to signal stuff appropriately. Wikipedia has a similar problem with non-expert and expert talk being confused, which is why it’s not seen as a reliable source on technical topics.
Being “credential-agnostic” is sort of being a bad Bayesian—you should condition on all available evidence if you aren’t certain of claims (and you shouldn’t be if you aren’t an expert). Argument only screens authority under complete certainty.
Non-experts may not know the boundary of their own knowledge, and may also have trouble knowing where the boundaries of the knowledge of others are as well.
In fact, I think that quite frequently even experts have trouble knowing the extent of their own expertise. You can find countless examples of academics weighing in on matters they aren’t really qualified for. I think this is a particularly acute problem in the philosophy of science. This is a problem I had a lot when I read books by authors of pop-sci / pop-philosophy. They sure seem like experts to the non-initiated. I attribute this mainly to them becoming disconnected from academia and living in a bubble containing mostly just them and their fans, who don’t offer much in the way of substantive disagreement. But this is one of the reasons I value discussion so highly.
When I began writing this post, I did not honestly perceive my level of knowledge to be at the “freshman” level. As I’ve mentioned before, many of the points are re-hashes of stuff from people like Jaynes, and although I might have missed some of his subtle points, is there any good way for me to know that he represents a minority or obsolete position without being deeply familiar with the aspects of that field, as someone with decades of experience would?
The simplest solution is just to read until I have that level of experience with the topics as measured by actual time spent on it, but I feel like that would come at the very high cost of not being able to participate in online discussions, which are valuable. But even then, I probably would still not know where my limits are until I bump into opposing views, which would need to occur through discussion.
Yes, absolutely. See also SMBC’s “send in the bishops, they can move diagonally” (chess masters on the Iraq war).
I don’t know if Jaynes represents a minority position (there are a lot of Bayesian statisticians). It’s more like the field moved on from this argument to more interesting arguments. Basically smart Bayesians and frequentists mostly understood each other’s arguments, and considered them mostly valid.
This is the type of B vs F argument people have these days (I linked this here before):
https://normaldeviate.wordpress.com/2012/08/28/robins-and-wasserman-respond-to-a-nobel-prize-winner/
If you really want the gory details, you can also read the Robins/Ritov paper. But it’s a hard paper.
Full disclosure: Robins was my former boss, and I am probably predisposed to liking his stuff.
Re: “what’s a good way to know”: I would say ask experts. Stat profs love talking about this stuff, you can email your local one, and try to go for coffee or something.
Re: “freshman level,” this was perhaps uncharitable phrasing. I just perceive, perhaps incorrectly, a lot of LW discussions as the type of discussion that takes place in dorms everywhere.
I skimmed this a bit, and it seems like the argument went several rounds but was never actually resolved in the end? See Chris Sim’s last comment here which Robins and Wasserman apparently never responded to. Also, besides this type of highly technical discussion, can you point us to some texts that explains the overall history and current state of the F vs B debate in the professional stats community? I’d like to understand how and why they moved on from the kinds of discussion that LW is still having.
There is a recent book Computer Age Statistical Inference by Efron and Hastie (who are well-respected statisticians). They start by distinguishing three kinds of statistics—frequentist (by which they mean Neyman and Pearson with some reliance on Fisher); Bayesian which everybody here knows well; and Fisherian by which they mean mostly maximum likelihood and derivatives. They say that Fisher, though the was dismissive of the Bayesian approach, didn’t fully embrace the frequentism either and blazed his own path somewhere in the middle.
The book is downloadable as a PDF via the link.
We can ask Chris and Larry (I can if/when I see them).
My take on the way this argument got resolved is that Chris and Larry/Jamie agree on the math—namely that to “solve” the example using B methods we need to have a prior that depends on pi. The possible source of disagreement is interpretational.
Larry and Jamie think that this is Bayesians doing “frequentist pursuit”, that is using B machinery to mimic a fundamentally F behavior. As they say, there is nothing wrong with this, but the B here seems extraneous. Chris probably doesn’t see it that way, he probably thinks this is the natural way to do this problem in a B way.
The weird thing about (what I think) Chris’ position here is that this example violates the “likelihood principle” some Bayesians like. The likelihood principle states that all information lives in the likelihood. Of course here the example is set up in such a way that the assignment probably pi(X) is (a) not a part of the likelihood and (b) is highly informative. The natural way for a Bayesian to deal with this is to stick pi(X) in the prior. This is formally ok, but kind of weird and unnatural.
How weird and unnatural it is is a matter of interpretation, I suppose.
This example is very simple, there are much more complicated versions of this. For example, what if we don’t know pi(X), but have to model it? Does pi(X) still go into the prior? That way lie dragons...
I guess my point is, these types of highly technical discussions are the discussions that professionals have if B vs F comes up. If this is too technical, may I ask why even get into this? Maybe this level of technicality is the natural point of technicality for this argument in this, the year of our Lord 2017? This is kind of my point, if you aren’t a professional, why are you even talking about this?
It’s a good question about a history text on B vs F. Let me ask around.
edit: re: dragons, I guess what I mean is, it seems most things in life can be phrased in F or B ways. But there are a lot of phenomena for which the B phrasing, though it exists, isn’t really very clarifying. These might include identification and model misspecification issues. In such cases the B phrasing just feels like carrying around ideological baggage.
My philosophy is inherently multiparadigm—you use the style of kung fu that yields the most benefit or the most clarity for the problem. Sometimes that’s B and sometimes that’s F and sometimes that’s something else. I guess in your language that would be “instrumental rationality in data analysis.”
I don’t think that having a conversation with someone who’s wrong is necessarily bad for myself. Arguing against someone who’s wrong can help me to clarify my own thoughts on a topic.
CFAR supports the notion that one of the best ways to learn is to teach. Mixing reading textbooks passively with active argument is good for learning a subject well.
That’s fine, but can OP at least preface with [Epistemic status: may not know what I am talking about]?
What did you expect with “Very partisan / opinionated”? I don’t think that’s how the average academic expert would preface his professional position if academics would be in the habit of stating the epistemic status.
I was not asking for a signal “I am not an academic.” I was asking for a signal “don’t take this too seriously, dear reader.”
There is a big difference between having strong opinions and being wrong.
I was hoping that “very partisan” would signal that I recognize there are a sizable chunk of people with very different views on the subject, and that recognition indicates some kind of epistemic humility. I was wrong about that, and in the future I’ll try to indicate that more explicitly.
The problem of how much knowledge is enough has an age old solution: academic credentials.
I think the real answer is about people’s motives.
Reading stuff without talking about it isn’t going to impress anyone, since they won’t even know.
Because “experts” are fucking it up left and right.
Ilya is a student and coauthor of Judea Pearl, whose work on causality and Bayes nets was cited by Eliezer many times. He’s an expert at the stuff that LW is amateuring in.
A: Ilya is a statistician.
B: Ilya is an expert in Bayes probability, and is never wrong.
So:
C: Every statistician is an expert in Bayes probability, and they are never wrong.
Corollary: the replication crysis is a conspiracy of the Bayes Shadow Government.
Psychologists are not statisticians, though. Generally they are relatively naive users of stats methods (as are a lot of other applied folks, e.g. doctors that publish, cognitive scientists, social scientists, epidemiologists, etc.) Ideally, methods folks and applied folks collaborate, but this does not always happen.
You can fish for positive findings with B methods just fine, the issue isn’t F vs B, the issue is bad publication incentives.
There is also a little bit of “there is a huge replication crisis on, long story short, we should read this random dude’s blog (with apologies to the OP).”
Pearl is, apparently, only half Bayesian.
I am wrong a lot—I can point you to some errors in my papers if you want.
The replication crysis is decomposable into many pieces, two of which are surely bad incentives and relative inexperience of the “applied folks”. Another though is, that’s the main point, that frequentist methods are a set of ad-hoc, poorly explained, poorly understood heuristics. No wonder that they are used improperly.
On the other hand, I’ve seen the crysis explained mostly by Bayesian statisticians, so I’m possibly in a bubble. If you can point me to a frequentist explanation I would be glad to pop it.
Apparently though, cousin_it thinks you cannot be criticized or argued against...
“Another though[t] is, that’s the main point, that frequentist methods are a set of ad-hoc, poorly explained, poorly understood heuristics.”
I don’t think so. This is what LW repeatedly gets wrong, and I am kind of tired of talking about it. How are you so confident re: what frequentist methods really are about, if you aren’t a statistician? This is incredibly bizarre to me.
Rather than argue about it constantly, which I am very very tired of doing (see above “negative externalities”), I can point you to Larry Wasserman’s book “All of Statistics.” It’s a nice frequentist book. Start there, perhaps. Larry is very smart, one of the smartest statisticians alive, I think.
My culture thrives on peer review, as much as we grumble about it. Emphasis on “peer,” of course.
You should probably be a bit more charitable to cousin_it, he’s very smart too.
I was under the impression that it was sufficient to read statistics books. Apparently though, you need also to be anointed by another statistician to even talk about the subject.
You seem to imply that no statistician has ever criticized frequentist methods. LW is just parroting what others, more expert men already said.
Isn’t it, as long as you’re making an incorrect statement, irrelevant how intelligent you are? Jaynes was wrong about quantum mechanics. Einstein was wrong about the unified field.
Everybody can be wrong, no matter how respected or intelligent they are.
“I was under the impression that it was sufficient to read statistics books.”
Ok, what have you read?
I am not the “blogging police,” I am just saying, based on past experience, that when people who aren’t statisticians talk about these issues, the result is very low quality. So low that it would have been better to stay silent. Statistics is a very mathematical field. These types of arguments are akin to “should we think about mathematics topologically or algebraically?”
“You seem to imply that no statistician has ever criticized frequentist methods.”
See “Tom Knight and the LISP machine”:
http://catb.org/jargon/html/koans.html
One of these koans is pretty Bayesian, actually, the one about tic-tac-toe.
“Isn’t it, as long as you’re making an incorrect statement, irrelevant how intelligent you are?”
Sure is, but how certain are you it’s incorrect? If uncertain, intelligence is useful information you should Bayes Theorem in.
And anyways, charity is about interpreting reasonably what people say.
The pretty standard Bayesian curriculum: De Finetti, Jaynes-Bretthorst, Sivia.
I love Lisp koans much more than I love Lisp… Anyway, it’s still a question of knowing a subject, not being part of a cabal.
Well, I prefer evidence to signalling: if the problems is only my tediousness, refusing to accept a settled argument, someone can simply point me to a paper, a blog post or a book saying “here, this shows clearly that the replication crysis happened for this reason, not because of the opaqueness of frequentist methods”. I am willing to update. I have done it in the past many times, I’m confident I can do this time too.
Here, all this “He is very intelligent! No, you are very intelligent!” is… sad.
I guess the natural question is—what about standard Frequentist curriculum? Lots of stuff is neither B or F in stats (for example the book my group and I are going through now).
“it’s still a question of knowing a subject”
Indeed. That’s exactly the point.
The most common way I see “fishing” manifest with Bayesian methods is changing the prior until you get the signal you want. In fact, the “clarity” of Bayesian machinery is even aiding and abetting this type of practice.
You say you are willing to update—don’t you find it weird that basically the only place people still talk about B vs F is here on LW? Professional statisticians moved on from this argument decades ago.
The charitable view is LW likes arguing about unsettled philosophy, but aren’t up to speed on what real philosophical arguments are in the field. (In my field, for example, one argument is about testability, and how much should causal models assume). The uncharitable view is LW is addicted to online wankery.
Let me retrace the steps of this conversation, so that we have at least a direction to move towards.
The OP argued that we keep a careful eye so that we don’t drift from Bayesianism as the only correct mathematical form of inference.
You try to silence him saying that if he is not a statistician, he should not talk about that.
I point out that those who routinely use frequentists statistics are commonly fucking it up (the disaster about the RDA of vitamin D is another easily mockable mistakes of frequentist statisticians).
The conversation then degenerates on dick-size measuring, only with IQ or academic credentials.
So, let me regroup what I believe to be true, so that specific parts of what I believe to be true can be attacked (but if it’s just: “you don’t have the credentials to talk about that” or “other intelligent people think differently”, please refrain).
1 the only correct foundation for inference and probability is Bayesian
2 Bayesian probability has a broader applicability than frequentist probability
3 basic frequentist statistics can be and should be reformulated from a Bayesian point of view
4 frequentist statistics is taught badly and applied even worse
5 point 4 bears a no small responsability in famous scientific mistakes
6 nor Bayesian or frequentist statiscs bound dishonest scientists
7 advanced statistics has much more in common with functional analysis and measure theory, so that whether it’s expressed in one or the other form is less important
8 LW has the merit of insisting on Bayes because frequentist statiscs, being the academic tradition, has a higher status, and no amount mistakes derived from it seems able to make a dent in its reputation
9 Bayes theorem is the basis of the first formally defined artificial intelligence
I hope this list can keep the discussion productive.
“The conversation then degenerates on dick-size measuring.”
“I hope this list can keep the discussion productive.”
Alright then, Bayes away!
Generic advice for others: the growth mindset for stats (which is a very hard mathematical subject) is to be more like a grad student, e.g. work very very hard and read a lot, and maybe even try to publish. Leave arguing about philosophy to undergrads.
This sounds a lot like the Neil Tyson / Bill Nye attitude of “science has made philosophy obsolete!”
I don’t agree with Tyson on this, I just think yall aren’t qualified to do philosophy of stats.
The Wikipedia page for replication crisis doesn’t mention frequentism or Bayesianism. The main reasons are more like the file drawer effect, publish or perish, etc. Of course an honest Bayesian wouldn’t be vulnerable to those, but neither would an honest frequentist.
Who else has said that science could and should be wholesale replaced by Bayes?
No one?
Also the point.
If I wanted to tell people what I thought they ought to do, I’d have written about decision theory instead. Depending on your decision theory, it might tell you to do something non Bayesian, because you might not have a Bayesian technique right in front of you, but maybe you have a good heuristic that you know from experience works well. All I’m saying is that, probably, your reasoning approximates Bayesian reasoning, even when the “methods” you are using don’t look Bayesian. The way you model those methods as a whole probably does though.
Even if I were writing about decision theory, I don’t really see why making an argument for a particular way of thinking is equivalent to “telling people what to do”, though. Everything that gets written on Less wrong are either arguments or proposals, never commands. Eliezer isnt a statistician either, and yet here we are on his site dedicated to trying to figure out the right way to think. Besides that, I’m pretty sure there are tons of low hanging fruit in my essay that you could easily argue against, without going directly to argument from authority.
I certainly agree with you that Eliezer isn’t a statistician. I may disagree with you on the implications of this.
“All I’m saying is that, probably, your reasoning approximates Bayesian reasoning, even when the “methods” you are using don’t look Bayesian.”
If by “my reasoning” you mean me as a human using my brain, I don’t really see in what sense this is true. I do lots of things with my brain that aren’t Bayesian. If by “my reasoning” you mean stuff I do with data as a statistician, that’s simply false. For example, stuff I do with influence functions has no Bayesian analogue at all.
edit: there is probably some way I could set up some semi-parametric influence function stuff in a Bayesian way—I am not sure.
Funny thing though. If Ilya ever used an argument from authority on me, I’d thank him and start thinking hard about where I went wrong. You’ve read the sequences, right? Remember the praise for Judea Pearl? Well, Ilya is his student and coauthor. If he disagrees with you, it’s strong evidence.