Work that is still outside the academic Overton window can be brought into academia if it can be approached with the technical rigor of academia, and work that meets academic standards is much more valuable than work that doesn’t; this is both because it can be picked up by the ML community, and because it’s much harder to tell if you are making meaningful progress if your work doesn’t meet these standards of rigor.
Strong agreement with this! I’m frequently told by people that you “cannot publish” on a certain area, but in my experience this is rarely true. Rather, you have to put more work into communicating your idea, and justifying the claims you make—both a valuable exercise! Of course you’ll have a harder time publishing than on something that people immediately understand—but people do respect novel and interesting work, so done well I think it’s much better for your career than one might naively expect.
I especially wish there was more emphasis on rigor on the Alignment Forum and elsewhere: it can be valuable to do early-stage work that’s more sloppy (rigor is slow and expensive), but when there’s long-standing disagreements it’s usually better to start formalizing things or performing empirical work than continuing to opine.
That said, I do think academia has some systemic blindspots. For one, I think CS is too dismissive of speculative and conceptual research—much of this work will end up being mistaken admittedly, but it’s an invaluable source of ideas. I also think there’s too much emphasis on an “algorithmic contribution” in ML, which leads to undervaluing careful empirical valuations and understanding failure modes of existing systems.
Presumably “too dismissive of speculative and conceptual research” is a direct consequence of increased emphasis on rigor. Rigor is to be preferred all else being equal, but all else is not equal.
It’s not clear to me how we can encourage rigor where effective without discouraging research on areas where rigor isn’t currently practical. If anyone has ideas on this, I’d be very interested.
I note that within rigorous fields, the downsides of rigor are not obvious: we can point to all the progress made; progress that wasn’t made due to the neglect of conceptual/speculative research is invisible. (has the impact of various research/publication norms ever been studied?)
Further, it seems limiting only to consider [must always be rigorous (in publications)] vs [no demand for rigor]. How about [50% of your publications must be rigorous] (and no incentive to maximise %-of-rigorous-publications), or any other not-all-or-nothing approach?
I’d contrast rigor with clarity here. Clarity is almost always a plus. I’d guess that the issue in social science fields isn’t a lack of rigor, but rather of clarity. Sometimes clarity without rigor may be unlikely, e.g. where there’s a lot of confusion or lack of good faith—in such cases an expectation of rigor may help. I don’t think this situation is universal.
What we’d want on LW/AF is a standard of clarity. Rigor is an often useful proxy. We should be careful when incentivizing proxies.
I think rigor and clarity are more similar than you indicate. I mostly think of rigor as either (i) formal definitions and proofs, or (ii) experiments well described, executed, and interpreted. I think it’s genuinely hard to reach a high level of clarity about many things without (i) or (ii). For instance, people argue about “optimization”, but without referencing (hypothetical) detailed experiments or formal notions, those arguments just won’t be very clear; experimental or definitional details just matter a lot, and this is very often the case in AI. LW has historically endorsed a bunch of arguments that are basically just wrong because they have a crucial reliance on unstated assumptions (e.g. AIs will be “rational agents”), and ML looks at this and concludes people on LW are at the peak of “mount stupid”.
I think I would support Joe’s view here that clarity and rigour are significantly different… but maybe—David—your comments are supposed to be specific to alignment work? e.g. I can think of plenty of times I have read books or articles in other areas and fields that contain zero formal definitions, proofs, or experiments but are obviously “clear”, well-explained, well-argued etc. So by your definitions is that not a useful and widespread form of rigour-less clarity? (One that we would want to ‘allow’ in alignment work?) Or would you instead maintain that such writing can’t ever really be clear without proofs or experiments?
I tend to think one issue is more that it’s really hard to do well (clear, useful, conceptual writing that is) and that many of the people trying to do it in alignment come from are inexperienced in doing it (and often have backgrounds in fields where things like proofs or experiments are the norm).
(To be clear, I think a lot of these arguments are pointing at important intuitions, and can be “rescued” via appropriate formalizations and rigorous technical work).
Mostly I agree with this. I have more thoughts, but probably better to put them in a top-level post—largely because I think this is important and would be interested to get more input on a good balance.
A few thoughts on LW endorsing invalid arguments: I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]. E.g. it doesn’t necessarily seem much of a problem for the former to have reliance on unstated assumptions. I wouldn’t formally specify an idea before sketching it, and it’s not clear to me that there’s anything wrong with collective sketching (so long as we know we’re sketching—and this part could certainly be improved). I’d first want to optimize the epistemic process, and then worry about the looking foolish part. (granted that there are instrumental reasons not to look foolish)
On ML’s view, are you mainly thinking of people who may do research on an important x-safety sub-problem without necessarily buying x-risk arguments? It seems unlikely to me that anyone gets persuaded of x-risk from the bottom up, whether or not the paper/post in question is rigorous—but perhaps this isn’t required for a lot of useful research?
I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]
Yeah I put those in one sentence in my comment but I agree that they are two separate points.
RE impact on ML community: I wasn’t thinking about anything in particular I just think the ML community should have more respect for LW/x-safety, and stuff like that doesn’t help.
It’s not clear to me how we can encourage rigor where effective without discouraging research on areas where rigor isn’t currently practical. If anyone has ideas on this, I’d be very interested.
A rough heuristic I have is that if the idea you’re introducing is highly novel, it’s OK to not be rigorous. Your contribution is bringing this new, potentially very promising, idea to people’s attention. You’re seeking feedback on how promising it really is and where people are confused , which will be helpful for then later formalizing it and studying it more rigorously.
But if you’re engaging with a large existing literature and everyone seems to be confused and talking past each other (which I’d characterize a significant fraction of the mesa-optimization literature, for example) -- then the time has come to make things more rigorous, and you are unlikely to make much further progress without it.
I think part of this has to do with growing pains in the LW/AF community… When it was smaller it was more like an ongoing discussion with a few people and signal-to-noise wasn’t as important, etc.
Strong agreement with this! I’m frequently told by people that you “cannot publish” on a certain area, but in my experience this is rarely true. Rather, you have to put more work into communicating your idea, and justifying the claims you make—both a valuable exercise! Of course you’ll have a harder time publishing than on something that people immediately understand—but people do respect novel and interesting work, so done well I think it’s much better for your career than one might naively expect.
I especially wish there was more emphasis on rigor on the Alignment Forum and elsewhere: it can be valuable to do early-stage work that’s more sloppy (rigor is slow and expensive), but when there’s long-standing disagreements it’s usually better to start formalizing things or performing empirical work than continuing to opine.
That said, I do think academia has some systemic blindspots. For one, I think CS is too dismissive of speculative and conceptual research—much of this work will end up being mistaken admittedly, but it’s an invaluable source of ideas. I also think there’s too much emphasis on an “algorithmic contribution” in ML, which leads to undervaluing careful empirical valuations and understanding failure modes of existing systems.
Presumably “too dismissive of speculative and conceptual research” is a direct consequence of increased emphasis on rigor. Rigor is to be preferred all else being equal, but all else is not equal.
It’s not clear to me how we can encourage rigor where effective without discouraging research on areas where rigor isn’t currently practical. If anyone has ideas on this, I’d be very interested.
I note that within rigorous fields, the downsides of rigor are not obvious: we can point to all the progress made; progress that wasn’t made due to the neglect of conceptual/speculative research is invisible. (has the impact of various research/publication norms ever been studied?)
Further, it seems limiting only to consider [must always be rigorous (in publications)] vs [no demand for rigor]. How about [50% of your publications must be rigorous] (and no incentive to maximise %-of-rigorous-publications), or any other not-all-or-nothing approach?
I’d contrast rigor with clarity here. Clarity is almost always a plus.
I’d guess that the issue in social science fields isn’t a lack of rigor, but rather of clarity. Sometimes clarity without rigor may be unlikely, e.g. where there’s a lot of confusion or lack of good faith—in such cases an expectation of rigor may help. I don’t think this situation is universal.
What we’d want on LW/AF is a standard of clarity.
Rigor is an often useful proxy. We should be careful when incentivizing proxies.
I think rigor and clarity are more similar than you indicate. I mostly think of rigor as either (i) formal definitions and proofs, or (ii) experiments well described, executed, and interpreted. I think it’s genuinely hard to reach a high level of clarity about many things without (i) or (ii). For instance, people argue about “optimization”, but without referencing (hypothetical) detailed experiments or formal notions, those arguments just won’t be very clear; experimental or definitional details just matter a lot, and this is very often the case in AI. LW has historically endorsed a bunch of arguments that are basically just wrong because they have a crucial reliance on unstated assumptions (e.g. AIs will be “rational agents”), and ML looks at this and concludes people on LW are at the peak of “mount stupid”.
Minor, but Dunning-Kruger neither claims to detect a Mount Stupid effect nor (probably) is the study powered enough to detect it.
Very good to know! I guess in the context of my comment it doesn’t matter as much because I only talk about others’ perception.
I think I would support Joe’s view here that clarity and rigour are significantly different… but maybe—David—your comments are supposed to be specific to alignment work? e.g. I can think of plenty of times I have read books or articles in other areas and fields that contain zero formal definitions, proofs, or experiments but are obviously “clear”, well-explained, well-argued etc. So by your definitions is that not a useful and widespread form of rigour-less clarity? (One that we would want to ‘allow’ in alignment work?) Or would you instead maintain that such writing can’t ever really be clear without proofs or experiments?
I tend to think one issue is more that it’s really hard to do well (clear, useful, conceptual writing that is) and that many of the people trying to do it in alignment come from are inexperienced in doing it (and often have backgrounds in fields where things like proofs or experiments are the norm).
(To be clear, I think a lot of these arguments are pointing at important intuitions, and can be “rescued” via appropriate formalizations and rigorous technical work).
Mostly I agree with this.
I have more thoughts, but probably better to put them in a top-level post—largely because I think this is important and would be interested to get more input on a good balance.
A few thoughts on LW endorsing invalid arguments:
I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]. E.g. it doesn’t necessarily seem much of a problem for the former to have reliance on unstated assumptions. I wouldn’t formally specify an idea before sketching it, and it’s not clear to me that there’s anything wrong with collective sketching (so long as we know we’re sketching—and this part could certainly be improved).
I’d first want to optimize the epistemic process, and then worry about the looking foolish part. (granted that there are instrumental reasons not to look foolish)
On ML’s view, are you mainly thinking of people who may do research on an important x-safety sub-problem without necessarily buying x-risk arguments? It seems unlikely to me that anyone gets persuaded of x-risk from the bottom up, whether or not the paper/post in question is rigorous—but perhaps this isn’t required for a lot of useful research?
Yeah I put those in one sentence in my comment but I agree that they are two separate points.
RE impact on ML community: I wasn’t thinking about anything in particular I just think the ML community should have more respect for LW/x-safety, and stuff like that doesn’t help.
A rough heuristic I have is that if the idea you’re introducing is highly novel, it’s OK to not be rigorous. Your contribution is bringing this new, potentially very promising, idea to people’s attention. You’re seeking feedback on how promising it really is and where people are confused , which will be helpful for then later formalizing it and studying it more rigorously.
But if you’re engaging with a large existing literature and everyone seems to be confused and talking past each other (which I’d characterize a significant fraction of the mesa-optimization literature, for example) -- then the time has come to make things more rigorous, and you are unlikely to make much further progress without it.
I think part of this has to do with growing pains in the LW/AF community… When it was smaller it was more like an ongoing discussion with a few people and signal-to-noise wasn’t as important, etc.
Agree RE systemic blindspots, although the “algorithmic contribution” thing is sort of a known issue that a lot of senior people disagree with, IME.