I think rigor and clarity are more similar than you indicate. I mostly think of rigor as either (i) formal definitions and proofs, or (ii) experiments well described, executed, and interpreted. I think it’s genuinely hard to reach a high level of clarity about many things without (i) or (ii). For instance, people argue about “optimization”, but without referencing (hypothetical) detailed experiments or formal notions, those arguments just won’t be very clear; experimental or definitional details just matter a lot, and this is very often the case in AI. LW has historically endorsed a bunch of arguments that are basically just wrong because they have a crucial reliance on unstated assumptions (e.g. AIs will be “rational agents”), and ML looks at this and concludes people on LW are at the peak of “mount stupid”.
I think I would support Joe’s view here that clarity and rigour are significantly different… but maybe—David—your comments are supposed to be specific to alignment work? e.g. I can think of plenty of times I have read books or articles in other areas and fields that contain zero formal definitions, proofs, or experiments but are obviously “clear”, well-explained, well-argued etc. So by your definitions is that not a useful and widespread form of rigour-less clarity? (One that we would want to ‘allow’ in alignment work?) Or would you instead maintain that such writing can’t ever really be clear without proofs or experiments?
I tend to think one issue is more that it’s really hard to do well (clear, useful, conceptual writing that is) and that many of the people trying to do it in alignment come from are inexperienced in doing it (and often have backgrounds in fields where things like proofs or experiments are the norm).
(To be clear, I think a lot of these arguments are pointing at important intuitions, and can be “rescued” via appropriate formalizations and rigorous technical work).
Mostly I agree with this. I have more thoughts, but probably better to put them in a top-level post—largely because I think this is important and would be interested to get more input on a good balance.
A few thoughts on LW endorsing invalid arguments: I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]. E.g. it doesn’t necessarily seem much of a problem for the former to have reliance on unstated assumptions. I wouldn’t formally specify an idea before sketching it, and it’s not clear to me that there’s anything wrong with collective sketching (so long as we know we’re sketching—and this part could certainly be improved). I’d first want to optimize the epistemic process, and then worry about the looking foolish part. (granted that there are instrumental reasons not to look foolish)
On ML’s view, are you mainly thinking of people who may do research on an important x-safety sub-problem without necessarily buying x-risk arguments? It seems unlikely to me that anyone gets persuaded of x-risk from the bottom up, whether or not the paper/post in question is rigorous—but perhaps this isn’t required for a lot of useful research?
I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]
Yeah I put those in one sentence in my comment but I agree that they are two separate points.
RE impact on ML community: I wasn’t thinking about anything in particular I just think the ML community should have more respect for LW/x-safety, and stuff like that doesn’t help.
I think rigor and clarity are more similar than you indicate. I mostly think of rigor as either (i) formal definitions and proofs, or (ii) experiments well described, executed, and interpreted. I think it’s genuinely hard to reach a high level of clarity about many things without (i) or (ii). For instance, people argue about “optimization”, but without referencing (hypothetical) detailed experiments or formal notions, those arguments just won’t be very clear; experimental or definitional details just matter a lot, and this is very often the case in AI. LW has historically endorsed a bunch of arguments that are basically just wrong because they have a crucial reliance on unstated assumptions (e.g. AIs will be “rational agents”), and ML looks at this and concludes people on LW are at the peak of “mount stupid”.
Minor, but Dunning-Kruger neither claims to detect a Mount Stupid effect nor (probably) is the study powered enough to detect it.
Very good to know! I guess in the context of my comment it doesn’t matter as much because I only talk about others’ perception.
I think I would support Joe’s view here that clarity and rigour are significantly different… but maybe—David—your comments are supposed to be specific to alignment work? e.g. I can think of plenty of times I have read books or articles in other areas and fields that contain zero formal definitions, proofs, or experiments but are obviously “clear”, well-explained, well-argued etc. So by your definitions is that not a useful and widespread form of rigour-less clarity? (One that we would want to ‘allow’ in alignment work?) Or would you instead maintain that such writing can’t ever really be clear without proofs or experiments?
I tend to think one issue is more that it’s really hard to do well (clear, useful, conceptual writing that is) and that many of the people trying to do it in alignment come from are inexperienced in doing it (and often have backgrounds in fields where things like proofs or experiments are the norm).
(To be clear, I think a lot of these arguments are pointing at important intuitions, and can be “rescued” via appropriate formalizations and rigorous technical work).
Mostly I agree with this.
I have more thoughts, but probably better to put them in a top-level post—largely because I think this is important and would be interested to get more input on a good balance.
A few thoughts on LW endorsing invalid arguments:
I’d want to separate considerations of impact on [LW as collective epistemic process] from [LW as outreach to ML researchers]. E.g. it doesn’t necessarily seem much of a problem for the former to have reliance on unstated assumptions. I wouldn’t formally specify an idea before sketching it, and it’s not clear to me that there’s anything wrong with collective sketching (so long as we know we’re sketching—and this part could certainly be improved).
I’d first want to optimize the epistemic process, and then worry about the looking foolish part. (granted that there are instrumental reasons not to look foolish)
On ML’s view, are you mainly thinking of people who may do research on an important x-safety sub-problem without necessarily buying x-risk arguments? It seems unlikely to me that anyone gets persuaded of x-risk from the bottom up, whether or not the paper/post in question is rigorous—but perhaps this isn’t required for a lot of useful research?
Yeah I put those in one sentence in my comment but I agree that they are two separate points.
RE impact on ML community: I wasn’t thinking about anything in particular I just think the ML community should have more respect for LW/x-safety, and stuff like that doesn’t help.