Mod note: As part of our revamp of moderation norms, one subject coming up is what to do with AI 101 content/questions, and arguments from people who seem unfamiliar with a lot of background material on LessWrong.
A key value-prop of LessWrong is that some arguments get to be “reasonably settled”, rather than endlessly rehashed. You’re welcome (encouraged!) to revisit old arguments if you have new evidence, but not make people repeat basic points we’ve covered many times.
After thinking about it some and discussing with other mods, and how it relates to posts like this, my take is:
On most posts, you’re basically expected to be caught up on major AI Risk arguments on LessWrong. If someone makes a post, which makes an argument assuming (as background) that AI is a significant risk, and people make comments like “why would AI even be a threat?”, moderators will most likely delete that comment and message the commenter telling them they can ask about it in the latest AI Questions Open Thread, or writing a top-level post that engages with the many previous arguments and claims about AI risk.
Some posts are explicitly about either being a 101 space, or calling into question whether previous specific arguments about AI risk are valid. 101 spaces are basically open for arbitrary skeptical comments, posts making specific arguments can have comments engaging with those specific arguments.
This particular post (and other variants of this post like Zvi’s response or the original linkpost) is not a 101 level post, it’s making an argument building off of previous work. Questions and comments about the specific wisdom of “given a high risk of AI catastrophe, we need this particular government response” are fine. Arguments questioning the basics of AI risk should go in the open thread. Arguments about Eliezer’s specific confidence level… I’d say are somewhat on the fence. But then it’s better to frame your comment as “I think risk of global AI catastrophe is [some low number], here’s why. [And then engages with common arguments about why it might be higher, or linking to a previous post where you’ve discussed it].”
By contrast, various linkposts for Eliezer appearing on various podcasts seem more like the topic is specifically discussing x-risk 101 material, and questions/arguments about that seem fine there. (I still encourage users to focus on “why I disagree with a claim” than just generally saying “think Eliezer is wrong about his key assumptions” without saying why)
Meanwhile, if you’re reading this and are like “but, I don’t know why Eliezer believes these things, why isn’t this just science fiction? I’m happy to read up on the background arguments, but, where?”, here are a couple places off the top of my head:
These norms / rules make me slightly worried that disagreement with Eliezer will be conflated with not being up-to-speed on the Sequences, or the basic LessWrong material.
I suppose that the owners and moderators of this website are afforded the right to consider anything said on the website to be, or not to be, at the level of quality or standards they wish to keep and maintain here.
But this is a discussion forum, and the incentives of the owners of this website are to facilitate discussion of some kind. Any discussion will be composed of questions and attempts to answer such questions. Questions can implicitly or explicitly point back to any material, no matter how old it is. This is not necessarily debate, if so. However, even if it is, if the intent of the “well-kept garden” is to produce a larger meta-process that produces useful insights, then the garden should be engineered such that even debate produces useful results.
I think it goes without saying that one can disagree with anything in the Sequences and can also be assumed to have read and understood it. If you engage with someone in conversation under the assumption that their disagreement means that they have not understood something about what they are arguing about, then you are at a disadvantage in regards to a charitability asymmetry. This asymmetry carries the risk that you won’t be able to convince the person you’re talking to that they actually don’t understand what they are talking about.
I have, for most of my (adult) life (and especially in intellectual circles), been under the impression that it is always good to assume that whoever you are talking to understands what they are talking about to the maximum extent possible, even if they don’t. To not do this can be treated negatively in many situations.
I think it goes without saying that one can disagree with anything in the Sequences and can also be assumed to have read and understood it
This seems false as stated—some nontrivial content in the Sequences consists of theorems.
More generally, there are some claims in the original Sequences that are false (so agreeing with the claim may be at least some evidence that you didn’t understand it), some that I’d say “I think that’s true, but reasonable people can definitely disagree”, some where it’s very easy for disagreement to update me toward “you didn’t understand that claim”, etc. Possibly you agree with all that, but I want to state it explicitly; this seems extra important to be clear about if you plan to behave as though it’s not true in object-level conversation.
It depends on whether you think what I stated was closer to “completely false” or “technically false, because of the word ‘anything’.” If I had instead said “I think it goes without saying that one can disagree with nearly anything in the Sequences and can also be assumed to have read and understood it”, that might bring it out of “false” territory for you, but I feel we would still have a disagreement.
There are theorems in the Sequences that I disagree with Eliezer’s characterization of, like Löb’s Theorem, where I feel very confident that I have fully understood both my reading of the theorem as well as Eliezer’s interpretation of it to arrive at my conclusions. Also, that this disagreement is fairly substantial, and also may be a key pillar of Eliezer’s case for very high AI Risk in general.
My worry still stands that disagreement with Eliezer (especially about how high AI Risk actually is) will be conflated with not being up-to-speed on the Sequences, or about misunderstanding key material, or about misunderstanding theorems or things that have allegedly been proven. I think the example I gave is one specific case of something where Eliezer’s interpretation of the theorem (which I believe to have been incorrect) was characterized as the theorem itself.
My position that is regardless of whether or not you think all what I just said is preposterous and proof that I don’t understand key material, the norm(s) of good-faith assumption and charitability are still highly advisable to have. I generally believe that in most disagreements, it is possible for both parties to assume that the other party understands them well enough, just that they have assigned very different probabilities to the same statements.
The comment seems to be saying that they will remove off-topic comments or low-effort posts on things that have been discussed endlessly here, not block posts about AI risk in general. It’s fair to write posts about why you think AI risk is overblown and it’s important for the community to have outside input, but also it’s important to be able to write posts that aren’t about re-arguing the same thing over and over or the community will atrophy and die.
Note that this anti-doom post has a reasonably high karma score for being a link post, presumably because the writer is actually aware of and engages the best arguments against her position.
such posts are generally not banned to my knowledge but, ah, won’t have positive score unless you can describe mechanistically why a lot of hyperskeptical people should be convinced you’re definitely right. Can you demonstrate a bound on the possible behaviors of a system, the way I can demonstrate a bound on the possible behaviors of a safe rust program?
I don’t think it’s quite that; a more central example I think would be something like a post about extrapolating demographic trends to 2070 under the UN’s assumptions, where then justifying whether or not 2070 is a real year is kind of a different field.
Mod note: As part of our revamp of moderation norms, one subject coming up is what to do with AI 101 content/questions, and arguments from people who seem unfamiliar with a lot of background material on LessWrong.
A key value-prop of LessWrong is that some arguments get to be “reasonably settled”, rather than endlessly rehashed. You’re welcome (encouraged!) to revisit old arguments if you have new evidence, but not make people repeat basic points we’ve covered many times.
After thinking about it some and discussing with other mods, and how it relates to posts like this, my take is:
On most posts, you’re basically expected to be caught up on major AI Risk arguments on LessWrong. If someone makes a post, which makes an argument assuming (as background) that AI is a significant risk, and people make comments like “why would AI even be a threat?”, moderators will most likely delete that comment and message the commenter telling them they can ask about it in the latest AI Questions Open Thread, or writing a top-level post that engages with the many previous arguments and claims about AI risk.
Some posts are explicitly about either being a 101 space, or calling into question whether previous specific arguments about AI risk are valid. 101 spaces are basically open for arbitrary skeptical comments, posts making specific arguments can have comments engaging with those specific arguments.
This particular post (and other variants of this post like Zvi’s response or the original linkpost) is not a 101 level post, it’s making an argument building off of previous work. Questions and comments about the specific wisdom of “given a high risk of AI catastrophe, we need this particular government response” are fine. Arguments questioning the basics of AI risk should go in the open thread. Arguments about Eliezer’s specific confidence level… I’d say are somewhat on the fence. But then it’s better to frame your comment as “I think risk of global AI catastrophe is [some low number], here’s why. [And then engages with common arguments about why it might be higher, or linking to a previous post where you’ve discussed it].”
By contrast, various linkposts for Eliezer appearing on various podcasts seem more like the topic is specifically discussing x-risk 101 material, and questions/arguments about that seem fine there. (I still encourage users to focus on “why I disagree with a claim” than just generally saying “think Eliezer is wrong about his key assumptions” without saying why)
Meanwhile, if you’re reading this and are like “but, I don’t know why Eliezer believes these things, why isn’t this just science fiction? I’m happy to read up on the background arguments, but, where?”, here are a couple places off the top of my head:
Superintelligence FAQ (very accessible to layfolk)
The Alignment Problem from a Deep Learning Perspective (written with ML researchers in mind)
(I’ll work on compiling more of these soon)
These norms / rules make me slightly worried that disagreement with Eliezer will be conflated with not being up-to-speed on the Sequences, or the basic LessWrong material.
I suppose that the owners and moderators of this website are afforded the right to consider anything said on the website to be, or not to be, at the level of quality or standards they wish to keep and maintain here.
But this is a discussion forum, and the incentives of the owners of this website are to facilitate discussion of some kind. Any discussion will be composed of questions and attempts to answer such questions. Questions can implicitly or explicitly point back to any material, no matter how old it is. This is not necessarily debate, if so. However, even if it is, if the intent of the “well-kept garden” is to produce a larger meta-process that produces useful insights, then the garden should be engineered such that even debate produces useful results.
I think it goes without saying that one can disagree with anything in the Sequences and can also be assumed to have read and understood it. If you engage with someone in conversation under the assumption that their disagreement means that they have not understood something about what they are arguing about, then you are at a disadvantage in regards to a charitability asymmetry. This asymmetry carries the risk that you won’t be able to convince the person you’re talking to that they actually don’t understand what they are talking about.
I have, for most of my (adult) life (and especially in intellectual circles), been under the impression that it is always good to assume that whoever you are talking to understands what they are talking about to the maximum extent possible, even if they don’t. To not do this can be treated negatively in many situations.
This seems false as stated—some nontrivial content in the Sequences consists of theorems.
More generally, there are some claims in the original Sequences that are false (so agreeing with the claim may be at least some evidence that you didn’t understand it), some that I’d say “I think that’s true, but reasonable people can definitely disagree”, some where it’s very easy for disagreement to update me toward “you didn’t understand that claim”, etc. Possibly you agree with all that, but I want to state it explicitly; this seems extra important to be clear about if you plan to behave as though it’s not true in object-level conversation.
It depends on whether you think what I stated was closer to “completely false” or “technically false, because of the word ‘anything’.” If I had instead said “I think it goes without saying that one can disagree with nearly anything in the Sequences and can also be assumed to have read and understood it”, that might bring it out of “false” territory for you, but I feel we would still have a disagreement.
There are theorems in the Sequences that I disagree with Eliezer’s characterization of, like Löb’s Theorem, where I feel very confident that I have fully understood both my reading of the theorem as well as Eliezer’s interpretation of it to arrive at my conclusions. Also, that this disagreement is fairly substantial, and also may be a key pillar of Eliezer’s case for very high AI Risk in general.
My worry still stands that disagreement with Eliezer (especially about how high AI Risk actually is) will be conflated with not being up-to-speed on the Sequences, or about misunderstanding key material, or about misunderstanding theorems or things that have allegedly been proven. I think the example I gave is one specific case of something where Eliezer’s interpretation of the theorem (which I believe to have been incorrect) was characterized as the theorem itself.
My position that is regardless of whether or not you think all what I just said is preposterous and proof that I don’t understand key material, the norm(s) of good-faith assumption and charitability are still highly advisable to have. I generally believe that in most disagreements, it is possible for both parties to assume that the other party understands them well enough, just that they have assigned very different probabilities to the same statements.
.
The comment seems to be saying that they will remove off-topic comments or low-effort posts on things that have been discussed endlessly here, not block posts about AI risk in general. It’s fair to write posts about why you think AI risk is overblown and it’s important for the community to have outside input, but also it’s important to be able to write posts that aren’t about re-arguing the same thing over and over or the community will atrophy and die.
Note that this anti-doom post has a reasonably high karma score for being a link post, presumably because the writer is actually aware of and engages the best arguments against her position.
.
such posts are generally not banned to my knowledge but, ah, won’t have positive score unless you can describe mechanistically why a lot of hyperskeptical people should be convinced you’re definitely right. Can you demonstrate a bound on the possible behaviors of a system, the way I can demonstrate a bound on the possible behaviors of a safe rust program?
I don’t think it’s quite that; a more central example I think would be something like a post about extrapolating demographic trends to 2070 under the UN’s assumptions, where then justifying whether or not 2070 is a real year is kind of a different field.
Out of curiosity, what do you plan to do when people keep bringing up Penrose?
Thank you for writing these up! I think they are good guidelines for making discussion more productive.
Are these / are you planning to put these in a top level post as well?