‘Before doing any project or entering any field, you need to catch up on existing intellectual discussion on the subject.’
I think this is way too strong.
Still probably directionally correct, though, especially for the typical EA / rationalist, especially in AI safety research (most often relatively young and junior in terms of research experience / taste).
On the tradeoff between (A) “try to understand the work / ideas of previous thinkers” and (B) “just sit down and try to figure out the right answer”, I think for A) might have already been made significantly easier by chatbots like Claude 3.5, while B) probably hasn’t changed anywhere near as much. I expect the differential to probably increase in the near-term, with better LLMs.
This is insanely wrong; it’s exactly opposite of the truth. If you want to do something cool in the world, you should learn more stuff from what other humans have done. If, on the other hand, you want to solve the insanely hard engineering/philosophy problem of AGI alignment in time for humanity to not be wiped out, you absolutely should prioritize solving the problem from scratch.
Can you expand on this concisely inline? I agree with the comment you’re replying to strongly and think it has been one of miri’s biggest weaknesses in the past decade that they didn’t build the fortitude to be able to read existing work without becoming confused by its irrelevance. But I also think your and Abram’s research direction intuitions seem like some of the most important in the field right now, alongside wentworth. I’d like to understand what it is that has held you back from speed reading external work for hunch seeding for so long. To me, it seems like solving from scratch is best done not from scratch, if that makes sense. Don’t defer to what you read.
I’d like to understand what it is that has held you back from speed reading external work for hunch seeding for so long.
Well currently I’m not really doing alignment research. My plans / goals / orientation / thinking style have changed over the years, so I’ve read stuff or tried to read stuff more or less during different periods. When I’m doing my best thinking, yes, I read things for idea seeding / as provocations, but it’s only that—I most certainly am not speed reading, the opposite really: read one paragraph, think for an hour and then maybe write stuff. And I’m obviously not reading some random ML paper, jesus christ. Philosophy, metamathematics, theoretical biology, linguistics, psychology, ethology, … much more interesting and useful.
To me, it seems like solving from scratch is best done not from scratch, if that makes sense.
Absolutely, I 100% agree, IIUC. I also think:
A great majority of the time, when people talk about reading stuff (to “get up to speed”, to “see what other people have done on the subject”, to “get inspiration”, to “become more informed”, to “see what approaches/questions there are”...), they are not doing this “from scratch not from scratch” thing.
“the typical EA / rationalist, especially in AI safety research (most often relatively young and junior in terms of research experience / taste)” is absolutely and pretty extremely erring on the side of failing to ever even try to solve the actual problem at all.
Don’t defer to what you read.
Yeah, I generally agree (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html), though you probably should defer about some stuff at least provisionally (for example, you should probably try out, for a while, the stance of deferring to well-respected philosophers about what questions are interesting).
I think it’s just not appreciated how much people defer to what they read. Specifically, there’s a lot of frame deference. This is usually fine and good in lots of contexts (you don’t need to, like, question epistemology super hard to become a good engineer, or question whether we should actually be basing our buildings off of liquid material rather than solid material or something). It’s catastrophic in AGI alignment, because our frames are bad.
I’d like to offer some serious pushback on the practice of using words like “insane” to describe positions that are not obviously false and which a great number of generally reasonable and well-informed members of the community agree with. It is particularly inappropriate to do that when you have offered no concrete, object-level arguments or explanations [1]for why AI safety researchers should prioritize “solving the problem from scratch.”
Adding in phrases like “it’s exactly opposite of the truth” and “absolutely” not only fails to help your case, but in my view actively makes things worse by using substance-free rhetoric that misleads readers into thinking the case you are bringing forward is stronger than it actually is or that this matter is so obvious and trivial that they shouldn’t even need to think very hard about it before taking your side.
By which I mean, you have included no such arguments in this particular comment, nor have you linked to any other place containing arguments that you agree with on this topic, nor have you offered any explanations in any other comments on this post (I checked, and you have made no other comments on it yet), nor does a cursory look at your profile seem to indicate any posts or recent comments where such ideas might appear.
I disagree re/ the word “insane”. The position to which I stated a counterposition is insane.
“it’s exactly opposite of the truth” and “absolutely” not only fails to help your case, but in my view actively makes things worse by using substance-free rhetoric that misleads readers into thinking the case you are bringing forward is stronger than it actually is or that this matter is so obvious and trivial that they shouldn’t even need to think very hard about it before taking your side.
I disagree, I think I should state my actual position. The phrases you quoted have meaning and conveys my position more than if they were removed.
I disagree, I think I should state my actual position. The phrases you quoted have meaning and conveys my position more than if they were removed.
It does not matter one bit if this is your “actual position”. The point of community norms about discourse is that they constrain what is or isn’t appropriate to say in a given situation; they function on the meta-level by setting up proper incentives for users to take into account when crafting their contributions here, independently of their personal assessments about who is right on the object-level. So your response is entirely off-topic, and the fact you expected it not to be is revealing of a more fundamental confusion in your thinking about this matter.
Moderation (when done properly) does not act solely to resolve individual disputes on the basis of purely local characteristics to try to ensure specific outcomes. Remember, the law is not an optimizer, but rather a system informed by principles of mechanism design that generates specific, legible, and predictable set of real rules about what is or isn’t acceptable, a system that does not bend in response to the clever arguments of an individual who thinks that he alone is special and exempt from them.
Standards are not really popular. Most people don’t like them. Or rather, most people like them in the abstract, but chafe when they get in the way, and it’s pretty rare for someone to not think that their personal exception to the standard is more justified than others’ violations. Half the people here, I think, don’t even see the problem that I’m trying to point at. Or they see it, but they don’t see it as a problem.
I think it would have been weird and useless for you to straight-up lie in your previous comment, so of course you thought what you were saying communicated your real position. Why else would you have written it? But “communicate whatever you truly feel about something, regardless of what form your writing takes” is a truly terribleway of organizing any community in which meaningful intellectual progress is intended. By contrast, giving explanations and reasoning in situations where you label the beliefs of others as “insane” prevents conversations from becoming needlessly heated and spiraling into Demon Threads, while also building towards a community that maintains high-quality contributions.
All of this stuff has already been covered in the large number of expositions people have given over the last few years on what LessWrong is about and what principles animate norms and user behavior (1, 2, 3, 4, 5, etc).
I doubt that we’re going to get anything useful here, but as an indication of where I’m coming from:
I would basically agree with what you’re saying if my first comment had been ad hominem, like “Bogdan is a doo-doo head”. That’s unhelpful, irrelevant, mean, inflammatory, and corrosive to the culture. (Also it’s false lol.)
I think a position can be wrong, can be insanely wrong (which means something like “is very far from the truth, is wrong in a way that produces very wrong actions, and is being produced by a process which is failing to update in a way that it should and is failing to notice that fact”), and can be exactly opposite of the truth (for example, “Redwoods are short, grass is tall” is, perhaps depending on contexts, just about the exact opposite of the truth). And these facts are often knowable and relevant if true. And therefore should be said—in a truth-seeking context. And this is the situation we’re in.
If you had responded to my original comment with something like
“Your choice of words makes it seem like you’re angry or something, and this is coming out in a way that seems like a strong bid for something, e.g. attention or agreement or something. It’s a bit hard to orient to that because it’s not clear what if anything you’re angry about, and so readers are forced to either rudely ignore / dismiss, or engage with someone who seems a bit angry or standoffish without knowing why. Can you more directly say what’s going on, e.g. what you’re angry about and what you might request, so we can evaluate that more explicitly?”
or whatever is the analogous thing that’s true for you, then we could have talked about that. Instead you called my relatively accurate and intentional presentation of my views as “misleading readers into thinking the case you are bringing forward is stronger than it actually is or that this matter is so obvious and trivial...” which sounds to me like you have a problem in your own thinking and norms of discourse, which is that you’re requiring that statements other people make be from the perspective of [the theory that’s shared between the expected community of speakers and listeners] in order for you to think they’re appropriate or non-misleading.
The fact that I have to explain this to you is probably bad, and is probably mostly your responsibility, and you should reevaluate your behavior. (I’m not trying to be gentle here, and if gentleness would help then you deserve it—but you probably won’t get it here from me.)
I think a position can be wrong, can be insanely wrong (which means something like “is very far from the truth, is wrong in a way that produces very wrong actions, and is being produced by a process which is failing to update in a way that it should and is failing to notice that fact”), and can be exactly opposite of the truth (for example, “Redwoods are short, grass is tall” is, perhaps depending on contexts, just about the exact opposite of the truth). And these facts are often knowable and relevant if true. And therefore should be said—in a truth-seeking context.
I agree to some extent, which is why I said the following to gears:
It is fine[2] to label opinions you disagree with as “insane”.
It is fine to give your conclusions without explaining the reasons behind your positions.[3]
It is not fine to do 1 and 2 at the same time.
The fact that you chose the word “insane” to describe something that did not seem obviously false, had a fair bit of support in this community, and that you had not given any arguments against at the time was the problem.
The fact that you think something is “insane” is informationally useful to other people, and, all else equal, should be communicated. But all else is not equal, because (as I explained in my previous comments), it is a fabricated option to think that relaxing norms around the way in which particular kinds of information is communicating will not negatively affect the quality of the conversation that unfolds afterwards.
So you could (at least in my view, not sure what the mods think) say something is “insane” if you explain why, because this allows for opportunities to drag the conversation away from mud-slinging Demon Threads and towards the object-level arguments being discussed (and, in this case, saying you think your interlocutor’s position is crazy could actually be helpful at times, since it signals a great level of disagreement and allows for the quicker identification of how many inferential distances between you and the other commenters). Likewise, you could give your conclusions without presenting arguments or explanations for it, as long as your position is not stated in an overly inflammatory manner, because this then incentivizes useful and clear-headed discourse later on when users can ask what the arguments actually are. But if you go the third route, then you maximize the likelihood of the conversation getting derailed.
“Your choice of words makes it seem like you’re angry or something, and this is coming out in a way that seems like a strong bid for something, e.g. attention or agreement or something. It’s a bit hard to orient to that because it’s not clear what if anything you’re angry about, and so readers are forced to either rudely ignore / dismiss, or engage with someone who seems a bit angry or standoffish without knowing why. Can you more directly say what’s going on, e.g. what you’re angry about and what you might request, so we can evaluate that more explicitly?”
This framing focuses on the wrong part, I think. You can be as angry as you want to when you are commenting on LessWrong, and it seems to be inappropriate to enforce norms about the emotions one is supposed to feel when contributing here. The part that matters is whether specific norms of discourse are getting violated (about the literal things someone is writing, not how they feel in that moment), in which case (as I have argued above) I believe the internal state of mind of the person violating them is primarily irrelevant.
you have a problem in your own thinking and norms of discourse
I’m also not sure what you mean by this. You also implied later on that “requiring that statements other people make be from the perspective of [the theory that’s shared between the expected community of speakers and listeners] in order for you to think they’re appropriate” is wrong, which… doesn’t make sense to me, because that’s the very definition of the word appropriate: “meeting the requirements [i.e. norms] of a purpose or situation.”
The same statement can be appropriate or inappropriate, depending on the rules and norms of the community it is made in.
to think that relaxing norms around the way in which particular kinds of information is communicating will not negatively affect the quality of the conversation that unfolds afterwards.
If this happens because someone says something true, relevant, and useful, in a way that doesn’t have alternative expressions that are really easy and obvious to do (such as deleting the statement “So and so is a doo-doo head”), then it’s the fault of the conversation, not the statement.
The alternative expression, in this particular case (not in the mine run of cases), is not to change the word “insane” (because it seems you are certain enough in your belief that it is applicable here that it makes sense for you to communicate this idea some way), but rather to simply write more (or link to a place that contain arguments which relate, with particularity, to the situation at hand) by explaining why you think it’s is true that the statement is “insane”.
If you are so confident in your conclusion that you are willing to label the articulation of the opposing view as “insane”, then it should be straightforward (and more importantly, should not take so much time that it becomes daunting) to give reasons for that, at the time you make that labeling.
The comment I was responding to also didn’t offer serious relevant arguments.
And it didn’t label the position it was arguing against as “insane”, so this is also entirely off-topic.
It would be ideal for users to always describe why they have reached the conclusions they have, but that is a fabricated option which does not take into account the basic observation that requiring such explanations creates such a tremendous dis-incentive to commenting that it would drastically reduce the quantity of useful contributions in the community, thus making things worse off than they were before.
So the compromise we reach is one in which users can state their conclusions in a relatively neutral manner that does not poison the discourse that comes afterwards, and then if another user has a question or a disagreement about this matter, later on they can then have a regular, non-Demon Thread discussion about it in which they explain their models and the evidence they had to reach their positions.
I think you are also expressing high emotive confidence in your comments. You are presenting a case, and your expressed confidence slightly lower, but still elevated.
I agree[1], and I think it is entirely appropriate to do so, given that I have given some explanations of the mental models behind my positions on these matters.
For clarity, I’ll summarize my conclusion here, on the basis of what I have explained before (1, 2, 3):
It is fine[2] to label opinions you disagree with as “insane”.
It is fine to give your conclusions without explaining the reasons behind your positions.[3]
It is not fine to do 1 and 2 at the same time.
With regards to your “taboo off-topic” reaction, what I mean by “off-topic” in this case is “irrelevant to the discussion at hand, by focusing on the wrong level of abstraction (meta-level norms vs object-level discourse) and by attempting to say the other person behaved similarly, which is incorrect as a factual matter (see the distinction between points 2 and 3 above), but more importantly, immaterial to the topic at hand even if true”.
The comment I was responding to also didn’t offer serious relevant arguments.
I’m time-bottlenecked now, but I’ll give one example. Consider the Natural Abstraction Hypothesis (NAH) agenda (which, fwiw, I think is an example of considerably-better-than-average work on trying to solve the problem from scratch). I’d argue that even for someone interested in this agenda: 1. most of the relevant work has come (and will keep coming) from outside the LW community (see e.g. The Platonic Representation Hypothesis and compare the literature reviewed there with NAH-related work on LW); 2. (given the previous point) the typical AI safety researcher interested in NAH would do better to spend most of their time (at least at the very beginning) looking at potentially relevant literature outside LW, rather than either trying to start from scratch, or mostly looking at LW literature.
considerably-better-than-average work on trying to solve the problem from scratch
It’s considerably better than average but is a drop in the bucket and is probably mostly wasted motion. And it’s a pretty noncentral example of trying to solve the problem from scratch. I think most people reading this comment just don’t even know what that would look like.
even for someone interested in this agenda
At a glance, this comment seems like it might be part of a pretty strong case that [the concrete ML-related implications of NAH] are much better investigated by the ML community compared to LW alignment people. I doubt that the philosophically more interesting aspects of Wentworth’s perspectives relating to NAH are better served by looking at ML stuff, compared to trying from scratch or looking at Wentworth’s and related LW-ish writing. (I’m unsure about the mathematically interesting aspects; the alternative wouldn’t be in the ML community but would be in the mathematical community.)
And most importantly “someone interested in this agenda” is already a somewhat nonsensical or question-begging conditional. You brought up “AI safety research” specifically, and by that term you are morally obliged to mean [the field of study aimed at figuring out how to make cognitive systems that are more capable than humanity and also serve human value]. That pursuit is better served by trying from scratch. (Yes, I still haven’t presented an affirmative case. That’s because we haven’t even communicated about the proposition yet.)
Links have high attrition rate, cf ratio of people overcoming a trivial inconvenience. Post your arguments compressed inline to get more eyeballs on them.
I think this is particularly incorrect for alignment, relative to a more typical STEM research field. Alignment is very young[1]. There’s a lot less existing work worth reading than in a field like, say, lattice quantum field theory. Due to this, the time investment required to start contributing at the research frontier is very low, relatively speaking.
This is definitely changing. There’s a lot more useful work than there was when I started dipping my toe into alignment three years ago. But compared to something like particle physics, it’s still very little.
Still probably directionally correct, though, especially for the typical EA / rationalist, especially in AI safety research (most often relatively young and junior in terms of research experience / taste).
On the tradeoff between (A) “try to understand the work / ideas of previous thinkers” and
(B) “just sit down and try to figure out the right answer”, I think for A) might have already been made significantly easier by chatbots like Claude 3.5, while B) probably hasn’t changed anywhere near as much. I expect the differential to probably increase in the near-term, with better LLMs.
This is insanely wrong; it’s exactly opposite of the truth. If you want to do something cool in the world, you should learn more stuff from what other humans have done. If, on the other hand, you want to solve the insanely hard engineering/philosophy problem of AGI alignment in time for humanity to not be wiped out, you absolutely should prioritize solving the problem from scratch.
Can you expand on this concisely inline? I agree with the comment you’re replying to strongly and think it has been one of miri’s biggest weaknesses in the past decade that they didn’t build the fortitude to be able to read existing work without becoming confused by its irrelevance. But I also think your and Abram’s research direction intuitions seem like some of the most important in the field right now, alongside wentworth. I’d like to understand what it is that has held you back from speed reading external work for hunch seeding for so long. To me, it seems like solving from scratch is best done not from scratch, if that makes sense. Don’t defer to what you read.
Well currently I’m not really doing alignment research. My plans / goals / orientation / thinking style have changed over the years, so I’ve read stuff or tried to read stuff more or less during different periods. When I’m doing my best thinking, yes, I read things for idea seeding / as provocations, but it’s only that—I most certainly am not speed reading, the opposite really: read one paragraph, think for an hour and then maybe write stuff. And I’m obviously not reading some random ML paper, jesus christ. Philosophy, metamathematics, theoretical biology, linguistics, psychology, ethology, … much more interesting and useful.
Absolutely, I 100% agree, IIUC. I also think:
A great majority of the time, when people talk about reading stuff (to “get up to speed”, to “see what other people have done on the subject”, to “get inspiration”, to “become more informed”, to “see what approaches/questions there are”...), they are not doing this “from scratch not from scratch” thing.
“the typical EA / rationalist, especially in AI safety research (most often relatively young and junior in terms of research experience / taste)” is absolutely and pretty extremely erring on the side of failing to ever even try to solve the actual problem at all.
Yeah, I generally agree (https://tsvibt.blogspot.com/2022/09/dangers-of-deferrence.html), though you probably should defer about some stuff at least provisionally (for example, you should probably try out, for a while, the stance of deferring to well-respected philosophers about what questions are interesting).
I think it’s just not appreciated how much people defer to what they read. Specifically, there’s a lot of frame deference. This is usually fine and good in lots of contexts (you don’t need to, like, question epistemology super hard to become a good engineer, or question whether we should actually be basing our buildings off of liquid material rather than solid material or something). It’s catastrophic in AGI alignment, because our frames are bad.
Not sure I answered your question.
I’d like to offer some serious pushback on the practice of using words like “insane” to describe positions that are not obviously false and which a great number of generally reasonable and well-informed members of the community agree with. It is particularly inappropriate to do that when you have offered no concrete, object-level arguments or explanations [1] for why AI safety researchers should prioritize “solving the problem from scratch.”
Adding in phrases like “it’s exactly opposite of the truth” and “absolutely” not only fails to help your case, but in my view actively makes things worse by using substance-free rhetoric that misleads readers into thinking the case you are bringing forward is stronger than it actually is or that this matter is so obvious and trivial that they shouldn’t even need to think very hard about it before taking your side.
By which I mean, you have included no such arguments in this particular comment, nor have you linked to any other place containing arguments that you agree with on this topic, nor have you offered any explanations in any other comments on this post (I checked, and you have made no other comments on it yet), nor does a cursory look at your profile seem to indicate any posts or recent comments where such ideas might appear.
I disagree re/ the word “insane”. The position to which I stated a counterposition is insane.
I disagree, I think I should state my actual position. The phrases you quoted have meaning and conveys my position more than if they were removed.
It does not matter one bit if this is your “actual position”. The point of community norms about discourse is that they constrain what is or isn’t appropriate to say in a given situation; they function on the meta-level by setting up proper incentives for users to take into account when crafting their contributions here, independently of their personal assessments about who is right on the object-level. So your response is entirely off-topic, and the fact you expected it not to be is revealing of a more fundamental confusion in your thinking about this matter.
Moderation (when done properly) does not act solely to resolve individual disputes on the basis of purely local characteristics to try to ensure specific outcomes. Remember, the law is not an optimizer, but rather a system informed by principles of mechanism design that generates specific, legible, and predictable set of real rules about what is or isn’t acceptable, a system that does not bend in response to the clever arguments of an individual who thinks that he alone is special and exempt from them.
As Duncan Sabien once wrote:
I think it would have been weird and useless for you to straight-up lie in your previous comment, so of course you thought what you were saying communicated your real position. Why else would you have written it? But “communicate whatever you truly feel about something, regardless of what form your writing takes” is a truly terrible way of organizing any community in which meaningful intellectual progress is intended. By contrast, giving explanations and reasoning in situations where you label the beliefs of others as “insane” prevents conversations from becoming needlessly heated and spiraling into Demon Threads, while also building towards a community that maintains high-quality contributions.
All of this stuff has already been covered in the large number of expositions people have given over the last few years on what LessWrong is about and what principles animate norms and user behavior (1, 2, 3, 4, 5, etc).
I doubt that we’re going to get anything useful here, but as an indication of where I’m coming from:
I would basically agree with what you’re saying if my first comment had been ad hominem, like “Bogdan is a doo-doo head”. That’s unhelpful, irrelevant, mean, inflammatory, and corrosive to the culture. (Also it’s false lol.)
I think a position can be wrong, can be insanely wrong (which means something like “is very far from the truth, is wrong in a way that produces very wrong actions, and is being produced by a process which is failing to update in a way that it should and is failing to notice that fact”), and can be exactly opposite of the truth (for example, “Redwoods are short, grass is tall” is, perhaps depending on contexts, just about the exact opposite of the truth). And these facts are often knowable and relevant if true. And therefore should be said—in a truth-seeking context. And this is the situation we’re in.
If you had responded to my original comment with something like
or whatever is the analogous thing that’s true for you, then we could have talked about that. Instead you called my relatively accurate and intentional presentation of my views as “misleading readers into thinking the case you are bringing forward is stronger than it actually is or that this matter is so obvious and trivial...” which sounds to me like you have a problem in your own thinking and norms of discourse, which is that you’re requiring that statements other people make be from the perspective of [the theory that’s shared between the expected community of speakers and listeners] in order for you to think they’re appropriate or non-misleading.
The fact that I have to explain this to you is probably bad, and is probably mostly your responsibility, and you should reevaluate your behavior. (I’m not trying to be gentle here, and if gentleness would help then you deserve it—but you probably won’t get it here from me.)
I’d be open to alternative words for “insane” the way I intended it.
I agree to some extent, which is why I said the following to gears:
The fact that you chose the word “insane” to describe something that did not seem obviously false, had a fair bit of support in this community, and that you had not given any arguments against at the time was the problem.
The fact that you think something is “insane” is informationally useful to other people, and, all else equal, should be communicated. But all else is not equal, because (as I explained in my previous comments), it is a fabricated option to think that relaxing norms around the way in which particular kinds of information is communicating will not negatively affect the quality of the conversation that unfolds afterwards.
So you could (at least in my view, not sure what the mods think) say something is “insane” if you explain why, because this allows for opportunities to drag the conversation away from mud-slinging Demon Threads and towards the object-level arguments being discussed (and, in this case, saying you think your interlocutor’s position is crazy could actually be helpful at times, since it signals a great level of disagreement and allows for the quicker identification of how many inferential distances between you and the other commenters). Likewise, you could give your conclusions without presenting arguments or explanations for it, as long as your position is not stated in an overly inflammatory manner, because this then incentivizes useful and clear-headed discourse later on when users can ask what the arguments actually are. But if you go the third route, then you maximize the likelihood of the conversation getting derailed.
This framing focuses on the wrong part, I think. You can be as angry as you want to when you are commenting on LessWrong, and it seems to be inappropriate to enforce norms about the emotions one is supposed to feel when contributing here. The part that matters is whether specific norms of discourse are getting violated (about the literal things someone is writing, not how they feel in that moment), in which case (as I have argued above) I believe the internal state of mind of the person violating them is primarily irrelevant.
I’m also not sure what you mean by this. You also implied later on that “requiring that statements other people make be from the perspective of [the theory that’s shared between the expected community of speakers and listeners] in order for you to think they’re appropriate” is wrong, which… doesn’t make sense to me, because that’s the very definition of the word appropriate: “meeting the requirements [i.e. norms] of a purpose or situation.”
The same statement can be appropriate or inappropriate, depending on the rules and norms of the community it is made in.
If this happens because someone says something true, relevant, and useful, in a way that doesn’t have alternative expressions that are really easy and obvious to do (such as deleting the statement “So and so is a doo-doo head”), then it’s the fault of the conversation, not the statement.
The alternative expression, in this particular case (not in the mine run of cases), is not to change the word “insane” (because it seems you are certain enough in your belief that it is applicable here that it makes sense for you to communicate this idea some way), but rather to simply write more (or link to a place that contain arguments which relate, with particularity, to the situation at hand) by explaining why you think it’s is true that the statement is “insane”.
If you are so confident in your conclusion that you are willing to label the articulation of the opposing view as “insane”, then it should be straightforward (and more importantly, should not take so much time that it becomes daunting) to give reasons for that, at the time you make that labeling.
NOPE!
I think I’m going to bow out of this conversation right now, since it doesn’t seem you want to meaningfully engage.
The comment I was responding to also didn’t offer serious relevant arguments.
https://tsvibt.blogspot.com/2023/09/a-hermeneutic-net-for-agency.html
And it didn’t label the position it was arguing against as “insane”, so this is also entirely off-topic.
It would be ideal for users to always describe why they have reached the conclusions they have, but that is a fabricated option which does not take into account the basic observation that requiring such explanations creates such a tremendous dis-incentive to commenting that it would drastically reduce the quantity of useful contributions in the community, thus making things worse off than they were before.
So the compromise we reach is one in which users can state their conclusions in a relatively neutral manner that does not poison the discourse that comes afterwards, and then if another user has a question or a disagreement about this matter, later on they can then have a regular, non-Demon Thread discussion about it in which they explain their models and the evidence they had to reach their positions.
I think you are also expressing high emotive confidence in your comments. You are presenting a case, and your expressed confidence slightly lower, but still elevated.
I agree[1], and I think it is entirely appropriate to do so, given that I have given some explanations of the mental models behind my positions on these matters.
For clarity, I’ll summarize my conclusion here, on the basis of what I have explained before (1, 2, 3):
It is fine[2] to label opinions you disagree with as “insane”.
It is fine to give your conclusions without explaining the reasons behind your positions.[3]
It is not fine to do 1 and 2 at the same time.
With regards to your “taboo off-topic” reaction, what I mean by “off-topic” in this case is “irrelevant to the discussion at hand, by focusing on the wrong level of abstraction (meta-level norms vs object-level discourse) and by attempting to say the other person behaved similarly, which is incorrect as a factual matter (see the distinction between points 2 and 3 above), but more importantly, immaterial to the topic at hand even if true”.
I suspect my regular use of italics is part of what is giving off this impression.
Although not ideal in most situations, and should be (lightly) discouraged in most spots.
Although it would be best to be willing to engage in discussion about those reasons later on if other users challenge you on them.
I’m time-bottlenecked now, but I’ll give one example. Consider the Natural Abstraction Hypothesis (NAH) agenda (which, fwiw, I think is an example of considerably-better-than-average work on trying to solve the problem from scratch). I’d argue that even for someone interested in this agenda: 1. most of the relevant work has come (and will keep coming) from outside the LW community (see e.g. The Platonic Representation Hypothesis and compare the literature reviewed there with NAH-related work on LW); 2. (given the previous point) the typical AI safety researcher interested in NAH would do better to spend most of their time (at least at the very beginning) looking at potentially relevant literature outside LW, rather than either trying to start from scratch, or mostly looking at LW literature.
It’s considerably better than average but is a drop in the bucket and is probably mostly wasted motion. And it’s a pretty noncentral example of trying to solve the problem from scratch. I think most people reading this comment just don’t even know what that would look like.
At a glance, this comment seems like it might be part of a pretty strong case that [the concrete ML-related implications of NAH] are much better investigated by the ML community compared to LW alignment people. I doubt that the philosophically more interesting aspects of Wentworth’s perspectives relating to NAH are better served by looking at ML stuff, compared to trying from scratch or looking at Wentworth’s and related LW-ish writing. (I’m unsure about the mathematically interesting aspects; the alternative wouldn’t be in the ML community but would be in the mathematical community.)
And most importantly “someone interested in this agenda” is already a somewhat nonsensical or question-begging conditional. You brought up “AI safety research” specifically, and by that term you are morally obliged to mean [the field of study aimed at figuring out how to make cognitive systems that are more capable than humanity and also serve human value]. That pursuit is better served by trying from scratch. (Yes, I still haven’t presented an affirmative case. That’s because we haven’t even communicated about the proposition yet.)
Links have high attrition rate, cf ratio of people overcoming a trivial inconvenience. Post your arguments compressed inline to get more eyeballs on them.
I think this is particularly incorrect for alignment, relative to a more typical STEM research field. Alignment is very young[1]. There’s a lot less existing work worth reading than in a field like, say, lattice quantum field theory. Due to this, the time investment required to start contributing at the research frontier is very low, relatively speaking.
This is definitely changing. There’s a lot more useful work than there was when I started dipping my toe into alignment three years ago. But compared to something like particle physics, it’s still very little.
In terms of # total smart people hours invested