So it isn’t so much that I think people who would work on X-risk would be redirected, as that I think there will be a point where people adjacent to X-risk research will have difficulty telling which people are actually trying to work on X-risk, and also what the state of the X-risk concerns is (I mean to what extent it has been addressed by the research).
That makes more sense. I haven’t thought enough about this aspect to have a strong opinion yet. My initial thoughts are that
this problem can be basically avoided if this kind of work clearly points out where the problems would be if scaled.
I do think it’s plausible that some less-connected funding sources might get confused (NSF), but I’d be surprised if later FLI funding got diverted because of this. I think this kind of work will be done anyways, and it’s better to have people who think carefully about scale issues doing it.
your second bullet point reminds me of how some climate change skeptics will point to “evidence” from “scientists”, as if that’s what convinced them. In reality, however, they’re drawing the bottom line first, and then pointing to what they think is the most dignified support for their position. I don’t think that avoiding this kind of work would ameliorate that problem—they’d probably just find other reasons.
most people on the outside don’t understand x-risk anyways, because it requires thinking rigorously in a lot of ways to not end up a billion miles off of any reasonable conclusion. I don’t think that this additional straw will marginally add significant confusion.
IE, what’s important is not any contribution to near- or medium- term AI safety
I’m confused how “contributes meaningfully to short-term safety” and “maybe useful for having better thoughts” are mutually-exclusive outcomes, or why it’s wrong to say that I think my work contributes to short-term efforts. Sure, that may not be what you care about, but I think it’s still reasonable that I mention it.
I’m saying that exploratory work should not be justified as “confident that this contributes meaningfully to short-term safety”. Almost everything at this stage is more like “maybe useful for one day having better thoughts about reducing X-risk, maybe not”.
I’m confused why that latter statement wasn’t what came across! Later in that sentence, I state that I don’t think it will scale. I also made sure to highlight how it breaks down in a serious way when scaled up, and I don’t think I otherwise implied that it’s presently safe for long-term efforts.
I totally agree that having better thoughts about x-risk is a worthy goal at this point.
I’m confused how “contributes meaningfully to short-term safety” and “maybe useful for having better thoughts” are mutually-exclusive outcomes, or why it’s wrong to say that I think my work contributes to short-term efforts. Sure, that may not be what you care about, but I think it’s still reasonable that I mention it.
In hindsight I am regretting the way my response went. While it was my honest response, antagonizing newcomers to the field for paying any attention to whether their work might be useful for sub-AGI safety doesn’t seem like a good way to create the ideal research atmosphere. Sorry for being a jerk about it.
Although I did flinch a bit, my S2 reaction was “this is Abram, so if it’s criticism, it’s likely very high-quality. I’m glad I’m getting detailed feedback, even if it isn’t all positive”. Apology definitely accepted (although I didn’t view you as being a jerk), and really—thank you for taking the time to critique me a bit. :)
That makes more sense. I haven’t thought enough about this aspect to have a strong opinion yet. My initial thoughts are that
this problem can be basically avoided if this kind of work clearly points out where the problems would be if scaled.
I do think it’s plausible that some less-connected funding sources might get confused (NSF), but I’d be surprised if later FLI funding got diverted because of this. I think this kind of work will be done anyways, and it’s better to have people who think carefully about scale issues doing it.
your second bullet point reminds me of how some climate change skeptics will point to “evidence” from “scientists”, as if that’s what convinced them. In reality, however, they’re drawing the bottom line first, and then pointing to what they think is the most dignified support for their position. I don’t think that avoiding this kind of work would ameliorate that problem—they’d probably just find other reasons.
most people on the outside don’t understand x-risk anyways, because it requires thinking rigorously in a lot of ways to not end up a billion miles off of any reasonable conclusion. I don’t think that this additional straw will marginally add significant confusion.
I’m confused how “contributes meaningfully to short-term safety” and “maybe useful for having better thoughts” are mutually-exclusive outcomes, or why it’s wrong to say that I think my work contributes to short-term efforts. Sure, that may not be what you care about, but I think it’s still reasonable that I mention it.
I’m confused why that latter statement wasn’t what came across! Later in that sentence, I state that I don’t think it will scale. I also made sure to highlight how it breaks down in a serious way when scaled up, and I don’t think I otherwise implied that it’s presently safe for long-term efforts.
I totally agree that having better thoughts about x-risk is a worthy goal at this point.
In hindsight I am regretting the way my response went. While it was my honest response, antagonizing newcomers to the field for paying any attention to whether their work might be useful for sub-AGI safety doesn’t seem like a good way to create the ideal research atmosphere. Sorry for being a jerk about it.
Although I did flinch a bit, my S2 reaction was “this is Abram, so if it’s criticism, it’s likely very high-quality. I’m glad I’m getting detailed feedback, even if it isn’t all positive”. Apology definitely accepted (although I didn’t view you as being a jerk), and really—thank you for taking the time to critique me a bit. :)