There are things that are robustly good in the world, and things that are good on highly specific inside-view models and terrible if those models are wrong. Slowing dangerous tech development seems like the former, whereas forwarding arms races for dangerous tech between world superpowers seems more like the latter.
It may seem the opposite to some people. For instance, my impression is that for many adjacent to the US government, “being ahead of China in every technology” would be widely considered robustly good, and nobody would question you at all if you said that was robustly good. Under this perspective the idea that AI could pose an existential risk is a “highly specific inside-view model” and it would be terrible if we acted on the model and it is wrong.
I don’t think your readers will mostly think this, but I actually think a lot of people would, which for me makes this particular argument seem entirely subjective and thus suspect.
I’m confused, of course the people in government in every country thinks that they should have more global power, but this doesn’t seem like something everyone (i.e. including people in all of the other countries) would agree is robustly good, and I don’t think you should think so either (for any country, be it Saudi Arabia, France, or South Korea). I am not aware of a coherent perspective that says “slowing down dangerous tech development” is not robustly good in most situations (conditional on our civilization’s inability to “put black balls back into the urn”, a la Bostrom).
Your argument sounds to me like “A small group with a lot of political power disagrees with your claim therefore it cannot be accepted as true.” Care to make a better argument?
I think the claim being made is that the “dangerous” part of “slowing down dangerous tech development” is the analogous “highly specific inside-view model” which would be terrible to act on, if it were wrong. That seems valid to me. Obviously I believe that highly specific inside-view model is not wrong, but, you know, that’s how it goes.
I… don’t think that ThomasW is himself challenging the assumption. He’s saying “Read this from the perspective of someone in the US government, doesn’t seem so compelling now does it?” I’m not sure what about the post gave the impression that it was aimed to be persuasive to (for example) the US Secretary of State, but I am confident Katja did not write it for them.
The claim being made is something like the following:
1) AGI is a dangerous technology.
2) It is robustly good to slow down dangerous technologies.
3) Some people might say that you should not actually do this because of [complicated unintelligible reason].
4) But you should just do the thing that is more robustly good.
I argue that many people (yes, you’re right, in ways that conflict with one another) believe the following:
1) X is a dangerous country.
2) It is robustly good to always be ahead of X in all technologies, including dangerous ones.
3) Some people might say that you should not actually do this because of [complicated unintelligible reason]. This doesn’t make very much sense.
4) But you should just do the thing that is more robustly good.
My point is that which argument is the obvious, robust one, and which one is the weird inside view one depends on your perspective. Therefore, it doesn’t seem like (4) is a very good generalized argument. For example, if I were one of these powerful people, I think it would be wrong for me to be convinced to “focus on the robustly good measures, not the weird inside view measures” because it would lead me to do bad things like trying to advance AI capabilities. As a result, the argument seems suspect to me. It feels like it only works for this community, or people who are already very concerned by AI x-risk.
In comparison, there are specific arguments like “AGI is dangerous” and “slowing down dangerous technologies is actually robustly good” (some of these were presented in this post) that I think are, ironically, must more robustly good, because they don’t seem to have negative effects as reliably when presented to people who hold beliefs I think are wrong.
Edit: I no longer endorse this comment. It claims too much, specifically that any reasoning procedure is suspect if it leads to people who believe false premises taking bad actions.
I think what I was really trying to get at in my original comment was that that particular argument seems aimed at people who already think that it would be robustly good to slow down dangerous technologies. But the people who would most benefit from this post are those who do not already think this; for them it doesn’t help much and might actively hurt.
I think what I was really trying to get at in my original comment was that that particular argument seems aimed at people who already think that it would be robustly good to slow down dangerous technologies. But the people who would most benefit from this post are those who do not already think this; for them it doesn’t help much and might actively hurt.
This is kind of a strange comment to me. The argument, and indeed the whole post, is clearly written to people in the ecosystem (“my impression is that for people worried about extinction risk from artificial intelligence, strategies under the heading ‘actively slow down AI progress’ have historically been dismissed and ignored”), for which differential technological progress is a pretty common concept and relied upon in lots of arguments. It’s pretty clear that this post is written to point out an undervalued position to those people.
Sometimes I feel like people in the AI x-risk ecosystem who interface with policy and DC replace their epistemologies with a copy of the epistemology they find in various parts of the policy-control machine in DC, in order to better predict them and perform the correct signals — asking themselves what people in DC would think, rather than what they themselves would think. I don’t know why you think this post was aimed at those people, or why you point out that the post is making false inferences about its audience when the post is pretty clear that it’s primary audience is the people directly in the ecosystem (“The conversation near me over the years has felt a bit like this”).
I just do not think that the post is written for people who think “slowing down AI capabilities is robustly good.” If people thought that, then why do they need this post? Surely they don’t need somebody to tell them to think about it?
So it seems to me like the best audience for this post would be those (including those at some AI companies, or those involved in policy, which includes people reading this post) who currently think something else, for example that the robustly good thing is for their chosen group to be ahead so that they can execute whatever strategy they think they alone can do correctly.
The people I’ve met who don’t want to think about slowing down AI capabilities just don’t seem to think that slowing down AI progress would be robustly good, because that just wouldn’t be a consistent view! They often seem to have some view that nothing is robustly good, or maybe some other thing (“get more power”) is robustly good. Such people just won’t really be swayed by the robust priors thing, or maybe they’d be swayed in the other direction.
I see. You’re not saying “staffers of the US government broadly won’t find this argument persuasive”, you’re saying “there are some people in the AI x-risk ecosystem who don’t think slowing down is robustly good, and won’t find this particular argument persuasive”.
I have less of a disagreement with that sentence.
I’ll add that:
I think most of the arguments in the post are relevant to those people, and Katja only says that these moods are “playing a role” which does not mean all people agree with them.
You write “If people thought that, then why do they need this post? Surely they don’t need somebody to tell them to think about it?”. Sometimes people need help noticing the implications of their beliefs, due to all sorts of motivated cognitions. I don’t think the post relies on that and it shouldn’t be the primary argument, but I think it’s honestly helpful for some people (and was a bit helpful for me to read it).
It may seem the opposite to some people. For instance, my impression is that for many adjacent to the US government, “being ahead of China in every technology” would be widely considered robustly good, and nobody would question you at all if you said that was robustly good. Under this perspective the idea that AI could pose an existential risk is a “highly specific inside-view model” and it would be terrible if we acted on the model and it is wrong.
I don’t think your readers will mostly think this, but I actually think a lot of people would, which for me makes this particular argument seem entirely subjective and thus suspect.
I’m confused, of course the people in government in every country thinks that they should have more global power, but this doesn’t seem like something everyone (i.e. including people in all of the other countries) would agree is robustly good, and I don’t think you should think so either (for any country, be it Saudi Arabia, France, or South Korea). I am not aware of a coherent perspective that says “slowing down dangerous tech development” is not robustly good in most situations (conditional on our civilization’s inability to “put black balls back into the urn”, a la Bostrom).
Your argument sounds to me like “A small group with a lot of political power disagrees with your claim therefore it cannot be accepted as true.” Care to make a better argument?
I think the claim being made is that the “dangerous” part of “slowing down dangerous tech development” is the analogous “highly specific inside-view model” which would be terrible to act on, if it were wrong. That seems valid to me. Obviously I believe that highly specific inside-view model is not wrong, but, you know, that’s how it goes.
I… don’t think that ThomasW is himself challenging the assumption. He’s saying “Read this from the perspective of someone in the US government, doesn’t seem so compelling now does it?” I’m not sure what about the post gave the impression that it was aimed to be persuasive to (for example) the US Secretary of State, but I am confident Katja did not write it for them.
The claim being made is something like the following:1) AGI is a dangerous technology.2) It is robustly good to slow down dangerous technologies.3) Some people might say that you should not actually do this because of [complicated unintelligible reason].4) But you should just do the thing that is more robustly good.I argue that many people (yes, you’re right, in ways that conflict with one another) believe the following:1) X is a dangerous country.2) It is robustly good to always be ahead of X in all technologies, including dangerous ones.3) Some people might say that you should not actually do this because of [complicated unintelligible reason]. This doesn’t make very much sense.4) But you should just do the thing that is more robustly good.My point is that which argument is the obvious, robust one, and which one is the weird inside view one depends on your perspective. Therefore, it doesn’t seem like (4) is a very good generalized argument. For example, if I were one of these powerful people, I think it would be wrong for me to be convinced to “focus on the robustly good measures, not the weird inside view measures” because it would lead me to do bad things like trying to advance AI capabilities. As a result, the argument seems suspect to me. It feels like it only works for this community, or people who are already very concerned by AI x-risk.In comparison, there are specific arguments like “AGI is dangerous” and “slowing down dangerous technologies is actually robustly good” (some of these were presented in this post) that I think are, ironically, must more robustly good, because they don’t seem to have negative effects as reliably when presented to people who hold beliefs I think are wrong.Edit: I no longer endorse this comment. It claims too much, specifically that any reasoning procedure is suspect if it leads to people who believe false premises taking bad actions.
I think what I was really trying to get at in my original comment was that that particular argument seems aimed at people who already think that it would be robustly good to slow down dangerous technologies. But the people who would most benefit from this post are those who do not already think this; for them it doesn’t help much and might actively hurt.
This is kind of a strange comment to me. The argument, and indeed the whole post, is clearly written to people in the ecosystem (“my impression is that for people worried about extinction risk from artificial intelligence, strategies under the heading ‘actively slow down AI progress’ have historically been dismissed and ignored”), for which differential technological progress is a pretty common concept and relied upon in lots of arguments. It’s pretty clear that this post is written to point out an undervalued position to those people.
Sometimes I feel like people in the AI x-risk ecosystem who interface with policy and DC replace their epistemologies with a copy of the epistemology they find in various parts of the policy-control machine in DC, in order to better predict them and perform the correct signals — asking themselves what people in DC would think, rather than what they themselves would think. I don’t know why you think this post was aimed at those people, or why you point out that the post is making false inferences about its audience when the post is pretty clear that it’s primary audience is the people directly in the ecosystem (“The conversation near me over the years has felt a bit like this”).
I just do not think that the post is written for people who think “slowing down AI capabilities is robustly good.” If people thought that, then why do they need this post? Surely they don’t need somebody to tell them to think about it?
So it seems to me like the best audience for this post would be those (including those at some AI companies, or those involved in policy, which includes people reading this post) who currently think something else, for example that the robustly good thing is for their chosen group to be ahead so that they can execute whatever strategy they think they alone can do correctly.
The people I’ve met who don’t want to think about slowing down AI capabilities just don’t seem to think that slowing down AI progress would be robustly good, because that just wouldn’t be a consistent view! They often seem to have some view that nothing is robustly good, or maybe some other thing (“get more power”) is robustly good. Such people just won’t really be swayed by the robust priors thing, or maybe they’d be swayed in the other direction.
I see. You’re not saying “staffers of the US government broadly won’t find this argument persuasive”, you’re saying “there are some people in the AI x-risk ecosystem who don’t think slowing down is robustly good, and won’t find this particular argument persuasive”.
I have less of a disagreement with that sentence.
I’ll add that:
I think most of the arguments in the post are relevant to those people, and Katja only says that these moods are “playing a role” which does not mean all people agree with them.
You write “If people thought that, then why do they need this post? Surely they don’t need somebody to tell them to think about it?”. Sometimes people need help noticing the implications of their beliefs, due to all sorts of motivated cognitions. I don’t think the post relies on that and it shouldn’t be the primary argument, but I think it’s honestly helpful for some people (and was a bit helpful for me to read it).
Yeah, I agree with all this.
Thread success!