I’m still not sure why exactly people (I’m thinking of a few in particular, but this applies to many in the field) tell very detailed stories of AI domination like “AI will use protein nanofactories to embed tiny robots in our bodies to destroy all of humanity at the press of a button.” This seems like a classic use of the conjunction fallacy, and it doesn’t seem like those people really flinch from the word “and” like the Sequences tell them they should.
Furthermore, it seems like people within AI alignment aren’t taking the “sci-fi” criticism as seriously as they could. I don’t think most people who have that objection are saying “this sounds like science fiction, therefore it’s wrong.” I think they’re more saying “these hypothetical scenarios are popular because they make good science fiction, not because they’re likely.” And I have yet to find a strong argument against the latter form of that point.
Please let me know if I’m doing an incorrect “steelman,” or if I’m missing something fundamental here.
I don’t think the point of the detailed stories is that they strongly expect that particular thing to happen? It’s just useful to have a concrete possibility in mind.
I think they’re more saying “these hypothetical scenarios are popular because they make good science fiction, not because they’re likely.” And I have yet to find a strong argument against the latter form of that point.
Yeah I imagine that’s hard to argue against, because it’s basically correct, but importantly it’s also not a criticism of the ideas. If someone makes the argument “These ideas are popular, and therefore probably true”, then it’s a very sound criticism to point out that they may be popular for reasons other than being true. But if the argument is “These ideas are true because of <various technical and philosophical arguments about the ideas themselves>”, then pointing out a reason that the ideas might be popular is just not relevant to the question of their truth. Like, cancer is very scary and people are very eager to believe that there’s something that can be done to help, and, perhaps partly as a consequence, many come to believe that chemotherapy can be effective. This fact does not constitute a substantive criticism of the research on the effectiveness of chemotherapy.
I’m still not sure why exactly people (I’m thinking of a few in particular, but this applies to many in the field) tell very detailed stories of AI domination like “AI will use protein nanofactories to embed tiny robots in our bodies to destroy all of humanity at the press of a button.” This seems like a classic use of the conjunction fallacy, and it doesn’t seem like those people really flinch from the word “and” like the Sequences tell them they should.
Furthermore, it seems like people within AI alignment aren’t taking the “sci-fi” criticism as seriously as they could. I don’t think most people who have that objection are saying “this sounds like science fiction, therefore it’s wrong.” I think they’re more saying “these hypothetical scenarios are popular because they make good science fiction, not because they’re likely.” And I have yet to find a strong argument against the latter form of that point.
Please let me know if I’m doing an incorrect “steelman,” or if I’m missing something fundamental here.
I don’t think the point of the detailed stories is that they strongly expect that particular thing to happen? It’s just useful to have a concrete possibility in mind.
Yeah I imagine that’s hard to argue against, because it’s basically correct, but importantly it’s also not a criticism of the ideas. If someone makes the argument “These ideas are popular, and therefore probably true”, then it’s a very sound criticism to point out that they may be popular for reasons other than being true. But if the argument is “These ideas are true because of <various technical and philosophical arguments about the ideas themselves>”, then pointing out a reason that the ideas might be popular is just not relevant to the question of their truth.
Like, cancer is very scary and people are very eager to believe that there’s something that can be done to help, and, perhaps partly as a consequence, many come to believe that chemotherapy can be effective. This fact does not constitute a substantive criticism of the research on the effectiveness of chemotherapy.