I regularly encounter the impression that AI safety people are significantly afraid about direct consequences of open sourcing current models, from those who don’t understand the actual concerns. I don’t particularly see it from those who do. This (from what I can tell, false) impression seems to be one of relatively few major memes that keep people from bothering to investigate. I hypothesize that this dynamic of ridiculing of AI safety with such memes is what keeps them alive, instead of there being significant truth to them keeping them alive.
To be clear: The mechanism you’re hypothesizing is:
Critics say “AI alignment is dumb because you want to ban open source AI!”
Naive supporters read this, believe the claim that AI alignment-ers want to ban open sourcing AI and think ‘AI alignment is not dumb, therefore open sourcing AI must be bad’. When the next weight release happens they say “This is bad! Open sourcing weights is bad and should be banned!”
Naive supporters read other naive supporters saying this, and believe it themselves. Wise supporters try to explain no, but are either labeled as a critic or weird & ignored.
Thus, a group think is born. Perhaps some wise critics “defer to the community” on the subject.
I don’t think here is a significant confused naive supporter source of the meme that gives it teeth. It’s more that reasonable people who are not any sort of supporters of AI safety propagate this idea, on the grounds that it illustrates the way AI safety is not just dumb, but also dangerous, and therefore worth warning others about.
From the supporter side, “Open Model Weights are Unsafe and Nothing Can Fix This” is a shorter and more convenient way of gesturing to the concern, and convenience is the main force in the Universe that determines all that actually happens in practice. On naive reading such gesturing centrally supports the meme. This doesn’t require the source of such support to have a misconception or to oppose publishing open weights of current models on the grounds of direct consequences.
I regularly encounter the impression that AI safety people are significantly afraid about direct consequences of open sourcing current models, from those who don’t understand the actual concerns. I don’t particularly see it from those who do. This (from what I can tell, false) impression seems to be one of relatively few major memes that keep people from bothering to investigate. I hypothesize that this dynamic of ridiculing of AI safety with such memes is what keeps them alive, instead of there being significant truth to them keeping them alive.
To be clear: The mechanism you’re hypothesizing is:
Critics say “AI alignment is dumb because you want to ban open source AI!”
Naive supporters read this, believe the claim that AI alignment-ers want to ban open sourcing AI and think ‘AI alignment is not dumb, therefore open sourcing AI must be bad’. When the next weight release happens they say “This is bad! Open sourcing weights is bad and should be banned!”
Naive supporters read other naive supporters saying this, and believe it themselves. Wise supporters try to explain no, but are either labeled as a critic or weird & ignored.
Thus, a group think is born. Perhaps some wise critics “defer to the community” on the subject.
I don’t think here is a significant confused naive supporter source of the meme that gives it teeth. It’s more that reasonable people who are not any sort of supporters of AI safety propagate this idea, on the grounds that it illustrates the way AI safety is not just dumb, but also dangerous, and therefore worth warning others about.
From the supporter side, “Open Model Weights are Unsafe and Nothing Can Fix This” is a shorter and more convenient way of gesturing to the concern, and convenience is the main force in the Universe that determines all that actually happens in practice. On naive reading such gesturing centrally supports the meme. This doesn’t require the source of such support to have a misconception or to oppose publishing open weights of current models on the grounds of direct consequences.