[In the context of SIAI folks thinking an unpleasant AI was likely]
The SIAI derives its funding from convincing people that the end is probably
nigh—and that they are working on a potential solution. This is not the type
of organisation you should trust to be objective on such an issue—they have
obvious vested interests.
I’ve noticed this structural vulnerability to bias too—Can you think of any structural changes that might reduce or eliminate this bias?
Maybe SIAI ought to be offering a prize for substantially justified criticism of some important positional documents, as judged by some disinterested agent?
I think I made much the same points in my DOOM! video. DOOM mongers:
tend to do things like write books about THE END OF THE WORLD—which gives them a stake in promoting the topic …and...
are a self-selected sample of those who think DOOM is very important (and so, often, highly likely) - so naturally they hold extreme views—and represent a sample from the far end of the spectrum;
clump together, cite each others papers, and enjoy a sense of community based around their unusual views.
It seems tricky for the SIAI to avoid the criticism that they have a stake in promoting the idea of DOOM—while they are funded the way they are.
Similarly, I don’t see an easy way of avoiding the criticism that they are a self-selected sample from the extreme end of a spectrum of DOOM beliefs either.
If we could independently establish p(DOOM), that would help—but measuring it seems pretty challenging.
IMO, a prize wouldn’t help much—but I don’t know for sure. Many people behave irrationally around prizes—so it is hard to be very confident here.
I gather they are working on publishing some positional documents. It seems to be a not-unreasonable move. If there is something concrete to criticise, critics will have something to get their teeth into.
[In the context of SIAI folks thinking an unpleasant AI was likely]
The SIAI derives its funding from convincing people that the end is probably nigh—and that they are working on a potential solution. This is not the type of organisation you should trust to be objective on such an issue—they have obvious vested interests.
I’ve noticed this structural vulnerability to bias too—Can you think of any structural changes that might reduce or eliminate this bias?
Maybe SIAI ought to be offering a prize for substantially justified criticism of some important positional documents, as judged by some disinterested agent?
They are already getting some critical feedback.
I think I made much the same points in my DOOM! video. DOOM mongers:
tend to do things like write books about THE END OF THE WORLD—which gives them a stake in promoting the topic …and...
are a self-selected sample of those who think DOOM is very important (and so, often, highly likely) - so naturally they hold extreme views—and represent a sample from the far end of the spectrum;
clump together, cite each others papers, and enjoy a sense of community based around their unusual views.
It seems tricky for the SIAI to avoid the criticism that they have a stake in promoting the idea of DOOM—while they are funded the way they are.
Similarly, I don’t see an easy way of avoiding the criticism that they are a self-selected sample from the extreme end of a spectrum of DOOM beliefs either.
If we could independently establish p(DOOM), that would help—but measuring it seems pretty challenging.
IMO, a prize wouldn’t help much—but I don’t know for sure. Many people behave irrationally around prizes—so it is hard to be very confident here.
I gather they are working on publishing some positional documents. It seems to be a not-unreasonable move. If there is something concrete to criticise, critics will have something to get their teeth into.
For the curious: DOOM!