If SIAI had the ethos I’d like, we’d be going over and kicking every one of the supporting arguments for the likelihood of fooming and the nature of intelligence to make sure they were sound.
And then, hypothetically, if they found that fooming is not likely at all, and that dangerous fooming can be rendered nearly impossible by some easily enforced precautions/regulations, what then? If they found that the SIAI has no particular unique expertise to contribute to the development of FAI? An organization with an ethos you would like: what would it do then? To make it a bit more interesting, suppose they find themselves sitting on a substantial endowment when they reason their way to their own obsolescence?
How often in human history have organizations announced, “Mission accomplished—now we will release our employees to go out and do something else”?
It doesn’t seem likely. The paranoid can usually find something scary to worry about. If something turns out to be not really-frightening, fear mongers can just go on to the next-most frightening thing in line. People have been concerned about losing their jobs to machines for over a century now. Machines are a big and scary enough domain to keep generating fear for a long time.
I think that what SIAI works on is real and urgent, but if I’m wrong and what you describe here does come to pass, the world gets yet another organisation campaigning about something no-one sane should care about. It doesn’t seem like a disastrous outcome.
From a less cynical angle, building organizations is hard. If an organization has fulfilled its purpose, or that purpose turns out to be a mistake, it isn’t awful to look for something useful for the organization to do rather than dissolving it.
The American charity organization, The March of Dimes was originally created to combat polio. Now they are involved with birth defects and other infant health issues.
Since they are the one case I know of (other than ad hoc disaster relief efforts) in which an organized charity accomplished its mission, I don’t begrudge them a few additional decades of corporate existence.
And then, hypothetically, if they found that fooming is not likely at all, and that dangerous fooming can be rendered nearly impossible by some easily enforced precautions/regulations, what then? If they found that the SIAI has no particular unique expertise to contribute to the development of FAI? An organization with an ethos you would like: what would it do then? To make it a bit more interesting, suppose they find themselves sitting on a substantial endowment when they reason their way to their own obsolescence?
How often in human history have organizations announced, “Mission accomplished—now we will release our employees to go out and do something else”?
It doesn’t seem likely. The paranoid can usually find something scary to worry about. If something turns out to be not really-frightening, fear mongers can just go on to the next-most frightening thing in line. People have been concerned about losing their jobs to machines for over a century now. Machines are a big and scary enough domain to keep generating fear for a long time.
I think that what SIAI works on is real and urgent, but if I’m wrong and what you describe here does come to pass, the world gets yet another organisation campaigning about something no-one sane should care about. It doesn’t seem like a disastrous outcome.
From a less cynical angle, building organizations is hard. If an organization has fulfilled its purpose, or that purpose turns out to be a mistake, it isn’t awful to look for something useful for the organization to do rather than dissolving it.
The American charity organization, The March of Dimes was originally created to combat polio. Now they are involved with birth defects and other infant health issues.
Since they are the one case I know of (other than ad hoc disaster relief efforts) in which an organized charity accomplished its mission, I don’t begrudge them a few additional decades of corporate existence.