SIAI’s narrow focus on things that “look like HAL” neglects the risks of entities that are formed of humans and computers (and other objects) interacting. These entities already exist, they’re already beyond human intelligence, and they’re already existential risks.
Indeed, Lesswrong and SIAI are two obvious examples of these entities, and it’s not clear at all how to steer them to become Friendly. Increasing individual rationality will help, but we also need to do social engineering—checks and balances and incentives (not just financial, but social incentives such as attention and praise) - and groupware research (e.g. karma and moderation systems, expert aggregation).
I don’t think that “entities that are formed of humans and computers (and other objects) interacting” is sufficiently specific to be considered a type of existential risk. Any organization can be put into that category and unlike AGI, it’s not true that most possible organizations have goal systems indifferent to human morals.
Also, the fact that organizations can be dangerous is well known and there doesn’t seem to be a simple solution to that or anything else a small organization could do. The problem isn’t about coming up with checks and balances or incentive systems, it’s about making people sane enough to use those solutions.
I don’t think that “entities that are formed of humans and computers (and other objects) interacting” is sufficiently specific to be considered a type of existential risk.
True, but Johnicholas still has a point about “things that look like HAL,” namely, that such scenarios presents the uFAI risk in an unconvincing manner. To most people, I suspect a scenario in which individuals and organizations gradually come to depend too much on AI would be more plausible.
On some measures (breadth of knowledge, responsiveness at all hours, words-typed-per-month), LW is superhuman. On most other measures, LW can default to using one of its (human) component’s capabilities, and thereby achieve human- comparable performance. I admit it has problems with cohesiveness and coherency.
SIAI’s narrow focus on things that “look like HAL” neglects the risks of entities that are formed of humans and computers (and other objects) interacting. These entities already exist, they’re already beyond human intelligence, and they’re already existential risks.
Indeed, Lesswrong and SIAI are two obvious examples of these entities, and it’s not clear at all how to steer them to become Friendly. Increasing individual rationality will help, but we also need to do social engineering—checks and balances and incentives (not just financial, but social incentives such as attention and praise) - and groupware research (e.g. karma and moderation systems, expert aggregation).
I don’t think that “entities that are formed of humans and computers (and other objects) interacting” is sufficiently specific to be considered a type of existential risk. Any organization can be put into that category and unlike AGI, it’s not true that most possible organizations have goal systems indifferent to human morals.
Also, the fact that organizations can be dangerous is well known and there doesn’t seem to be a simple solution to that or anything else a small organization could do. The problem isn’t about coming up with checks and balances or incentive systems, it’s about making people sane enough to use those solutions.
True, but Johnicholas still has a point about “things that look like HAL,” namely, that such scenarios presents the uFAI risk in an unconvincing manner. To most people, I suspect a scenario in which individuals and organizations gradually come to depend too much on AI would be more plausible.
What makes you think LW is smarter than a human?
On some measures (breadth of knowledge, responsiveness at all hours, words-typed-per-month), LW is superhuman. On most other measures, LW can default to using one of its (human) component’s capabilities, and thereby achieve human- comparable performance. I admit it has problems with cohesiveness and coherency.