Can someone give an example of a disaster involving such an AI?
The AI might find answers that satisfy the question but violate background assumptions we never thought to include and wouldn’t realize until it was too late (if even then). An easy-to-imagine one that we wouldn’t fall for is a cure for cancer that succeeds by eradicating all cellular life. Of course, it’s more difficult to come up with one that we would fall for, but anything involving cognitive modifications would be a candidate.
So, the reason we wouldn’t fall for that one is that the therapy wouldn’t pass the safety tests required by first-world governments. We have safety tests for all sorts of new technologies, with the stringency of the tests depending on the kind of technology — some testing for children’s toys, more testing for drugs, hopefully more testing for permanent cognitive enhancement. It seems like these tests should protect us from a Question-Answerer as much as from human mistakes.
Actual unfriendly AI seems scarier because it could try to pass our safety tests, in addition to accomplishing its terminal goals. But a Question-Answerer designing something that passes all the tests and nevertheless causes disaster seems about as likely as a well-intentioned but not completely competent human doing the same.
I guess I should have asked for a disaster involving a Question-Answerer which is more plausible than the same scenario with the AI replaced by a human.
The AI might find answers that satisfy the question but violate background assumptions we never thought to include and wouldn’t realize until it was too late (if even then). An easy-to-imagine one that we wouldn’t fall for is a cure for cancer that succeeds by eradicating all cellular life. Of course, it’s more difficult to come up with one that we would fall for, but anything involving cognitive modifications would be a candidate.
So, the reason we wouldn’t fall for that one is that the therapy wouldn’t pass the safety tests required by first-world governments. We have safety tests for all sorts of new technologies, with the stringency of the tests depending on the kind of technology — some testing for children’s toys, more testing for drugs, hopefully more testing for permanent cognitive enhancement. It seems like these tests should protect us from a Question-Answerer as much as from human mistakes.
Actual unfriendly AI seems scarier because it could try to pass our safety tests, in addition to accomplishing its terminal goals. But a Question-Answerer designing something that passes all the tests and nevertheless causes disaster seems about as likely as a well-intentioned but not completely competent human doing the same.
I guess I should have asked for a disaster involving a Question-Answerer which is more plausible than the same scenario with the AI replaced by a human.