I tried to think of the most harmless thing. Something I loved from my childhood. Something that could never ever possibly destroy us.
A thought occurred to me a while back. Call it the “Ghostbusters” approach to the existential risk of AI research. The basic idea is that rather than trying to make the best FAI on the first try, you hedge your bets. Work to make an AI that is a)unlikely to disrupt human civilization in a permanent way at all, and b)available for study.
Part of the stress of the ‘one big AI’ interpretation of the intelligence explosion is the sense that we’d better get it right the first time. But on the other hand, surely the space of all nonthreatening superintelligences is larger than the space of all helpful ones, and a comparatively easier target to hit on our first shot. You’re still taking a gamble. But minimizing this risk seems much easier when you are not simultaneously trying to change human experience in positive ways. And having performed the action once, there would be a wealth of new information to inform later choices.
So I’m trying to decide if this is obviously true or obviously false: p(being destroyed by a primary FAI attempt) > p(being destroyed by a “Ghostbusters” attempt) * p(being destroyed by a subsequent more informed FAI attempt)
If you’re making AI for study it shouldn’t be super-intelligent at all, ideally it should be dumber than you. I can imagine an AGI that can usefully perform some tasks but is too stupid to self-modify itself into fooming if constrained. You can let it be in charge of opening and closing doors!
Well, I definitely agree that we should make non-super intelligent AIs for study, and also for a great many other reasons. But it’s perhaps less clear what ‘too stupid to foom’ actually means for an AGI. There was a moment when a hominid brain crossed an invisible line and civilization became possible; but the mutation precipitating that change may not have obviously been a major event from the perspective of an outside observer. It may just have looked like another in a sequence of iterative steps. Is the foom line in about the same place as the agriculture line? Is it simpler? Harder?
On the other hand, it’s possible to imagine an experimental AGI with values like “Fulfill [utility function X] in the strictly defined spatial domain of Neptune, using only materials that were contained in the gravity well of Neptune in the year 2000, including the construction of your own brain, and otherwise avoid >epsilon changes to probable outcomes for the universe outside the domain of Neptune.” Then fill in whatever utility function you’d like to test; you could try this with each new iteration of AGI methodology, once you are actionably worried about the possibility of fooming.
A thought occurred to me a while back. Call it the “Ghostbusters” approach to the existential risk of AI research. The basic idea is that rather than trying to make the best FAI on the first try, you hedge your bets. Work to make an AI that is a)unlikely to disrupt human civilization in a permanent way at all, and b)available for study.
Part of the stress of the ‘one big AI’ interpretation of the intelligence explosion is the sense that we’d better get it right the first time. But on the other hand, surely the space of all nonthreatening superintelligences is larger than the space of all helpful ones, and a comparatively easier target to hit on our first shot. You’re still taking a gamble. But minimizing this risk seems much easier when you are not simultaneously trying to change human experience in positive ways. And having performed the action once, there would be a wealth of new information to inform later choices.
So I’m trying to decide if this is obviously true or obviously false: p(being destroyed by a primary FAI attempt) > p(being destroyed by a “Ghostbusters” attempt) * p(being destroyed by a subsequent more informed FAI attempt)
If you’re making AI for study it shouldn’t be super-intelligent at all, ideally it should be dumber than you. I can imagine an AGI that can usefully perform some tasks but is too stupid to self-modify itself into fooming if constrained. You can let it be in charge of opening and closing doors!
Well, I definitely agree that we should make non-super intelligent AIs for study, and also for a great many other reasons. But it’s perhaps less clear what ‘too stupid to foom’ actually means for an AGI. There was a moment when a hominid brain crossed an invisible line and civilization became possible; but the mutation precipitating that change may not have obviously been a major event from the perspective of an outside observer. It may just have looked like another in a sequence of iterative steps. Is the foom line in about the same place as the agriculture line? Is it simpler? Harder?
On the other hand, it’s possible to imagine an experimental AGI with values like “Fulfill [utility function X] in the strictly defined spatial domain of Neptune, using only materials that were contained in the gravity well of Neptune in the year 2000, including the construction of your own brain, and otherwise avoid >epsilon changes to probable outcomes for the universe outside the domain of Neptune.” Then fill in whatever utility function you’d like to test; you could try this with each new iteration of AGI methodology, once you are actionably worried about the possibility of fooming.