I don’t really know the SIAI people, but I have the impression that they’re not against AI at all. Sure, an unfriendly AI would be awful—but a friendly one would be awesome. And they probably think AI is inevitable, anyway.
This is true as far as it goes; however if you actually visit SIAI, you may find significantly more worry about UFAI in the short term than you would have expected just from reading Eliezer Yudkowsky’s writings.
I think that you interacted most with a pretty uncharacteristically biased sample of characters: most of the long-term SIAI folk have longer timelines than good ol’ me and Justin by about 15-20 years. That said, it’s true that everyone is still pretty worried about AI-soon, no matter the probability.
Well, 15-20 years doesn’t strike me as that much of a time difference, actually. But in any case I was really talking about my surprise at the amount of emphasis on “preventing UFAI” as opposed to “creating FAI”. Do you suppose that’s also reflective of a biased sample?
Well, 15-20 years doesn’t strike me as that much of a time difference, actually.
Really? I mean, relative to your estimate it might not be big, but absolutely speaking, doom 15 years versus doom 35 years seems to make a huge difference in expected utility.
Do you suppose that’s also reflective of a biased sample?
Probably insofar as Eliezer and Marcello weren’t around: FAI and the Visiting Fellows intersect at decision theory only. But the more direct (and potentially dangerous) AGI stuff isn’t openly discussed for obvious reasons.
relative to your estimate it might not be big, but absolutely speaking, doom 15 years versus doom 35 years seems to make a huge difference in expected utility.
A good point. By the way, I should mention that I updated my estimate after it was pointed out to me that other folks’ estimates were taking Outside View considerations into account, and after I learned I had been overestimating the information-theoretic complexity of existing minds. FOOM before 2100 looks significantly more likely to me now than it did before.
Probably insofar as Eliezer and Marcello weren’t around: FAI and the Visiting Fellows intersect at decision theory only.
Well I didn’t expect that AGI technicalities would be discussed openly, of course. What I’m thinking of is Eliezer’s attitude that (for now) AGI is unlikely to be developed by anyone not competent enough to realize Friendliness is a problem, versus the apparent fear among some other people that AGI might be cobbled together more or less haphazardly, even in the near term.
Eliezer’s attitude that (for now) AGI is unlikely to be developed by anyone not competent enough to realize Friendliness is a problem
Huh. I didn’t get that from the sequences, perhaps I should check again. It always seemed to me as if he saw AGI as really frickin’ hard but not excessively so, whereas Friendliness is the Impossible Problem made up of smaller but also impossible problems.
I don’t really know the SIAI people, but I have the impression that they’re not against AI at all. Sure, an unfriendly AI would be awful—but a friendly one would be awesome.
True. I know the SIAI people pretty well (I’m kind of one of them) and can confirm they agree. But they’re pretty heavily against uFAI development, which is what I thought XiXiDu’s quote was talking about.
And they probably think AI is inevitable, anyway.
Well… hopefully not, in a sense. SIAI’s working to improve widespread knowledge of the need for Friendliness among AGI researchers. It’s inevitable (barring a global catastrophe), but they’re hoping to make FAI more inevitable than uFAI.
As someone who was a volunteered for SIAI at the Singularity Summit, a critique of SIAI could be to ask why we’re letting people who aren’t concerned about uFAI speak at our conferences and affiliate with our memes. I think there are good answers to that critique, but the critique itself is a pretty reasonable one. Most complains about SIAI are comparatively maddeningly irrational (in my own estimation).
A stronger criticism, I think, is why the only mention of friendliness at the Summit was some very veiled hints in Eliezer’s speech. Again, I think there are good reasons, but not good reasons that a lot of people know, so I don’t understand why people bring up other criticisms before this one.
I don’t really know the SIAI people, but I have the impression that they’re not against AI at all. Sure, an unfriendly AI would be awful—but a friendly one would be awesome. And they probably think AI is inevitable, anyway.
This is true as far as it goes; however if you actually visit SIAI, you may find significantly more worry about UFAI in the short term than you would have expected just from reading Eliezer Yudkowsky’s writings.
I think that you interacted most with a pretty uncharacteristically biased sample of characters: most of the long-term SIAI folk have longer timelines than good ol’ me and Justin by about 15-20 years. That said, it’s true that everyone is still pretty worried about AI-soon, no matter the probability.
Well, 15-20 years doesn’t strike me as that much of a time difference, actually. But in any case I was really talking about my surprise at the amount of emphasis on “preventing UFAI” as opposed to “creating FAI”. Do you suppose that’s also reflective of a biased sample?
Really? I mean, relative to your estimate it might not be big, but absolutely speaking, doom 15 years versus doom 35 years seems to make a huge difference in expected utility.
Probably insofar as Eliezer and Marcello weren’t around: FAI and the Visiting Fellows intersect at decision theory only. But the more direct (and potentially dangerous) AGI stuff isn’t openly discussed for obvious reasons.
A good point. By the way, I should mention that I updated my estimate after it was pointed out to me that other folks’ estimates were taking Outside View considerations into account, and after I learned I had been overestimating the information-theoretic complexity of existing minds. FOOM before 2100 looks significantly more likely to me now than it did before.
Well I didn’t expect that AGI technicalities would be discussed openly, of course. What I’m thinking of is Eliezer’s attitude that (for now) AGI is unlikely to be developed by anyone not competent enough to realize Friendliness is a problem, versus the apparent fear among some other people that AGI might be cobbled together more or less haphazardly, even in the near term.
Huh. I didn’t get that from the sequences, perhaps I should check again. It always seemed to me as if he saw AGI as really frickin’ hard but not excessively so, whereas Friendliness is the Impossible Problem made up of smaller but also impossible problems.
True. I know the SIAI people pretty well (I’m kind of one of them) and can confirm they agree. But they’re pretty heavily against uFAI development, which is what I thought XiXiDu’s quote was talking about.
Well… hopefully not, in a sense. SIAI’s working to improve widespread knowledge of the need for Friendliness among AGI researchers. It’s inevitable (barring a global catastrophe), but they’re hoping to make FAI more inevitable than uFAI.
As someone who was a volunteered for SIAI at the Singularity Summit, a critique of SIAI could be to ask why we’re letting people who aren’t concerned about uFAI speak at our conferences and affiliate with our memes. I think there are good answers to that critique, but the critique itself is a pretty reasonable one. Most complains about SIAI are comparatively maddeningly irrational (in my own estimation).
A stronger criticism, I think, is why the only mention of friendliness at the Summit was some very veiled hints in Eliezer’s speech. Again, I think there are good reasons, but not good reasons that a lot of people know, so I don’t understand why people bring up other criticisms before this one.