I did listen to that post, and while I don’t remember all the points, I do remember that it didn’t convince me that alignment is easy and, like Christiano’s post “Where I agree and disagree with Eliezer”, it just seems to be like “p(doom) of 95%+ plus is too much, it’s probably something like 10-50%” which is still incredibly unacceptably high to continue “business as usual”. I have faith that something will be done: regulation and breakthrough will happen, but it seems likely that it won’t be enough.
It comes down to safety mindset. There are very few and sketchy reasons to expect that by default an ASI will care about humans enough, so it not safe to build one until shown otherwise (preferably without actually creating one). And if I had to point out a single cause for my own high p(doom), it is the fact that we humans iterate all of our engineering to iron out all of the kinks, while with a technology that is itself adversarial, iteration might not be available (get it right the first time we deploy powerful AI).
Who do you think are the two or three smartest people to be skeptical of AI killing all humans? I think maybe Yann LeCunn and Andrew Ng.
Sure, those two. I don’t know about Ng (he recently had a private discussion with Hinton, but I don’t know what he thinks now), but I know LeCun hasn’t really engaged with the ideas and just relies on the concept that “it’s an extreme idea”. But as I said, having the position “AI doesn’t pose an existential threat” seems to be fringe nowadays.
If I dumb the argument down enough I get stuff like “intelligence/cognition/optimization is dangerous, and, whatever the reasons, we currently have zero reliable ideas on how to make a powerful general intelligence safe (eg. RLHF doesn’t work well enough as GPT-4 still lies/hallucinates and is jailbroken way too easily)” which is evidence based, not weird and not extreme.
Consider also reading Scott Aaronson (whose sabbatical at OpenAI is about to end):
https://scottaaronson.blog/?p=7266
https://scottaaronson.blog/?p=7230
https://scottaaronson.blog/?p=7174
https://scottaaronson.blog/?p=7064
I did listen to that post, and while I don’t remember all the points, I do remember that it didn’t convince me that alignment is easy and, like Christiano’s post “Where I agree and disagree with Eliezer”, it just seems to be like “p(doom) of 95%+ plus is too much, it’s probably something like 10-50%” which is still incredibly unacceptably high to continue “business as usual”. I have faith that something will be done: regulation and breakthrough will happen, but it seems likely that it won’t be enough.
It comes down to safety mindset. There are very few and sketchy reasons to expect that by default an ASI will care about humans enough, so it not safe to build one until shown otherwise (preferably without actually creating one). And if I had to point out a single cause for my own high p(doom), it is the fact that we humans iterate all of our engineering to iron out all of the kinks, while with a technology that is itself adversarial, iteration might not be available (get it right the first time we deploy powerful AI).
Sure, those two. I don’t know about Ng (he recently had a private discussion with Hinton, but I don’t know what he thinks now), but I know LeCun hasn’t really engaged with the ideas and just relies on the concept that “it’s an extreme idea”. But as I said, having the position “AI doesn’t pose an existential threat” seems to be fringe nowadays.
If I dumb the argument down enough I get stuff like “intelligence/cognition/optimization is dangerous, and, whatever the reasons, we currently have zero reliable ideas on how to make a powerful general intelligence safe (eg. RLHF doesn’t work well enough as GPT-4 still lies/hallucinates and is jailbroken way too easily)” which is evidence based, not weird and not extreme.