Here is the Q+A section: [In the video, the timestamp is 5:42 onward.] [The Transcript is taken from YouTube’s “Show transcript” feature, then cleaned by me for readability. If you think the transcription is functionally erroneous somewhere, let me know.]
Eliezer: Thank you for coming to my brief TED talk.
(Applause)
Host: So, Eliezer, thank you for coming and giving that. It seems like what you’re raising the alarm about is that for an AI to basically destroy humanity, it has to break out, to escape controls of the internet and start commanding real-world resources. You say you can’t predict how that will happen, but just paint one or two possibilities.
Eliezer: Okay. First, why is this hard? Because you can’t predict exactly where a smarter chess program will move. Imagine sending the design for an air conditioner back to the 11th century. Even if there is enough detail for them to build it, they will be surprised when cold air comes out. The air conditioner will use the temperature-pressure relation, and they don’t know about that law of nature. If you want me to sketch what a super intelligence might do, I can go deeper and deeper into places where we think there are predictable technological advancements that we haven’t figured out yet. But as I go deeper and deeper, it gets harder and harder to follow.
It could be super persuasive. We do not understand exactly how the brain works, so it’s a great place to exploit—laws of nature that we do not know about, rules of the environment, new technologies beyond that. Can you build a synthetic virus that gives humans a cold, then a bit of neurological change such that they are easier to persuade? Can you build your own synthetic biology? Synthetic cyborgs? Can you blow straight past that to covalently-bonded equivalence of biology, where instead of proteins that fold up and are held together by static cling, you’ve got things that go down much sharper potential energy gradients and are bonded together? People have done advanced design work about this sort of thing for artificial red blood cells that could hold a hundred times as much oxygen if they were using tiny sapphire vessels to store the oxygen. There’s lots and lots of room above biology, but it gets harder and harder to understand.
Host: So what I hear you saying is you know there are these terrifying possibilities, but your real guess is that AIs will work out something more devious than that. How is that really a likely pathway in your mind?
Eliezer: Which part? That they’re smarter than I am? Absolutely. *Eliezer makes facial expression of stupidity upward, then the audience laughs.
Host: No, not that they’re smarter, but that they would… Why would they want to go in that direction? The AIs don’t have our feelings of envy, jealousy, anger, and so forth. So why might they go in that direction?
Eliezer: Because it is convergently implied by almost any of the strange and scrutable things that they might end up wanting, as a result of gradient descent on these thumbs-up and thumbs-down internal controls. If all you want is to make tiny molecular squiggles, or that’s one component of what you want but it’s a component that never saturates, you just want more and more of it—the same way that we want and would want more and more galaxies filled with life and people living happily ever after. By wanting anything that just keeps going, you are wanting to use more and more material. That could kill everyone on Earth as a side effect. It could kill us because it doesn’t want us making other super intelligences to compete with it. It could kill us because it’s using up all the chemical energy on Earth.
Host: So, some people in the AI world worry that your views are strong enough that you’re willing to advocate extreme responses to it. Therefore, they worry that you could be a very destructive figure. Do you draw the line yourself in terms of the measures that we should take to stop this happening? Or is anything justifiable to stop the scenarios you’re talking about happening?
Eliezer: I don’t think that “anything” works. I think that this takes takes state, actors, and international agreements. All International agreements, by their nature, tend to ultimately be backed by force on the signatory countries and on the non-signatory countries, which is a more extreme measure. I have not proposed that individuals run out and use violence, and I think that the killer argument for that is that it would not work.
Host: Well, you are definitely not the only person to propose that what we need is some kind of international Reckoning here on how to manage this going forward. Thank you so much for coming here to TED.
Here is the Q+A section:
[In the video, the timestamp is 5:42 onward.]
[The Transcript is taken from YouTube’s “Show transcript” feature, then cleaned by me for readability. If you think the transcription is functionally erroneous somewhere, let me know.]
Eliezer: Thank you for coming to my brief TED talk.
(Applause)
Host: So, Eliezer, thank you for coming and giving that. It seems like what you’re raising the alarm about is that for an AI to basically destroy humanity, it has to break out, to escape controls of the internet and start commanding real-world resources. You say you can’t predict how that will happen, but just paint one or two possibilities.
Eliezer: Okay. First, why is this hard? Because you can’t predict exactly where a smarter chess program will move. Imagine sending the design for an air conditioner back to the 11th century. Even if there is enough detail for them to build it, they will be surprised when cold air comes out. The air conditioner will use the temperature-pressure relation, and they don’t know about that law of nature. If you want me to sketch what a super intelligence might do, I can go deeper and deeper into places where we think there are predictable technological advancements that we haven’t figured out yet. But as I go deeper and deeper, it gets harder and harder to follow.
It could be super persuasive. We do not understand exactly how the brain works, so it’s a great place to exploit—laws of nature that we do not know about, rules of the environment, new technologies beyond that. Can you build a synthetic virus that gives humans a cold, then a bit of neurological change such that they are easier to persuade? Can you build your own synthetic biology? Synthetic cyborgs? Can you blow straight past that to covalently-bonded equivalence of biology, where instead of proteins that fold up and are held together by static cling, you’ve got things that go down much sharper potential energy gradients and are bonded together? People have done advanced design work about this sort of thing for artificial red blood cells that could hold a hundred times as much oxygen if they were using tiny sapphire vessels to store the oxygen. There’s lots and lots of room above biology, but it gets harder and harder to understand.
Host: So what I hear you saying is you know there are these terrifying possibilities, but your real guess is that AIs will work out something more devious than that. How is that really a likely pathway in your mind?
Eliezer: Which part? That they’re smarter than I am? Absolutely. *Eliezer makes facial expression of stupidity upward, then the audience laughs.
Host: No, not that they’re smarter, but that they would… Why would they want to go in that direction? The AIs don’t have our feelings of envy, jealousy, anger, and so forth. So why might they go in that direction?
Eliezer: Because it is convergently implied by almost any of the strange and scrutable things that they might end up wanting, as a result of gradient descent on these thumbs-up and thumbs-down internal controls. If all you want is to make tiny molecular squiggles, or that’s one component of what you want but it’s a component that never saturates, you just want more and more of it—the same way that we want and would want more and more galaxies filled with life and people living happily ever after. By wanting anything that just keeps going, you are wanting to use more and more material. That could kill everyone on Earth as a side effect. It could kill us because it doesn’t want us making other super intelligences to compete with it. It could kill us because it’s using up all the chemical energy on Earth.
Host: So, some people in the AI world worry that your views are strong enough that you’re willing to advocate extreme responses to it. Therefore, they worry that you could be a very destructive figure. Do you draw the line yourself in terms of the measures that we should take to stop this happening? Or is anything justifiable to stop the scenarios you’re talking about happening?
Eliezer: I don’t think that “anything” works. I think that this takes takes state, actors, and international agreements. All International agreements, by their nature, tend to ultimately be backed by force on the signatory countries and on the non-signatory countries, which is a more extreme measure. I have not proposed that individuals run out and use violence, and I think that the killer argument for that is that it would not work.
Host: Well, you are definitely not the only person to propose that what we need is some kind of international Reckoning here on how to manage this going forward. Thank you so much for coming here to TED.