There seems to be only one way this story can end.
Eliezer is concerned with people building AIs that have tremendous power.
The powerful wizards in HPMoR represent the AIs.
Harry wants, and appears destined, to become the singleton AI.
Eliezer believes that any AI that is not provably constrained, by its design, to follow rules that lead to , will inevitably—and I use that word to mean “with such high probability that you shouldn’t even think about the cases where it doesn’t”—lead to violating those rules.
Harry cannot be redesigned from the ground up to provably implement any ethical system.
Therefore, for Harry to do anything other than destroy everything that we, the readers, hold dear, would go against everything Eliezer has written about AI.
(It could alternately end with Harry killing himself and Voldemort, neutralizing Voldemort and giving up his own power, letting Voldemort kill him on realizing that Voldemort is less of a threat than he himself is, or in some other way kicking the problem down the road for the next generation. But the only final resolution of the story’s central theme that is consistent with Eliezer’s writings seems to be for Harry to destroy the world.)
No he doesn’t. He’s just an AI creator. If he was representing an AI the entire story would have been over at Chapter 1.
Eliezer would not write a parable in which an AI made a slow journey to power, on the way learning valuable moral lessons through social interaction with the people he cares about. That completely undermines all that he stands for.
Therefore, for Harry to do anything other than destroy everything that we, the readers, hold dear, would go against everything Eliezer has written about AI.
Harry destroying the world would make a reasonable conclusion albeit for different reasons than Harry being an AI. That said, the lesson to be learned is too complicated for fanfic readers to be expected to learn from just a story. It is hard to convey that Harry made the correct decision to try to save the world even though he failed and destroyed everything because the alternative was certain failure and at least his chances were better than that.
Which reminds me: I hope there is a chapter in which Harry (or someone else) loses something dramatically but realizes (or has Harry explain to them) that they made the right choice anyway. And the converse in which a decision is wrong even though it turned out well.
Which reminds me: I hope there is a chapter in which Harry (or someone else) loses something dramatically but realizes (or has Harry explain to them) that they made the right choice anyway. And the converse in which a decision is wrong even though it turned out well.
Harry is not an AI creator. That would require him to create an AI. There is no AI in the story, and no hints that there will be any AI. Harry is on his way to achieving personal power, without any technological intermediator.
Eliezer would not write a parable in which an AI made a slow journey to power, on the way learning valuable moral lessons through social interaction with the people he cares about. That completely undermines all that he stands for.
Why? Do you think Eliezer believes AIs are incapable of learning valuable moral lessons through social interaction with the people it cares about? Presumably, even within Eliezer’s model, an AI can be moral and kind and loving to humans for some time period. If you think that “unfriendly AI” will necessarily be amoral and incapable of love—well, that’s just silly. The reasons for believing that are exactly the same as the reasons for believing that computers can never be intelligent.
As to “slow”, that’s just one of the things that scales differently because of the analogy being used.
Harry is not an AI creator. That would require him to create an AI.
What the? I’m downgrading from your “Harry represents an AI” to “Harry represents an AI creator” and now you want to get technical and say “there is no AI in the story”?
Why? Do you think Eliezer believes AIs are incapable of learning valuable moral lessons through social interaction with the people it cares about? Presumably, even within Eliezer’s model, an AI can be moral and kind and loving to humans for some time period. If you think that “unfriendly AI” will necessarily be amoral and incapable of love—well, that’s just silly.
I believe Eliezer would rather dip his… arm… in a vat of acid than suggest to his readers that an AI will act like a clever well intentioned child that learns valuable moral lessons through social interactions with those he loves, over a long period of a slow, rather inefficient quest for power.
I believe Eliezer would rather dip his… arm… in a vat of acid than suggest to his readers that an AI will act like a clever well intentioned child that learns valuable moral lessons through social interactions with those he loves, over a long period of a slow, rather inefficient quest for power.
Please cool it down a bit.
Just two chapters ago, Harry was compared to a all-powerful summoned entity that knew nothing about how humans worked and so had to be taught not to eat people and stuff. Hermione accepted that comparison as somewhat fitting. So the comparison of Harry to non-human intelligences is MoR!Canonical, even though it was a huge exaggeration even within the context of the story, expressed by people genuinely scared of Harry—the Harry who seemed more powerful than a Nameless Horror and might need the magical nations of the world to ally against him in order to seal off his incursion into our reality.
My hyperbole is going of like crazy over here. You should probably cease this conversation before it becomes even more heated. Both of you.
I don’t think you get it. If you think you can substitute “write something slightly at odds with his beliefs into a fanfiction” even remotely appropriately into that context then you do not have enough knowledge about Eliezer’s publicly expressed goals to be making the kind of moral judgement you presume to make.
I wrote my claim because it sounds like a hyperbole but isn’t. For the right value of ‘acid’ my claim is a literal counterfactual prediction. And I don’t mean a “dipping his arm in vinegar” technicality. I’d estimate the approximate value of acid required to make the preference stand is something that would fall short of leaving permanent scars but give him significant pain for a week. Maybe that is a little light… perhaps leaving some scarring but leaving full function and sensitivity on the hand so he can still type. Just so long as some of that typing constitutes a massive disclaimer and retraction to undo some of the damage his toxic propaganda would have done.
I said Harry represents an AI. You said Harry is an AI creator. I was just taking you at your word. It didn’t occur to me that you meant Harry represents an AI creator, because that wouldn’t make any more sense. Harry does not represent an AI creator unless he creates something that is in some way analogous to an AI. There is no hint he will do that. Harry is not creating an AI, and is not creating anything analogous to an AI.
Whereas Harry has properties analogous to a recursively self-improving AI on the way to having godlike power. And the danger that Harry will pose once he attains these godlike powers is one of the central themes of the fanfic, and is highlighted throughout the entire fanfic, over and over again.
I believe Eliezer would rather dip his… arm… in a vat of acid than suggest to his readers that an AI will act like a clever well intentioned child that learns valuable moral lessons through social interactions with those he loves, over a long period of a slow, rather inefficient quest for power.
The slowness is due to humans, who represent the AIs, operating only at human speeds, and does not detract from my point.
I still have the impression that you think Eliezer’s view implies that an AI cannot be well-intentioned, or learn valuable moral lessons through social interactions with those he loves. Why? That is exactly as absurd as saying that computers can never be intelligent. (I am ignoring questions of whether a computer program can subjective experience love, since “love” used as a verb refers to the action rather than to the subjective experience. But, still, anyone who follows Eliezer at all should be a materialist and therefore believe that a computer program can in principle love as well as a person can.)
I said Harry represents an AI. You said Harry is an AI creator. I was just taking you at your word.
Your interpretation is technically grammatically valid but not remotely reasonable.
I still have the impression that you think Eliezer’s view implies that an AI cannot be well-intentioned, or learn valuable moral lessons through social interactions with those he loves. Why? That is exactly as absurd as saying that computers can never be intelligent.
This impression, however, is not even a technically valid interpretation.
I’m not convinced of your second point. Powerful MoR wizards aren’t nearly as constrained as people in our world, but the only person in the story that’s explicitly trying to act as a optimizer over a generalized utility function rather than to achieve some set of concrete goals—that is, the only person acting as Eliezer’s concept of AI would—is Harry himself. There are a number of ways that plot thread could be resolved, but destroying the world as we know it isn’t the one I’d bet on—although I’d expect the threat of such destruction to come into play at some point down the road.
We’ve also seen Eliezer’s take on the Dark Lord-as-optimizer concept play itself out once before, in “The Sword of Good”, and that—avoiding spoilers—didn’t seem to resolve in a way consistent with your premise.
“A rationalist!hero should excel by thinking—moreover, thinking in understandable patterns that readers can, in principle, adopt for themselves.”—Eliezer
I think Harry is supposed to be a rationalist!hero. I think he’s supposed to succeed in the end. The fact that he also has the potential to be very dangerous and destructive is, I think, just about the dangers of power itself, and the importance of ethics + reason in making decisions. Especially when you have something to protect.
I think Harry’s ethics are actually not too far from EY’s. Of course he has ethical failures and character flaws, and that pesky dark side, but he’s mostly headed in the right direction. And though he’s quite unusual, he doesn’t have some sort of inhuman psychology. A protagonist’s quirks are often exaggerated, because stories have to shout.
I think Harry’s ethics are actually not too far from EY’s
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
(That is, both Harry and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and becoming a new Dark Lord. But they have a desire to make things better as well—well, Harry more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!)
That is, both Eliezer and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and the creation of a UFAI. But they have a desire to make things better as well—well, Eliezer more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!
This is why I stringently oppose all forms of AI research.
If you disagree with that last sentence, which part of the analogy do you think is flawed?
The bit where Eliezer isn’t a character in a novel.
Sure UFAI pattern-matches to the Dune jihad better than anything in HPMoR, but even then Eliezer isn’t acting like Paul at all. A Muad-dib!Eliezer would be working feverishly in the basement of the fortified SIAI bunker (which would actually be a cult) to put together an “at-least-it-isn’t-paperclips-forever” AI before the US government finds his location and drops a bunker buster on him. Right about now we would have tense negotiations with national entities grimly ending in Eliezer pressing the On switch.
This sounds nothing like what is actually happening, does it?
Well, no. (Hopefully not. Not as far as I know, anyway.)
The point I was making was less about the similarities to Dune (which I shamefully have only a vague knowledge of) and more about this assertion:
I think Harry’s ethics are actually not too far from EY’s
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
Cause see, the thing is, Harry isn’t working feverishly in a bunker, either. I guess the question is: what do you think Eliezer would do differently in Harry’s situation?
Edit: I’ve just realized I anchored onto the wrong part of that analogy. It may actually be fair to describe Harry’s recent approach as leaning towards “tense negotiations with national entities grimly ending in … pressing the On switch.” Um. So.
I’m not going to retract, but I think I’ve got more of an appreciation for your point of view, now.
For the life of me, I can’t find it—but there is a comment from EY about HJPEV’s actions (appearing suboptimal in some commentor’s view because he was busy winning a school wargames competition instead of Transfiguring more carbon nanotubes and building supercomputers and space elevators) to the sarcastic effect of “It’s almost as if he’s not me!”.
This is what prompted me to compare Eliezer’s HJPEV to another author’s “supposed-avatar-but-actually-the-author-is-smarter-than-that” character.
I feel like a fitting conclusion would be for Voldemort’s last remnant to end up on the Voyager spacecraft, so that he is forever in the stars, away from earth.
I think the Sword of Good focused on the question of how culture can blind our moral sensibility. I would say that story just wasn’t about that problem.
It is true that neither the Lord of Dark in that story, nor Harry in MoR, is expected to enter into the rapid recursive self-improvement anticipated for an AI. Though I would expect MoR’s Harry to figure out a way to do that, given wizardly powers over time.
There seems to be only one way this story can end.
Eliezer is concerned with people building AIs that have tremendous power.
The powerful wizards in HPMoR represent the AIs.
Harry wants, and appears destined, to become the singleton AI.
Eliezer believes that any AI that is not provably constrained, by its design, to follow rules that lead to , will inevitably—and I use that word to mean “with such high probability that you shouldn’t even think about the cases where it doesn’t”—lead to violating those rules.
Harry cannot be redesigned from the ground up to provably implement any ethical system.
Therefore, for Harry to do anything other than destroy everything that we, the readers, hold dear, would go against everything Eliezer has written about AI.
(It could alternately end with Harry killing himself and Voldemort, neutralizing Voldemort and giving up his own power, letting Voldemort kill him on realizing that Voldemort is less of a threat than he himself is, or in some other way kicking the problem down the road for the next generation. But the only final resolution of the story’s central theme that is consistent with Eliezer’s writings seems to be for Harry to destroy the world.)
Whoah, whoah. Wait. Back up.
What the heck?
If not all of them, at least Harry does.
No he doesn’t. He’s just an AI creator. If he was representing an AI the entire story would have been over at Chapter 1.
Eliezer would not write a parable in which an AI made a slow journey to power, on the way learning valuable moral lessons through social interaction with the people he cares about. That completely undermines all that he stands for.
Harry destroying the world would make a reasonable conclusion albeit for different reasons than Harry being an AI. That said, the lesson to be learned is too complicated for fanfic readers to be expected to learn from just a story. It is hard to convey that Harry made the correct decision to try to save the world even though he failed and destroyed everything because the alternative was certain failure and at least his chances were better than that.
Which reminds me: I hope there is a chapter in which Harry (or someone else) loses something dramatically but realizes (or has Harry explain to them) that they made the right choice anyway. And the converse in which a decision is wrong even though it turned out well.
Agreed.
Harry is not an AI creator. That would require him to create an AI. There is no AI in the story, and no hints that there will be any AI. Harry is on his way to achieving personal power, without any technological intermediator.
Why? Do you think Eliezer believes AIs are incapable of learning valuable moral lessons through social interaction with the people it cares about? Presumably, even within Eliezer’s model, an AI can be moral and kind and loving to humans for some time period. If you think that “unfriendly AI” will necessarily be amoral and incapable of love—well, that’s just silly. The reasons for believing that are exactly the same as the reasons for believing that computers can never be intelligent.
As to “slow”, that’s just one of the things that scales differently because of the analogy being used.
What the? I’m downgrading from your “Harry represents an AI” to “Harry represents an AI creator” and now you want to get technical and say “there is no AI in the story”?
I believe Eliezer would rather dip his… arm… in a vat of acid than suggest to his readers that an AI will act like a clever well intentioned child that learns valuable moral lessons through social interactions with those he loves, over a long period of a slow, rather inefficient quest for power.
Please cool it down a bit.
Just two chapters ago, Harry was compared to a all-powerful summoned entity that knew nothing about how humans worked and so had to be taught not to eat people and stuff. Hermione accepted that comparison as somewhat fitting. So the comparison of Harry to non-human intelligences is MoR!Canonical, even though it was a huge exaggeration even within the context of the story, expressed by people genuinely scared of Harry—the Harry who seemed more powerful than a Nameless Horror and might need the magical nations of the world to ally against him in order to seal off his incursion into our reality.
My hyperbole is going of like crazy over here. You should probably cease this conversation before it becomes even more heated. Both of you.
I don’t think you get it. If you think you can substitute “write something slightly at odds with his beliefs into a fanfiction” even remotely appropriately into that context then you do not have enough knowledge about Eliezer’s publicly expressed goals to be making the kind of moral judgement you presume to make.
I wrote my claim because it sounds like a hyperbole but isn’t. For the right value of ‘acid’ my claim is a literal counterfactual prediction. And I don’t mean a “dipping his arm in vinegar” technicality. I’d estimate the approximate value of acid required to make the preference stand is something that would fall short of leaving permanent scars but give him significant pain for a week. Maybe that is a little light… perhaps leaving some scarring but leaving full function and sensitivity on the hand so he can still type. Just so long as some of that typing constitutes a massive disclaimer and retraction to undo some of the damage his toxic propaganda would have done.
I said Harry represents an AI. You said Harry is an AI creator. I was just taking you at your word. It didn’t occur to me that you meant Harry represents an AI creator, because that wouldn’t make any more sense. Harry does not represent an AI creator unless he creates something that is in some way analogous to an AI. There is no hint he will do that. Harry is not creating an AI, and is not creating anything analogous to an AI.
Whereas Harry has properties analogous to a recursively self-improving AI on the way to having godlike power. And the danger that Harry will pose once he attains these godlike powers is one of the central themes of the fanfic, and is highlighted throughout the entire fanfic, over and over again.
The slowness is due to humans, who represent the AIs, operating only at human speeds, and does not detract from my point.
I still have the impression that you think Eliezer’s view implies that an AI cannot be well-intentioned, or learn valuable moral lessons through social interactions with those he loves. Why? That is exactly as absurd as saying that computers can never be intelligent. (I am ignoring questions of whether a computer program can subjective experience love, since “love” used as a verb refers to the action rather than to the subjective experience. But, still, anyone who follows Eliezer at all should be a materialist and therefore believe that a computer program can in principle love as well as a person can.)
Your interpretation is technically grammatically valid but not remotely reasonable.
This impression, however, is not even a technically valid interpretation.
I’m not convinced of your second point. Powerful MoR wizards aren’t nearly as constrained as people in our world, but the only person in the story that’s explicitly trying to act as a optimizer over a generalized utility function rather than to achieve some set of concrete goals—that is, the only person acting as Eliezer’s concept of AI would—is Harry himself. There are a number of ways that plot thread could be resolved, but destroying the world as we know it isn’t the one I’d bet on—although I’d expect the threat of such destruction to come into play at some point down the road.
We’ve also seen Eliezer’s take on the Dark Lord-as-optimizer concept play itself out once before, in “The Sword of Good”, and that—avoiding spoilers—didn’t seem to resolve in a way consistent with your premise.
“A rationalist!hero should excel by thinking—moreover, thinking in understandable patterns that readers can, in principle, adopt for themselves.”—Eliezer
I think Harry is supposed to be a rationalist!hero. I think he’s supposed to succeed in the end. The fact that he also has the potential to be very dangerous and destructive is, I think, just about the dangers of power itself, and the importance of ethics + reason in making decisions. Especially when you have something to protect.
I think Harry’s ethics are actually not too far from EY’s. Of course he has ethical failures and character flaws, and that pesky dark side, but he’s mostly headed in the right direction. And though he’s quite unusual, he doesn’t have some sort of inhuman psychology. A protagonist’s quirks are often exaggerated, because stories have to shout.
I don’t think he represents an AI at all.
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
(That is, both Harry and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and becoming a new Dark Lord. But they have a desire to make things better as well—well, Harry more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!)
If you disagree with that last sentence, which part of the analogy do you think is flawed?
The bit where Eliezer isn’t a character in a novel.
Sure UFAI pattern-matches to the Dune jihad better than anything in HPMoR, but even then Eliezer isn’t acting like Paul at all. A Muad-dib!Eliezer would be working feverishly in the basement of the fortified SIAI bunker (which would actually be a cult) to put together an “at-least-it-isn’t-paperclips-forever” AI before the US government finds his location and drops a bunker buster on him. Right about now we would have tense negotiations with national entities grimly ending in Eliezer pressing the On switch.
This sounds nothing like what is actually happening, does it?
Well, no. (Hopefully not. Not as far as I know, anyway.)
The point I was making was less about the similarities to Dune (which I shamefully have only a vague knowledge of) and more about this assertion:
Cause see, the thing is, Harry isn’t working feverishly in a bunker, either. I guess the question is: what do you think Eliezer would do differently in Harry’s situation?
Edit: I’ve just realized I anchored onto the wrong part of that analogy. It may actually be fair to describe Harry’s recent approach as leaning towards “tense negotiations with national entities grimly ending in … pressing the On switch.” Um. So.
I’m not going to retract, but I think I’ve got more of an appreciation for your point of view, now.
For the life of me, I can’t find it—but there is a comment from EY about HJPEV’s actions (appearing suboptimal in some commentor’s view because he was busy winning a school wargames competition instead of Transfiguring more carbon nanotubes and building supercomputers and space elevators) to the sarcastic effect of “It’s almost as if he’s not me!”.
This is what prompted me to compare Eliezer’s HJPEV to another author’s “supposed-avatar-but-actually-the-author-is-smarter-than-that” character.
I feel like a fitting conclusion would be for Voldemort’s last remnant to end up on the Voyager spacecraft, so that he is forever in the stars, away from earth.
No. Really?
(Pioneer spacecraft. “V’olde’Ger” is a joke based on a Star Trek movie.)
Edit: Raargh beaten by thirty seconds.
Pioneer, not Voyager.
How do you reconcile this view with the Sword of Good story?
I think the Sword of Good focused on the question of how culture can blind our moral sensibility. I would say that story just wasn’t about that problem.
It is true that neither the Lord of Dark in that story, nor Harry in MoR, is expected to enter into the rapid recursive self-improvement anticipated for an AI. Though I would expect MoR’s Harry to figure out a way to do that, given wizardly powers over time.
I’m confused. Didn’t The Sword of Good end with the destruction of the world, just as PhilGoetz would predict?
No.