“A rationalist!hero should excel by thinking—moreover, thinking in understandable patterns that readers can, in principle, adopt for themselves.”—Eliezer
I think Harry is supposed to be a rationalist!hero. I think he’s supposed to succeed in the end. The fact that he also has the potential to be very dangerous and destructive is, I think, just about the dangers of power itself, and the importance of ethics + reason in making decisions. Especially when you have something to protect.
I think Harry’s ethics are actually not too far from EY’s. Of course he has ethical failures and character flaws, and that pesky dark side, but he’s mostly headed in the right direction. And though he’s quite unusual, he doesn’t have some sort of inhuman psychology. A protagonist’s quirks are often exaggerated, because stories have to shout.
I think Harry’s ethics are actually not too far from EY’s
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
(That is, both Harry and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and becoming a new Dark Lord. But they have a desire to make things better as well—well, Harry more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!)
That is, both Eliezer and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and the creation of a UFAI. But they have a desire to make things better as well—well, Eliezer more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!
This is why I stringently oppose all forms of AI research.
If you disagree with that last sentence, which part of the analogy do you think is flawed?
The bit where Eliezer isn’t a character in a novel.
Sure UFAI pattern-matches to the Dune jihad better than anything in HPMoR, but even then Eliezer isn’t acting like Paul at all. A Muad-dib!Eliezer would be working feverishly in the basement of the fortified SIAI bunker (which would actually be a cult) to put together an “at-least-it-isn’t-paperclips-forever” AI before the US government finds his location and drops a bunker buster on him. Right about now we would have tense negotiations with national entities grimly ending in Eliezer pressing the On switch.
This sounds nothing like what is actually happening, does it?
Well, no. (Hopefully not. Not as far as I know, anyway.)
The point I was making was less about the similarities to Dune (which I shamefully have only a vague knowledge of) and more about this assertion:
I think Harry’s ethics are actually not too far from EY’s
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
Cause see, the thing is, Harry isn’t working feverishly in a bunker, either. I guess the question is: what do you think Eliezer would do differently in Harry’s situation?
Edit: I’ve just realized I anchored onto the wrong part of that analogy. It may actually be fair to describe Harry’s recent approach as leaning towards “tense negotiations with national entities grimly ending in … pressing the On switch.” Um. So.
I’m not going to retract, but I think I’ve got more of an appreciation for your point of view, now.
For the life of me, I can’t find it—but there is a comment from EY about HJPEV’s actions (appearing suboptimal in some commentor’s view because he was busy winning a school wargames competition instead of Transfiguring more carbon nanotubes and building supercomputers and space elevators) to the sarcastic effect of “It’s almost as if he’s not me!”.
This is what prompted me to compare Eliezer’s HJPEV to another author’s “supposed-avatar-but-actually-the-author-is-smarter-than-that” character.
“A rationalist!hero should excel by thinking—moreover, thinking in understandable patterns that readers can, in principle, adopt for themselves.”—Eliezer
I think Harry is supposed to be a rationalist!hero. I think he’s supposed to succeed in the end. The fact that he also has the potential to be very dangerous and destructive is, I think, just about the dangers of power itself, and the importance of ethics + reason in making decisions. Especially when you have something to protect.
I think Harry’s ethics are actually not too far from EY’s. Of course he has ethical failures and character flaws, and that pesky dark side, but he’s mostly headed in the right direction. And though he’s quite unusual, he doesn’t have some sort of inhuman psychology. A protagonist’s quirks are often exaggerated, because stories have to shout.
I don’t think he represents an AI at all.
Harry James Potter-Evans-Verres’ morality is to Eliezer Yudkowsky’s morality as Paul Muad-dib Atreides’ is to Frank Herbert’s.
(That is, both Harry and Paul are heroes, and have hero morality. They have their view of a horrifying future they may cause: the jihad, and becoming a new Dark Lord. But they have a desire to make things better as well—well, Harry more so than Paul. And in reading their story, you are supposed to fall into the same trap they do—thinking that if they just try hard enough, they can still do good, still have an impact, still be a hero, without bringing about the horrible future. And so they aim closer and closer to that horrible future, looking for the edge of the circle where the bad thing doesn’t happen but they are still a hero, with a hero’s legacy—when what they should be doing is making the bad future as unlikely as possible, not making the bad future only just barely fail to happen!)
If you disagree with that last sentence, which part of the analogy do you think is flawed?
The bit where Eliezer isn’t a character in a novel.
Sure UFAI pattern-matches to the Dune jihad better than anything in HPMoR, but even then Eliezer isn’t acting like Paul at all. A Muad-dib!Eliezer would be working feverishly in the basement of the fortified SIAI bunker (which would actually be a cult) to put together an “at-least-it-isn’t-paperclips-forever” AI before the US government finds his location and drops a bunker buster on him. Right about now we would have tense negotiations with national entities grimly ending in Eliezer pressing the On switch.
This sounds nothing like what is actually happening, does it?
Well, no. (Hopefully not. Not as far as I know, anyway.)
The point I was making was less about the similarities to Dune (which I shamefully have only a vague knowledge of) and more about this assertion:
Cause see, the thing is, Harry isn’t working feverishly in a bunker, either. I guess the question is: what do you think Eliezer would do differently in Harry’s situation?
Edit: I’ve just realized I anchored onto the wrong part of that analogy. It may actually be fair to describe Harry’s recent approach as leaning towards “tense negotiations with national entities grimly ending in … pressing the On switch.” Um. So.
I’m not going to retract, but I think I’ve got more of an appreciation for your point of view, now.
For the life of me, I can’t find it—but there is a comment from EY about HJPEV’s actions (appearing suboptimal in some commentor’s view because he was busy winning a school wargames competition instead of Transfiguring more carbon nanotubes and building supercomputers and space elevators) to the sarcastic effect of “It’s almost as if he’s not me!”.
This is what prompted me to compare Eliezer’s HJPEV to another author’s “supposed-avatar-but-actually-the-author-is-smarter-than-that” character.