There’s a reason it’s called heroic responsibility: it’s for a fictional hero, who can do Fictional Hero Things like upset the world order on a regular basis and get away with it. He has Plot Armor, and an innately limited world.
But it’s my understanding that HPMOR was meant to teach about real-world reasoning.
Is this really supposed to be one of the HPMOR passages which is solely about the fictional character and is not meant to have any application to the real world except as an example of something not to do? It certainly doesn’t sound like that.
No, it’s pretty clear that the author intends this to be a real-world lesson. It’s a recurring theme in the Sequences.
I think Eli was disagreeing with the naive application of that lesson to real-world situations, especially ones where established systems are functional.
That said, I don’t want to put words in Eli’s mouth, so I’ll say instead that I was disagreeing in that way when I said something similar above.
Keep in mind that the author perceives himself pretty much like a stereotypical fictional hero: he is the One chosen to Save the World from the Robot Apocalypse, and maybe even Defeat Death and bring us Heaven. No wonder he thinks that advice to fictional heroes is applicable to him.
But when you actually try to apply that advice to people with a “real-life” job which involves coordinating with other people in a complex organization that has to ultimately produce measurable results, you run into problems.
A complex organization, for instance a hospital, needs clear rules detailing who is responsible for what. Sometimes this yields suboptimal outcomes: you notice that somebody is making a mistake and they won’t listen to you, or you don’t tell them because it would be socially unacceptable to do so. But the alternative where any decision can be second-guessed and argued at length until a consensus is reached would paralyse the organization and amplify the negative outcomes of the Dunner-Kruger effect.
Moreover, a culture of heroic responsibility would make accountability essentially impossible: If everybody is responsible for everything, then nobody is responsible for anything. Yes, Alice made a mistake, but how can we blame her without also blaming Bob for not noticing it and stopping her? Or Carol, or Dan, or Erin, and so on.
You and Swimmer963 are making the mistake of applying heroic responsibility only to optimising some local properties. Of course that will mean damaging the greater environment: applying “heroic responsibility” basically means you do your best AGI impression, so if you only optimise for a certain subset of your morality your results aren’t going to be pleasant.
Heroic responsibility only works if you take responsibility for everything. Not just the one patient you’re officially being held accountable for, not just the most likely Everett branches, not just the events you see with your own eyes. If your calling a halt to the human machine you are a part of truly has an expected negative effect, then it is your heroic responsibility to shut up and watch others make horrible mistakes.
A culture of heroic responsibility demands appropriate humility; it demands making damn sure what you’re doing is correct before defying your assigned duties. And if human psychology is such that punishing specific people for specific events works, then it is everyone’s heroic responsibility to make sure that rule exists.
Applying this in practice would, for most people, boil down to effective altruism: acquiring and pooling resources to enable a smaller group to optimise the world directly (after acquiring enough evidence of the group’s reliability that you know they’ll do a better job at it than you), trying to influence policy through political activism, and/or assorted meta-goals, all the while searching for ways to improve the system and obeying the law. Insisting you help directly instead of funding others would be statistical murder in the framework of heroic responsibility.
No: the concept that our ethics is utilitarian is independent from the concept that it is the only acceptable way of making decisions (where “acceptable” is an emotional/moral term).
What is an acceptable way of making decisions (where “acceptable” is an emotional/moral term) looks like an ethical question, how can it be independent from your ethics?
In ethics, the question would be answered by “yes, this ethical system is the only acceptable way to make decisions” by definition. In practice, this fact is not sufficient to make more than 0.01% of the world anywhere near heroically responsible (~= considering ethics the only emotionally/morally/role-followingly acceptable way of making decisions), so apparently the question is not decided by ethics.
Instead, roles and emotions play a large part in determining what is acceptable. In western society, the role of someone who is responsible for everything and not in the corresponding position of power is “the hero”. Yudkowsky (and HPJEV) might have chosen to be heroically responsible because he knows it is the consistent/rational conclusion of human morality and he likes being consistent/rational very much, or because he likes being a hero, or more likely a combination of both. The decision is made due to the role he wants to lead, not due to the ethics itself.
There are various types of consequentalism. The lack of distinction between ethical necessity and supererogation, and the general focus about optimizing the world, are typical of utilitarianism, which is in fact often associated with effective altruism (although it is not strictly necessary for it).
I think it applies to any and all of them just as well, but I (very stupidly) didn’t realize until now that utilitarianism is (a type of) consequentialism.
You and Swimmer963 are making the mistake of applying heroic responsibility only to optimising some local properties. Of course that will mean damaging the greater environment: “heroic responsibility” basically means you do your best AGI impression, so if you only optimise for a certain subset of your morality your results aren’t going to be pleasant.
Heroic responsibility only works if you take responsibility for everything. Not just the one patient you’re officially being held accountable for, not just the most likely Everett branches, not just the events you see with your own eyes. If your calling a halt to the human machine you are a part of truly has an expected negative effect, then it is your heroic responsibility to shut up and watch others make horrible mistakes.
A culture of heroic responsibility demands appropriate humility; it demands making damn sure what you’re doing is correct before defying your assigned duties. And if human psychology is such that a criminal justice system is still appropriate (where specific individuals are punished for specific events), then it is everyone’s job to make sure there is a criminal justice system.
Well, I can’t answer for Eliezer’s intentions, but I can repeat something he has often said about HPMoR: the only statements in HPMoR he is guaranteed to endorse with a straight face and high probability are those made about science/rationality, preferably in an expo-speak section, or those made by Godric Gryffindor, his author-avatar. Harry, Dumbledore, Hermione, and Quirrell are fictional characters: you are not necessarily meant to emulate them, though of course you can if you independently arrive to the conclusion that doing so is a Good Idea.
Is this really supposed to be one of the HPMOR passages which is solely about the fictional character and is not meant to have any application to the real world except as an example of something not to do?
I personally think it is one of the passages in which the unavoidable conceits of literature (ie: that the protagonist’s actions actually matter on a local-world-historical scale) overcome the standard operation of real life. Eliezer might have a totally different view, but of course, he keeps info about HPMoR close to his chest for maximum Fun.
But it’s my understanding that HPMOR was meant to teach about real-world reasoning.
Is this really supposed to be one of the HPMOR passages which is solely about the fictional character and is not meant to have any application to the real world except as an example of something not to do? It certainly doesn’t sound like that.
(Saying this with a straight face)
No, it’s pretty clear that the author intends this to be a real-world lesson. It’s a recurring theme in the Sequences.
I think Eli was disagreeing with the naive application of that lesson to real-world situations, especially ones where established systems are functional.
That said, I don’t want to put words in Eli’s mouth, so I’ll say instead that I was disagreeing in that way when I said something similar above.
Keep in mind that the author perceives himself pretty much like a stereotypical fictional hero: he is the One chosen to Save the World from the Robot Apocalypse, and maybe even Defeat Death and bring us Heaven. No wonder he thinks that advice to fictional heroes is applicable to him.
But when you actually try to apply that advice to people with a “real-life” job which involves coordinating with other people in a complex organization that has to ultimately produce measurable results, you run into problems.
A complex organization, for instance a hospital, needs clear rules detailing who is responsible for what. Sometimes this yields suboptimal outcomes: you notice that somebody is making a mistake and they won’t listen to you, or you don’t tell them because it would be socially unacceptable to do so. But the alternative where any decision can be second-guessed and argued at length until a consensus is reached would paralyse the organization and amplify the negative outcomes of the Dunner-Kruger effect.
Moreover, a culture of heroic responsibility would make accountability essentially impossible:
If everybody is responsible for everything, then nobody is responsible for anything. Yes, Alice made a mistake, but how can we blame her without also blaming Bob for not noticing it and stopping her? Or Carol, or Dan, or Erin, and so on.
You and Swimmer963 are making the mistake of applying heroic responsibility only to optimising some local properties. Of course that will mean damaging the greater environment: applying “heroic responsibility” basically means you do your best AGI impression, so if you only optimise for a certain subset of your morality your results aren’t going to be pleasant.
Heroic responsibility only works if you take responsibility for everything. Not just the one patient you’re officially being held accountable for, not just the most likely Everett branches, not just the events you see with your own eyes. If your calling a halt to the human machine you are a part of truly has an expected negative effect, then it is your heroic responsibility to shut up and watch others make horrible mistakes.
A culture of heroic responsibility demands appropriate humility; it demands making damn sure what you’re doing is correct before defying your assigned duties. And if human psychology is such that punishing specific people for specific events works, then it is everyone’s heroic responsibility to make sure that rule exists.
Applying this in practice would, for most people, boil down to effective altruism: acquiring and pooling resources to enable a smaller group to optimise the world directly (after acquiring enough evidence of the group’s reliability that you know they’ll do a better job at it than you), trying to influence policy through political activism, and/or assorted meta-goals, all the while searching for ways to improve the system and obeying the law. Insisting you help directly instead of funding others would be statistical murder in the framework of heroic responsibility.
So “heroic responsibility” just means “total utilitarianism”?
No: the concept that our ethics is utilitarian is independent from the concept that it is the only acceptable way of making decisions (where “acceptable” is an emotional/moral term).
What is an acceptable way of making decisions (where “acceptable” is an emotional/moral term) looks like an ethical question, how can it be independent from your ethics?
In ethics, the question would be answered by “yes, this ethical system is the only acceptable way to make decisions” by definition. In practice, this fact is not sufficient to make more than 0.01% of the world anywhere near heroically responsible (~= considering ethics the only emotionally/morally/role-followingly acceptable way of making decisions), so apparently the question is not decided by ethics.
Instead, roles and emotions play a large part in determining what is acceptable. In western society, the role of someone who is responsible for everything and not in the corresponding position of power is “the hero”. Yudkowsky (and HPJEV) might have chosen to be heroically responsible because he knows it is the consistent/rational conclusion of human morality and he likes being consistent/rational very much, or because he likes being a hero, or more likely a combination of both. The decision is made due to the role he wants to lead, not due to the ethics itself.
It just means ‘consequentalism’.
There are various types of consequentalism. The lack of distinction between ethical necessity and supererogation, and the general focus about optimizing the world, are typical of utilitarianism, which is in fact often associated with effective altruism (although it is not strictly necessary for it).
I think it applies to any and all of them just as well, but I (very stupidly) didn’t realize until now that utilitarianism is (a type of) consequentialism.
I know. That’s why I had to try to keep a straight face when saying that.
You and Swimmer963 are making the mistake of applying heroic responsibility only to optimising some local properties. Of course that will mean damaging the greater environment: “heroic responsibility” basically means you do your best AGI impression, so if you only optimise for a certain subset of your morality your results aren’t going to be pleasant.
Heroic responsibility only works if you take responsibility for everything. Not just the one patient you’re officially being held accountable for, not just the most likely Everett branches, not just the events you see with your own eyes. If your calling a halt to the human machine you are a part of truly has an expected negative effect, then it is your heroic responsibility to shut up and watch others make horrible mistakes.
A culture of heroic responsibility demands appropriate humility; it demands making damn sure what you’re doing is correct before defying your assigned duties. And if human psychology is such that a criminal justice system is still appropriate (where specific individuals are punished for specific events), then it is everyone’s job to make sure there is a criminal justice system.
Applying this in practice
Well, I can’t answer for Eliezer’s intentions, but I can repeat something he has often said about HPMoR: the only statements in HPMoR he is guaranteed to endorse with a straight face and high probability are those made about science/rationality, preferably in an expo-speak section, or those made by Godric Gryffindor, his author-avatar. Harry, Dumbledore, Hermione, and Quirrell are fictional characters: you are not necessarily meant to emulate them, though of course you can if you independently arrive to the conclusion that doing so is a Good Idea.
I personally think it is one of the passages in which the unavoidable conceits of literature (ie: that the protagonist’s actions actually matter on a local-world-historical scale) overcome the standard operation of real life. Eliezer might have a totally different view, but of course, he keeps info about HPMoR close to his chest for maximum Fun.
I would like to hear eli_sennesh’s response to this...