I remember taking the ridiculous ‘moral compass quiz’ which was basically 101 repetitions of the trolley problem, sometimes by inaction, stated differently each time (and then symmetrized so each action variant was replaced by inaction). We were explicitly told to assume our knowledge is certain. Some of the circumstances assumed long chains of improbable-seeming events between sacrificing someone and saving 5 others.
I told ’em that maintaining uncertainty is irremovable from morality. Morality minus the capability of considering that you’re wrong is broken, and leads to horrible outcomes.
We were explicitly told to assume our knowledge is certain. Some of the circumstances assumed long chains of improbable-seeming events between sacrificing someone and saving 5 others.
Eliezer commented on this back in “Ends Don’t Justify Means (Among Humans)”, attempting to reconcile consequentialism with the possibility (observed in human politics) that humans may be running on hardware incapable of consequentialism accurate enough for extreme cases:
And now the philosopher comes and presents their “thought experiment”—setting up a scenario in which, by stipulation, the only possible way to save five innocent lives is to murder one innocent person, and this murder is certain to save the five lives. “There’s a train heading to run over five innocent people, who you can’t possibly warn to jump out of the way, but you can push one innocent person into the path of the train, which will stop the train. These are your only options; what do you do?”
An altruistic human, who has accepted certain deontological prohibits—which seem well justified by some historical statistics on the results of reasoning in certain ways on untrustworthy hardware—may experience some mental distress, on encountering this thought experiment.
So here’s a reply to that philosopher’s scenario, which I have yet to hear any philosopher’s victim give:
“You stipulate that the only possible way to save five innocent lives is to murder one innocent person, and this murder will definitely save the five lives, and that these facts are known to me with effective certainty. But since I am running on corrupted hardware, I can’t occupy the epistemic state you want me to imagine. Therefore I reply that, in a society of Artificial Intelligences worthy of personhood and lacking any inbuilt tendency to be corrupted by power, it would be right for the AI to murder the one innocent person to save five, and moreover all its peers would agree. However, I refuse to extend this reply to myself, because the epistemic state you ask me to imagine, can only exist among other kinds of people than human beings.”
Now, to me this seems like a dodge. I think the universe is sufficiently unkind that we can justly be forced to consider situations of this sort. The sort of person who goes around proposing that sort of thought experiment, might well deserve that sort of answer. But any human legal system does embody some answer to the question “How many innocent people can we put in jail to get the guilty ones?”, even if the number isn’t written down.
As a human, I try to abide by the deontological prohibitions that humans have made to live in peace with one another. But I don’t think that our deontological prohibitions are literally inherently nonconsequentially terminally right. I endorse “the end doesn’t justify the means” as a principle to guide humans running on corrupted hardware, but I wouldn’t endorse it as a principle for a society of AIs that make well-calibrated estimates. (If you have one AI in a society of humans, that does bring in other considerations, like whether the humans learn from your example.)
I have to admit, though, this does seem uncomfortably like the old aphorism quod licet Jovi, non licet bovi — “what is permitted to Jupiter is not permitted to a cow.”
But since I am running on corrupted hardware, I can’t occupy the epistemic state you want me to imagine.
It occurs to me that many (maybe even most) hypotheticals require you to accept an unreasonable epistemic state. Even something so simple as trusting that Omega is telling the truth [and that his “fair coin” was a quantum random number generator rather than, say, a metal disc that he flipped with a deterministic amount of force, but that’s easier to grant as simple sloppy wording]
Somehow, I doubt I could achieve any more than 1% confidence that, say, the best plan to save 5 children in a burning building was to stab the fireman with a piece of glass and knock his ladder over and pull him off it so his body fell nearly straight down and could serve a cushion for the 5 children, who would each get off the body soon enough to let the next one follow. Actually, I was not only to assume that that is the best plan, but that it is certain to work, and if I don’t carry it out, the 5 children will certainly die, but not the fireman or me.
Or, alternately, that I’m on a motorboat, and there’s a shark in the water who will eat 5 people unless I hit the accelerator soon enough and hard enough that one of my existing passengers will certainly be knocked off the back, and will certainly be eaten by the shark (perhaps it was a second shark, which would raise the likelihood that I couldn’t do some fancy boatwork to get ’em all). I do not have time to tell anyone to hold on—I absolutely MUST goose the gas to get there that extra half second early that somehow makes the difference between all five being eaten and none of the five being eaten.
So your issue isn’t actually with (moral) reasoning under uncertainty or the trolley problem in general, it’s just with highly specific, really bad examples. Gotcha.
I think in general, if you find your plans to be complicated, involve causing someone else a large up-front cost, and you have very high confidence in the plan, the moral thing is to audit your certainty.
Just because I feel 99% certain of some information, it does not mean that I am right in 99% of situations. This should be included in calculation.
Even if I were a perfect Bayesian reasoner, most people aren’t. Are we going to solve this one specific situation, or are we creating a general rule that all people will follow? Because it may be better to let 5 people die once than to create a precedent that will allow all kinds of irrational folks go around and kill a random person every time they feel that by doing so they have prevented hypothetical deaths of five other persons.
(If you want to go on and ask whether it is good to kill one person to prevent 99% chance of five people dying, assuming that we are absolutely sure about all these data, and assuming that this sets no kind of precedent or a slippery slope for people in similar circumstances… then the answer is: yes. -- But in real life the probability that such situation happens is much smaller than probability that I misunderstood the situation.)
Sure, but knowing that doesn’t necessarily help. If I, in my travels, find myself in a situation that seems to me to be standing by a switch on a train track, while what I estimate to be a train approaches in such a way that I expect it will go down track A if left alone or track B if I pull the switch, and I observe what appear to be six people, one of whom is tied to track B and five of whom are tied to track A, it is of course possible that all of my observations and estimations are incorrect. But I’m still left with the question of what to do.
I mean, sure, if I pull the switch and it turns out that the five people who I thought were tied to track A are just lifelike mannequins, then I’ve just traded away a world in which nobody dies for a world in which someone dies, which isn’t a choice I endorse.
On the other hand, if I don’t pull the switch and it turns out that the person I thought was tied to track B is just a lifelike mannequin, then I’ve just traded away a world in which nobody dies for a world in which five people die, which isn’t a choice I endorse either.
Any choice I make might be wrong, and might result in unnecessary deaths. But that doesn’t justify any particular choice, including the choice to not intervene.
The example you listed doesn’t come close to addressing the topic. It isn’t trolley problems in general—it’s a particular variant of trolley problems where the rest of the information in the problem is fighting the certainty you are told to assume about the options.
I remember taking the ridiculous ‘moral compass quiz’ which was basically 101 repetitions of the trolley problem, sometimes by inaction, stated differently each time (and then symmetrized so each action variant was replaced by inaction). We were explicitly told to assume our knowledge is certain. Some of the circumstances assumed long chains of improbable-seeming events between sacrificing someone and saving 5 others.
I told ’em that maintaining uncertainty is irremovable from morality. Morality minus the capability of considering that you’re wrong is broken, and leads to horrible outcomes.
Eliezer commented on this back in “Ends Don’t Justify Means (Among Humans)”, attempting to reconcile consequentialism with the possibility (observed in human politics) that humans may be running on hardware incapable of consequentialism accurate enough for extreme cases:
I have to admit, though, this does seem uncomfortably like the old aphorism quod licet Jovi, non licet bovi — “what is permitted to Jupiter is not permitted to a cow.”
It occurs to me that many (maybe even most) hypotheticals require you to accept an unreasonable epistemic state. Even something so simple as trusting that Omega is telling the truth [and that his “fair coin” was a quantum random number generator rather than, say, a metal disc that he flipped with a deterministic amount of force, but that’s easier to grant as simple sloppy wording]
In general, thought experiments that depend on an achievable epistemic state can actually be performed and don’t need to remain thought experiments.
They can depend on an achievable epistemic state, but be horribly impractical or immoral to set up (hello trolley problems).
Okay… so, you are 99 percent certain of your information. Does that change your answer?
Somehow, I doubt I could achieve any more than 1% confidence that, say, the best plan to save 5 children in a burning building was to stab the fireman with a piece of glass and knock his ladder over and pull him off it so his body fell nearly straight down and could serve a cushion for the 5 children, who would each get off the body soon enough to let the next one follow. Actually, I was not only to assume that that is the best plan, but that it is certain to work, and if I don’t carry it out, the 5 children will certainly die, but not the fireman or me.
Or, alternately, that I’m on a motorboat, and there’s a shark in the water who will eat 5 people unless I hit the accelerator soon enough and hard enough that one of my existing passengers will certainly be knocked off the back, and will certainly be eaten by the shark (perhaps it was a second shark, which would raise the likelihood that I couldn’t do some fancy boatwork to get ’em all). I do not have time to tell anyone to hold on—I absolutely MUST goose the gas to get there that extra half second early that somehow makes the difference between all five being eaten and none of the five being eaten.
So your issue isn’t actually with (moral) reasoning under uncertainty or the trolley problem in general, it’s just with highly specific, really bad examples. Gotcha.
I think in general, if you find your plans to be complicated, involve causing someone else a large up-front cost, and you have very high confidence in the plan, the moral thing is to audit your certainty.
Just because I feel 99% certain of some information, it does not mean that I am right in 99% of situations. This should be included in calculation.
Even if I were a perfect Bayesian reasoner, most people aren’t. Are we going to solve this one specific situation, or are we creating a general rule that all people will follow? Because it may be better to let 5 people die once than to create a precedent that will allow all kinds of irrational folks go around and kill a random person every time they feel that by doing so they have prevented hypothetical deaths of five other persons.
(If you want to go on and ask whether it is good to kill one person to prevent 99% chance of five people dying, assuming that we are absolutely sure about all these data, and assuming that this sets no kind of precedent or a slippery slope for people in similar circumstances… then the answer is: yes. -- But in real life the probability that such situation happens is much smaller than probability that I misunderstood the situation.)
Sure, but knowing that doesn’t necessarily help. If I, in my travels, find myself in a situation that seems to me to be standing by a switch on a train track, while what I estimate to be a train approaches in such a way that I expect it will go down track A if left alone or track B if I pull the switch, and I observe what appear to be six people, one of whom is tied to track B and five of whom are tied to track A, it is of course possible that all of my observations and estimations are incorrect. But I’m still left with the question of what to do.
I mean, sure, if I pull the switch and it turns out that the five people who I thought were tied to track A are just lifelike mannequins, then I’ve just traded away a world in which nobody dies for a world in which someone dies, which isn’t a choice I endorse.
On the other hand, if I don’t pull the switch and it turns out that the person I thought was tied to track B is just a lifelike mannequin, then I’ve just traded away a world in which nobody dies for a world in which five people die, which isn’t a choice I endorse either.
Any choice I make might be wrong, and might result in unnecessary deaths. But that doesn’t justify any particular choice, including the choice to not intervene.
The example you listed doesn’t come close to addressing the topic. It isn’t trolley problems in general—it’s a particular variant of trolley problems where the rest of the information in the problem is fighting the certainty you are told to assume about the options.