When a highly intelligent self-driving boat on the bank of a lake doesn’t try to save a drowning child, what is the nature of the problem? Perhaps the boat is morally repugnant and the world will be a better place if it experiences a rapid planned disassembly. Or the boat is a person and disassembling or punishing them would in itself be wrong, apart from any instrumental value gained in the other consequences of such an action. Or the fact that they are a person yet do nothing qualifies them as evil and deserving of disassembly, which would not be the case had they not been a person. Maybe the boat is making an error of judgement, that is according to some decision theory and under human-aligned values the correct actions importantly differ from the actual actions taken by the boat. Or maybe this particular boat is simply instrumentally useless for the purpose of saving drowning children, in the same way that a marker buoy would be useless.
What should be done about this situation? That’s again a different question, the one asking it might be the boat themself, and a solution might not involve the boat at all.
There are many ways of framing the situation, looking for models of what’s going on that have radically different shapes. It’s crucial to establish some sort of clarity about what kind of model we are looking for, what kind of questions or judgements we are trying to develop. You seem to be conflating a lot of this, so I gave examples of importantly different framings. Some of these might fit what you are looking for, or help with noticing specific cases where they are getting mixed up.
I feel like I was reasonably clear that the major concern was about how utilitarianism interacts with being human, as much of the focus is on moral luck.
Insofar as an intelligent boat can be made miserable by failing to live up to an impossible moral system, well, I don’t know, maybe don’t design it that way.
When a highly intelligent self-driving boat on the bank of a lake doesn’t try to save a drowning child, what is the nature of the problem? Perhaps the boat is morally repugnant and the world will be a better place if it experiences a rapid planned disassembly. Or the boat is a person and disassembling or punishing them would in itself be wrong, apart from any instrumental value gained in the other consequences of such an action. Or the fact that they are a person yet do nothing qualifies them as evil and deserving of disassembly, which would not be the case had they not been a person. Maybe the boat is making an error of judgement, that is according to some decision theory and under human-aligned values the correct actions importantly differ from the actual actions taken by the boat. Or maybe this particular boat is simply instrumentally useless for the purpose of saving drowning children, in the same way that a marker buoy would be useless.
What should be done about this situation? That’s again a different question, the one asking it might be the boat themself, and a solution might not involve the boat at all.
Would you mind unpacking this?
There are many ways of framing the situation, looking for models of what’s going on that have radically different shapes. It’s crucial to establish some sort of clarity about what kind of model we are looking for, what kind of questions or judgements we are trying to develop. You seem to be conflating a lot of this, so I gave examples of importantly different framings. Some of these might fit what you are looking for, or help with noticing specific cases where they are getting mixed up.
I feel like I was reasonably clear that the major concern was about how utilitarianism interacts with being human, as much of the focus is on moral luck.
Insofar as an intelligent boat can be made miserable by failing to live up to an impossible moral system, well, I don’t know, maybe don’t design it that way.