CEV, MR, MP … We do love complexity! Is such love a defining characteristic of intelligent entities?
The main point of all morality, as it is commonly practiced and understood, is restrictive, not promotional. A background moral code should not be expected to suggest goals to the AI, merely to denigrate some of them. The Libertarian “brass” rule is a case in point: “Do not unto others as you would not have them do unto you,” which may be summarized as “Do no harm.”
Of course, “others” has to be defined, perhaps as entities demonstrating sufficiently complex behavior, and exceptions have to be addressed, such as a third-party about to harm a second party. Must you restrain the third-party and likely harm her instead?
“Harm” will also need precise definition but that should be easier.
The brass rule does not require rendering assistance. Would ignoring external delivery of harm be immoral? Yes, by the “Good Samaritan” rule, but not by the brass rule. A near-absolute adherence to the brass rule would solve most moral issues, whether for AI or human.
“Near-absolute” because all the known consequences of an action must be considered in order to determine if any harm is involved and if so, how negatively the harm weighs on the goodness scale. An example of this might be a proposal to dam a river and thereby destroy a species of mussel. Presumably mussels would not exhibit sufficiently complex behavior in their own right, so the question for this consequence becomes how much their loss would harm those who do.
Should an AI protect its own existence? Not if doing so would harm a human or another AI. This addresses Asimov’s three laws, even the first. The brass rule does not require obeying anything.
Apart from avoiding significant harm, the selection of goals does not depend on morality.
--rLsj
“[Y]ou notice that [a proposed action] has some harmful effect in the future on someone you consider morally important. You then end up not being able to do anything …”
Not being able to do that thing, yes, and you shouldn’t do it—unless you can obviate the harm. A case in point is the AGI taking over management of all commodity production and thus putting the current producers out of work. But how is that harmful to them? They can still perform the acts if they wish. They can’t earn a living, you say? Well, then, let the AGI support them. Ah, but then, you suppose, they can’t enjoy the personal worth that meaningful employment reinforces. The what? Let’s stick to the material, please.
“You then end up not being able to do nothing—because doing nothing also leads to harm in the future.”
That does not follow. Doing nothing is always an option under the brass rule. Morally you are not the cause of any harm that then occurred, if any.
Commission vs. omission [of causative actions]: Omitting an action may indeed allow an entity to come to harm, but this is not a moral issue unless acting would harm that entity or another, perhaps to a lesser degree. Commission—taking action—is the problematic case. I repeat: a coded moral system should be restrictive, not promotional. Preventing external harm may be desirable and admirable but is never morally imperative, however physically imperative it may be.
“[Not adding a safety feature] does make the system-designer responsible for [the resulting] accidents in some way …”
Only by the “Good Samaritan” moral code, in which this society is so dolefully steeped. I prefer Caveat emptor. It may be that when AGIs are the principal operators of harmful equipment, the obsession with safety will moderate.