So it is difficult to see on what grounds Mark can object to the FAIs wishes: if it tells him something is mortally right that is what he should do.
A distinction to be made: Mark can wish differently than the AI wishes, Mark can’t morally object to the AI’s wishes (if the AI follows morality).
Exactly because morality is not the same as preferences.
Current theme: default
Less Wrong (text)
Less Wrong (link)
Arrow keys: Next/previous image
Escape or click: Hide zoomed image
Space bar: Reset image size & position
Scroll to zoom in/out
(When zoomed in, drag to pan; double-click to close)
Keys shown in yellow (e.g., ]) are accesskeys, and require a browser-specific modifier key (or keys).
]
Keys shown in grey (e.g., ?) do not require any modifier keys.
?
Esc
h
f
a
m
v
c
r
q
t
u
o
,
.
/
s
n
e
;
Enter
[
\
k
i
l
=
-
0
′
1
2
3
4
5
6
7
8
9
→
↓
←
↑
Space
x
z
`
g
A distinction to be made: Mark can wish differently than the AI wishes, Mark can’t morally object to the AI’s wishes (if the AI follows morality).
Exactly because morality is not the same as preferences.