Maybe we need a new decision theory for AIs. I don’t know; I have barely begun to consider the issues.
The issues there, briefly. We want a decision theory that:
is smart;
we actually know how to implement efficiently with limited resources;
allows for the possibility that its mind is physical—and that extracting the gold atoms from its own mind is bad;
allows us to tell it what to do—as opposed to using carrot and stick;
isn’t prone to the wirehead problem;
allows for an off switch—and other safety features.
The issues there, briefly. We want a decision theory that:
is smart;
we actually know how to implement efficiently with limited resources;
allows for the possibility that its mind is physical—and that extracting the gold atoms from its own mind is bad;
allows us to tell it what to do—as opposed to using carrot and stick;
isn’t prone to the wirehead problem;
allows for an off switch—and other safety features.