An AI could persuade you to become a kangaroo—this is a broken criterion for decision-making.
I am skeptical that rationality and exponentially greater-than-human intelligence actually confers this power.
It doesn’t matter if it does or not; the fact that you can conceive of situations where persuadability would fail as a criterion immediately means it fails.
Well, that was the big controversy over the AI Box experiments, so no need to rehash all that here.
An AI could persuade you to become a kangaroo—this is a broken criterion for decision-making.
I am skeptical that rationality and exponentially greater-than-human intelligence actually confers this power.
It doesn’t matter if it does or not; the fact that you can conceive of situations where persuadability would fail as a criterion immediately means it fails.
Well, that was the big controversy over the AI Box experiments, so no need to rehash all that here.