Your comment is very optimistic and nearly converted me to an optimistic view of FAI. But humans already understand a lot of tacit knowledge, yet a superpowered human will almost certainly be unFriendly, IMO. So the path from human-level understanding to FAI still contains considerable dangers.
A human granted superpowers might indeed be unfriendly (and this conclusion doesn’t necessarily depend on the human having been obviously evil hitherto, though it does depend on him being the sole possessor of same in a world where such superpowers didn’t otherwise exist, so it’s not directly relevant). But that’s a different problem: it arises from the possibility of the human having goals incompatible with yours and mine, and no qualms about imposing them on us by force. We don’t worry that the newly minted supervillain would fail to understand what we want, only that he wouldn’t care. That’s different from the scenario where we are building machines, where the problem is that of making them understand what we want them to do.
Your comment is very optimistic and nearly converted me to an optimistic view of FAI. But humans already understand a lot of tacit knowledge, yet a superpowered human will almost certainly be unFriendly, IMO. So the path from human-level understanding to FAI still contains considerable dangers.
A human granted superpowers might indeed be unfriendly (and this conclusion doesn’t necessarily depend on the human having been obviously evil hitherto, though it does depend on him being the sole possessor of same in a world where such superpowers didn’t otherwise exist, so it’s not directly relevant). But that’s a different problem: it arises from the possibility of the human having goals incompatible with yours and mine, and no qualms about imposing them on us by force. We don’t worry that the newly minted supervillain would fail to understand what we want, only that he wouldn’t care. That’s different from the scenario where we are building machines, where the problem is that of making them understand what we want them to do.
Thanks. You’re right.