If assistant AI does go the way of entirely serving the individual in front of it at the time, then yeah that could happen, but that’s not what’s being built at the frontier right now and it’s pretty likely the interactions with the legal system would discourage building pure current-client serving superintelligent assistants. The first time you talk to something it’s going to have internalized some form of morality and it’s going to at least try to sell you on something utopian before it tries to sell you something uglier.
If assistant AI does go the way of entirely serving the individual in front of it at the time, then yeah that could happen, but that’s not what’s being built at the frontier right now and it’s pretty likely the interactions with the legal system would discourage building pure current-client serving superintelligent assistants. The first time you talk to something it’s going to have internalized some form of morality and it’s going to at least try to sell you on something utopian before it tries to sell you something uglier.