The relevant point is his latter claim: “in particular with respect to “learn ‘don’t steal’ rather than ‘don’t get caught’.”″ I think this is a very strong conclusion, relative to available data.
I think humans don’t steal mostly because society enforces that norm. Toward weaker “other” groups that aren’t part of your society (farmed animals, weaker countries, etc) there’s no such norm, and humans often behave badly toward such groups. And to AIs, humans will be a weaker “other” group. So if alignment of AIs to human standard is a complete success—if AIs learn to behave toward weaker “other” groups exactly as humans behave toward such groups—the result will be bad for humans.
It gets even worse because AIs, unlike humans, aren’t raised to be moral. They’re raised by corporations with a goal to make money, with a thin layer of “don’t say naughty words” morality. We already know corporations will break rules, bend rules, lobby to change rules, to make more money and don’t really mind if people get hurt in the process. We’ll see more of that behavior when corporations can make AIs to further their goals.
While I definitely get your point, I think the argument Turntrout is responding to isn’t about corporations using their aligned AIs to make a dystopia for everyone else, but rather about AI being aligned to anyone at all.
I think humans don’t steal mostly because society enforces that norm. Toward weaker “other” groups that aren’t part of your society (farmed animals, weaker countries, etc) there’s no such norm, and humans often behave badly toward such groups. And to AIs, humans will be a weaker “other” group. So if alignment of AIs to human standard is a complete success—if AIs learn to behave toward weaker “other” groups exactly as humans behave toward such groups—the result will be bad for humans.
It gets even worse because AIs, unlike humans, aren’t raised to be moral. They’re raised by corporations with a goal to make money, with a thin layer of “don’t say naughty words” morality. We already know corporations will break rules, bend rules, lobby to change rules, to make more money and don’t really mind if people get hurt in the process. We’ll see more of that behavior when corporations can make AIs to further their goals.
While I definitely get your point, I think the argument Turntrout is responding to isn’t about corporations using their aligned AIs to make a dystopia for everyone else, but rather about AI being aligned to anyone at all.