That is a right sentiment about strength: there are no simple rules, only goals, which makes a creative mind extremely dangerous. And we shouldn’t build things like this without understanding what the outcome will be. This is one of the reasons it’s important to understand human values in this light, to guard them from this destructive potential.
Whatever you want accomplished, whatever you want averted, instrumental rationality defines an optimal way of doing that (without necessarily giving the real-world means, that’s a next step). If you really want life to continue as before, the correctly implemented explicit utility function for doing that won’t lead a Bayesian optimizer to do something horrible. (Although inaction may be considered horrible in itself, where so much more could’ve been done.)
That is a right sentiment about strength: there are no simple rules, only goals, which makes a creative mind extremely dangerous. And we shouldn’t build things like this without understanding what the outcome will be. This is one of the reasons it’s important to understand human values in this light, to guard them from this destructive potential.
Whatever you want accomplished, whatever you want averted, instrumental rationality defines an optimal way of doing that (without necessarily giving the real-world means, that’s a next step). If you really want life to continue as before, the correctly implemented explicit utility function for doing that won’t lead a Bayesian optimizer to do something horrible. (Although inaction may be considered horrible in itself, where so much more could’ve been done.)