I don’t think it will be very difficult to impart your intentions into a sufficiently advanced machine
Counterargument: it will be easy to impart an approximate version of your intentions, but hard to control the evolution of those values as you crank up the power. E.g. evolution, humans, make us want sex, we invent condoms.
No-one will really care about this until it’s way too late and we’re all locked up in nice padded cells and drugged up, or something equally bad but hard for me to imagine right now.
Counterargument: it will be easy to impart an approximate version of your intentions, but hard to control the evolution of those values as you crank up the power. E.g. evolution, humans, make us want sex, we invent condoms.
No-one will really care about this until it’s way too late and we’re all locked up in nice padded cells and drugged up, or something equally bad but hard for me to imagine right now.