However, arguments such as “you can’t exactly specify what you want it to do, so it might blackmail the president into building a road in order to reduce the map distance”
The reason that such arguments do not work is that you can specify exactly what it is you want to do, and the programmers did specify exactly that.
In more complex cases, where the programmers are unable to specify exactly what they want, you do get unexpected results that can be thought of as “the program wasn’t optimizing what the programmers thought it should be optimizing, but only a (crude) approximation thereof”.
(an even better example would be one where a genetic algorithm used in circuit design unexpectedly re-purposed some circuit elements to build an antenna, but I cannot find that reference right now)
The reason that such arguments do not work is that you can specify exactly what it is you want to do, and the programmers did specify exactly that.
Which is part of my point. Because you can specify exactly what you want—and because you can’t for the kinds of utility functions that are usually discussed on LW—describing it as having a utility function is technically true, but is misleading because the things you say about those other utility functions won’t carry over to it. Yeah, just because the programmer didn’t explicitly code a utility function doesn’t mean it doesn’t have one—but it often does mean that it doesn’t have one to which your other conclusions about utility functions apply.
The reason that such arguments do not work is that you can specify exactly what it is you want to do, and the programmers did specify exactly that.
In more complex cases, where the programmers are unable to specify exactly what they want, you do get unexpected results that can be thought of as “the program wasn’t optimizing what the programmers thought it should be optimizing, but only a (crude) approximation thereof”. (an even better example would be one where a genetic algorithm used in circuit design unexpectedly re-purposed some circuit elements to build an antenna, but I cannot find that reference right now)
Which is part of my point. Because you can specify exactly what you want—and because you can’t for the kinds of utility functions that are usually discussed on LW—describing it as having a utility function is technically true, but is misleading because the things you say about those other utility functions won’t carry over to it. Yeah, just because the programmer didn’t explicitly code a utility function doesn’t mean it doesn’t have one—but it often does mean that it doesn’t have one to which your other conclusions about utility functions apply.
Thanks for pointing thus out, a lot of people seem confused on the issue, (What’s worse, its largely a map/territory confusion)