Huh. It’s not clear to me that they’d have something equivalent to happiness, but if they did I might be able to tell. Even if they did, though, they wouldn’t necessarily care about happiness, unless we really screwed up in designing it (like evolution did). Even if it was some sort of direct measure of utility, it’d only be a valuable metric insofar as it reflected F0.
It seems somewhat arbitrary to pick “maximize the function stored in this location” as the “real” fundamental value of the AI. A proper utility maximizer would have “maximize this specific function”, or something. I mean, you could just as easily say that the AI would reason “hey, it’s tough to maximize utility functions, I might as well just switch from caring about utility to caring about nothing, that’d be pretty easy to deal with.”
Huh. It’s not clear to me that they’d have something equivalent to happiness, but if they did I might be able to tell. Even if they did, though, they wouldn’t necessarily care about happiness, unless we really screwed up in designing it (like evolution did). Even if it was some sort of direct measure of utility, it’d only be a valuable metric insofar as it reflected F0.
It seems somewhat arbitrary to pick “maximize the function stored in this location” as the “real” fundamental value of the AI. A proper utility maximizer would have “maximize this specific function”, or something. I mean, you could just as easily say that the AI would reason “hey, it’s tough to maximize utility functions, I might as well just switch from caring about utility to caring about nothing, that’d be pretty easy to deal with.”