That could work… if you take into account the behavior where, if you don’t get enough of one kind of utility, your meta-utility might actually go down.
Utility is whatever you’re trying to maximize the expected value of. If you act in a way that maximizes the expected value of log(happiness) + log(preference fulfillment), for example, this doesn’t mean that you’re risk averse and you have two different kinds of utility. It means that your utility function is log(happiness) + log(preference fulfillment).
That’s true. I’ve been using the term “utility” in a way that is probably wrong. What I really meant is that humans have different kinds of things they want to maximize, and get unhappy if some of them are fulfilled and others aren’t… so their utility functions are complicated.
The obvious problem with standard wireheading is that it only maximizes one of the things humans want.
That could work… if you take into account the behavior where, if you don’t get enough of one kind of utility, your meta-utility might actually go down.
Utility is whatever you’re trying to maximize the expected value of. If you act in a way that maximizes the expected value of log(happiness) + log(preference fulfillment), for example, this doesn’t mean that you’re risk averse and you have two different kinds of utility. It means that your utility function is log(happiness) + log(preference fulfillment).
That’s true. I’ve been using the term “utility” in a way that is probably wrong. What I really meant is that humans have different kinds of things they want to maximize, and get unhappy if some of them are fulfilled and others aren’t… so their utility functions are complicated.
The obvious problem with standard wireheading is that it only maximizes one of the things humans want.