The trouble with coming up with a utility function is that people seem to start by coming up with ones they think sound good, i.e. stated utility function as social signaling. Which is pretty much moral system as social signaling, i.e. saying “I work by this system, so you can predict me well enough to trust me to deal with me.”
This effect has to be understood and consciously worked past to get the actual answer to “What is my utility function even if no-one else can see what I’ve worked out?” The last is the big question in my life (and I suspect that of anyone, really) and I still only have fragments of an answer, mostly gained by attempting to observe my behaviour and feelings. This is a hard question.
In a community where instrumental rationality is high-status, there will be social pressure to behave according to your stated utility function. So you have to be careful to at least state a function that’s sufficiently compatible with your real one that people won’t notice the discrepancy. If they do, they will try to help you overcome your “akrasia”.
Note that I said “compatible”, not “similar”. So your real utility function could be “complete selfishness”. If complete selfishness requires support from a rationalist community, you may wish to signal cooperation by stating that your utility function is “complete altruism”.
The result of this is that you will find doing lots of world-improving stuff (to keep the support of the rationalist community) , and your true selfish utility function will ensure you have lots of fun, find lots of sexual opportunities, etc. while doing it.
I should have mentioned the “stated” vs. “private” distinction in the above post. I’ll write these ideas up in a future post, but I’ll probably need to explain a bit more of my view on signalling first.
The trouble with coming up with a utility function is that people seem to start by coming up with ones they think sound good, i.e. stated utility function as social signaling. Which is pretty much moral system as social signaling, i.e. saying “I work by this system, so you can predict me well enough to trust me to deal with me.”
This effect has to be understood and consciously worked past to get the actual answer to “What is my utility function even if no-one else can see what I’ve worked out?” The last is the big question in my life (and I suspect that of anyone, really) and I still only have fragments of an answer, mostly gained by attempting to observe my behaviour and feelings. This is a hard question.
In a community where instrumental rationality is high-status, there will be social pressure to behave according to your stated utility function. So you have to be careful to at least state a function that’s sufficiently compatible with your real one that people won’t notice the discrepancy. If they do, they will try to help you overcome your “akrasia”.
Note that I said “compatible”, not “similar”. So your real utility function could be “complete selfishness”. If complete selfishness requires support from a rationalist community, you may wish to signal cooperation by stating that your utility function is “complete altruism”.
The result of this is that you will find doing lots of world-improving stuff (to keep the support of the rationalist community) , and your true selfish utility function will ensure you have lots of fun, find lots of sexual opportunities, etc. while doing it.
I should have mentioned the “stated” vs. “private” distinction in the above post. I’ll write these ideas up in a future post, but I’ll probably need to explain a bit more of my view on signalling first.