Relevant question: what does the cognitive science literature on choice-making, preference, and valuation have to say about all this? What mathematical structure actually does model human preferences?
Given that we run on top of neural networks and seem to use some Bayesian algorithms for certain forms of learning (citations available), I currently expect that our choice-making mechanisms might involve conditioning on features or states of our environment at some fundamental level.
My first guess would be that evolution has selected us for circular preferences that our genes money-pump so that we will propagate them. You can’t get off this ride while you’re human.
:-) I mean that if you embody human value, you’ll probably be a money-pumpable entity. Very few humans actually achieve an end to desire while still alive and mentally active.
I’ll take the challenge, then. I was already walking around thinking that the Four Noble Truths of the Buddha are a bunch of depressing bullshit that need to be fixed.
I’ve seen a bunch of different theories backed with varying amounts of experimental data—for instance, this, this and this—but I haven’t looked at them enough to tell which ones seem most correct.
That said, I still don’t remember running into any thorough discussion of what human preferences are, other than just “something that makes us make some choice in some situations”. I mention here that
some of our preferences are implicit in our automatic habits (the things that we show we value with our daily routines), some in the preprocessing of sensory data that our brains carry out (the things and ideas that are ”painted with” positive associations or feelings), and some in the configuration of our executive processes (the actions we actually end up doing in response to novel or conflicting situations).
And I’m a little skeptical of any theory of human preferences that doesn’t attempt to make any such breakdown and only takes a “black box” approach of looking at the outputs of our choice mechanism.
Relevant question: what does the cognitive science literature on choice-making, preference, and valuation have to say about all this? What mathematical structure actually does model human preferences?
Given that we run on top of neural networks and seem to use some Bayesian algorithms for certain forms of learning (citations available), I currently expect that our choice-making mechanisms might involve conditioning on features or states of our environment at some fundamental level.
My first guess would be that evolution has selected us for circular preferences that our genes money-pump so that we will propagate them. You can’t get off this ride while you’re human.
Is that a challenge?
:-) I mean that if you embody human value, you’ll probably be a money-pumpable entity. Very few humans actually achieve an end to desire while still alive and mentally active.
I’ll take the challenge, then. I was already walking around thinking that the Four Noble Truths of the Buddha are a bunch of depressing bullshit that need to be fixed.
I’ve seen a bunch of different theories backed with varying amounts of experimental data—for instance, this, this and this—but I haven’t looked at them enough to tell which ones seem most correct.
That said, I still don’t remember running into any thorough discussion of what human preferences are, other than just “something that makes us make some choice in some situations”. I mention here that
And I’m a little skeptical of any theory of human preferences that doesn’t attempt to make any such breakdown and only takes a “black box” approach of looking at the outputs of our choice mechanism.
Looks like the relevant textbook came out with an updated edition this year.