Based on Kaj’s concept of needs-meeting machinery and subsystems in the brain, I developed a framework for thinking about human values (which may deviate from Kaj’s thinking).
I see human values as under-defined in many places. Sometimes you can get crystallized “life goals” where someone locks in an optimizing mindset around specific objectives. (This part may be particularly interesting for looking for analogies with AI?) The process of forming life goals seems to involve forming an identity. From my text (“The Life-Goals Framework: How I Reason About Morality as an Anti-Realist”):
One of many takeaways I got from reading Kaj Sotala’s multi-agent models of mind sequence (as well as comments by him) is that we can model people as pursuers of deep-seated needs. In particular, we have subsystems (or “subagents”) in our minds devoted to various needs-meeting strategies. The subsystems contribute behavioral strategies and responses to help maneuver us toward states where our brain predicts our needs will be satisfied. We can view many of our beliefs, emotional reactions, and even our self-concept/identity as part of this set of strategies. Like life plans, [“life plans” being objectives we set out to achieve but aren’t all that serious about] life goals are “merely” components of people’s needs-meeting machinery.[8]
Still, as far as components of needs-meeting machinery go, life goals are pretty unusual. Having life goals means to care about an objective enough to (do one’s best to) disentangle success on it from the reasons we adopted said objective in the first place. The objective takes on a life of its own, and the two aims (meeting one’s needs vs. progressing toward the objective) come apart. Having a life goal means having a particular kind of mental organization so that “we” – particularly the rational, planning parts of our brain – come to identify with the goal more so than with our human needs.
[...]
Whether someone forms a life goal may also depend on whether the life-goal identity is reinforced (at least initially) around the time of the first adoption or when the person initially contemplates what it could be like to adopt the life goal. If assuming a given identity was instantly detrimental to our needs, we’d be less likely to power up the mental machinery to make it stable / protect it from goal drift.
In humans, I think the way we adopt specific values isn’t too dissimilar from the way we adopt career paths, or even how we choose leisure and lifestyle activities. For instance, I discuss an example where someone wants to decide between spending the weekend cozily at home vs. going skiing:
There’s a normative component to something as mundane as choosing leisure activities. In the weekend example, I’m not just trying to assess the answer to empirical questions like “Which activity would contain fewer seconds of suffering/happiness” or “Which activity would provide me with lasting happy memories.” I probably already know the answer to those questions. What’s difficult about deciding is that some of my internal motivations conflict. For example, is it more important to be comfortable, or do I want to lead an active life? When I make up my mind in these dilemma situations, I tend to reframe my options until the decision seems straightforward. I know I’ve found the right decision when there’s no lingering fear that the currently-favored option wouldn’t be mine, no fear that I’m caving to social pressures or acting (too much) out of akrasia, impulsivity or some other perceived weakness of character.[21]
We tend to have a lot of freedom in how we frame our decision options. We use this freedom, this reframing capacity, to become comfortable with the choices we are about to make. In case skiing wins out, then “warm and cozy” becomes “lazy and boring,” and “cold and tired” becomes “an opportunity to train resilience / apply Stoicism.” This reframing ability is a double-edged sword: it enables rationalizing, but it also allows us to stick to our beliefs and values when we’re facing temptations and other difficulties.
Whether a given motivational pull – such as the need for adventure, or (e.g.,) the desire to have children – is a bias or a fundamental value is not set in stone; it depends on our other motivational pulls and the overarching self-concept we’ve formed.
Then, after discussing how we make career choice decisions in the same way, I argue that we even form life goals in this way:
Lastly, we also use “planning mode” to choose between life goals. A life goal is a part of our identity – just like one’s career or lifestyle (but it’s even more serious).
We can frame choosing between life goals as choosing between “My future with life goal A” and “My future with life goal B” (or “My future without a life goal”). (Note how this is relevantly similar to “My future on career path A” and “My future on career path B.”)
Consider morality-inspired life goals. For moral reflection to move from an abstract hobby to something that guides us, we have to move beyond contemplating how strangers should behave in thought experiments. At some point, we also have to envision ourselves adopting an identity of “wanting to do good.”
[...]
It’s important to note that choosing a life goal doesn’t necessarily mean that we predict ourselves to have the highest life satisfaction (let alone the most increased moment-to-moment well-being) with that life goal in the future. Instead, it means that we feel the most satisfied about the particular decision (to adopt the life goal) in the present, when we commit to the given plan, thinking about our future.
Kaj Sotala’s multi-agent models of mind sequence and his paper Defining human values for value learners may be relevant.
Based on Kaj’s concept of needs-meeting machinery and subsystems in the brain, I developed a framework for thinking about human values (which may deviate from Kaj’s thinking).
I see human values as under-defined in many places. Sometimes you can get crystallized “life goals” where someone locks in an optimizing mindset around specific objectives. (This part may be particularly interesting for looking for analogies with AI?) The process of forming life goals seems to involve forming an identity. From my text (“The Life-Goals Framework: How I Reason About Morality as an Anti-Realist”):
In humans, I think the way we adopt specific values isn’t too dissimilar from the way we adopt career paths, or even how we choose leisure and lifestyle activities. For instance, I discuss an example where someone wants to decide between spending the weekend cozily at home vs. going skiing:
Then, after discussing how we make career choice decisions in the same way, I argue that we even form life goals in this way: