First, it seems to me rather clear what macroscopic physics I attach utility to...
This does not strike me as the sort of thing which will be easy to write out.
Of course it is not easy to write out. Humanity preferences are highly complex. By “clear” I only meant that it’s clear something like this exists, not that I or anyone can write it out.
What if humans value something like observer-independent beauty? EG, valuing beautiful things existing regardless of whether anyone observes their beauty.
This seems ill-defined. What is a “thing”? What does it mean for a thing to “exist”? I can imagine valuing beautiful wild nature, by having “wild nature” be a part of the innate ontology. I can even imagine preferring certain computations to have results with certain properties. So, we can consider a preference that some kind of simplicity-prior-like computation outputs bit sequences with some complexity theoretic property we call “beauty”. But if you want to go even more abstract than that, I don’t know how to make sense of that (“make sense” not as “formalize” but just as “understand what you’re talking about”).
It would be best if you had a simple example, like a diamond maximizer, where it’s more or less clear that it makes sense to speak of agents with this preference.
What I have in mind is complicated interactions between different ontologies. Suppose that we have one ontology—the ontology of classical economics—in which...
And we have another ontology—the hippie ontology—in which...
And suppose what we want to do is try to reconcile the value-content of these two different perspectives.
Why do we want to reconcile them? I think that you might be mixing two different questions here. The first question is what kind of preferences ideal “non-myopic” agents can have. About this I maintain that my framework provides a good answer, or at least a good first approximation of the answer. The second question is what kind of preferences humans can have. But humans are agents with only semi-coherent preferences, and I see no reason to believe things like reconciling classical economics with hippies should follow from any natural mathematical formalism. Instead, I think we should model humans as having preferences that change over time, and the detailed dynamics of the change is just a function the AI needs to learn, not some consequence of mathematical principles of rationality.
Of course it is not easy to write out. Humanity preferences are highly complex. By “clear” I only meant that it’s clear something like this exists, not that I or anyone can write it out.
This seems ill-defined. What is a “thing”? What does it mean for a thing to “exist”? I can imagine valuing beautiful wild nature, by having “wild nature” be a part of the innate ontology. I can even imagine preferring certain computations to have results with certain properties. So, we can consider a preference that some kind of simplicity-prior-like computation outputs bit sequences with some complexity theoretic property we call “beauty”. But if you want to go even more abstract than that, I don’t know how to make sense of that (“make sense” not as “formalize” but just as “understand what you’re talking about”).
It would be best if you had a simple example, like a diamond maximizer, where it’s more or less clear that it makes sense to speak of agents with this preference.
Why do we want to reconcile them? I think that you might be mixing two different questions here. The first question is what kind of preferences ideal “non-myopic” agents can have. About this I maintain that my framework provides a good answer, or at least a good first approximation of the answer. The second question is what kind of preferences humans can have. But humans are agents with only semi-coherent preferences, and I see no reason to believe things like reconciling classical economics with hippies should follow from any natural mathematical formalism. Instead, I think we should model humans as having preferences that change over time, and the detailed dynamics of the change is just a function the AI needs to learn, not some consequence of mathematical principles of rationality.