You only need a language expressive enough to describe everything preference refers to, and no more.
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
This seems to be the correct solution to ontology problem—describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
ETA: And sometimes I suspect even “what is the right utility function?” is really more like “what is the right decision theory?” than we currently believe. In other words there is objective morality after all, but we’re currently just too stupid or philosophically incompetent to figure out what it is.
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
The general idea seems right. If the existing languages are inadequate, they at least seem adequate for a full-featured prototype: figure out decision theory (and hence notion of preference) in terms of standard logic, then move on as necessary for extending expressive power. This should stop at some point, since this exercise at formality is aimed at construction of a program.
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
I don’t see clearly the distinction you’re making, so let me describe how I see it. Some design choices in constructing FAI would certainly be specific to our minds (values), but the main assumption to my approach to FAI is exactly that a large portion of design choices in FAI can be specified as a natural category in human brains, something we can point a simple mirror at and say “there!”, with the mirror doing most of the work in determining what goes into the FAI. I call the automated design choices “preference”, and the mirror (theory of mirror) “decision theory”, with the slot “notion of preference” that is to be filled in automatically. So, there is no question of which one of “decision theory” and “preference” is “essential”, both play a role. The worry is about the necessary size of the manually designed “decision theory” part, and whether it’s humanly possible to construct it.
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
ETA: And sometimes I suspect even “what is the right utility function?” is really more like “what is the right decision theory?” than we currently believe. In other words there is objective morality after all, but we’re currently just too stupid or philosophically incompetent to figure out what it is.
The general idea seems right. If the existing languages are inadequate, they at least seem adequate for a full-featured prototype: figure out decision theory (and hence notion of preference) in terms of standard logic, then move on as necessary for extending expressive power. This should stop at some point, since this exercise at formality is aimed at construction of a program.
I don’t see clearly the distinction you’re making, so let me describe how I see it. Some design choices in constructing FAI would certainly be specific to our minds (values), but the main assumption to my approach to FAI is exactly that a large portion of design choices in FAI can be specified as a natural category in human brains, something we can point a simple mirror at and say “there!”, with the mirror doing most of the work in determining what goes into the FAI. I call the automated design choices “preference”, and the mirror (theory of mirror) “decision theory”, with the slot “notion of preference” that is to be filled in automatically. So, there is no question of which one of “decision theory” and “preference” is “essential”, both play a role. The worry is about the necessary size of the manually designed “decision theory” part, and whether it’s humanly possible to construct it.