Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability. A human would also think that such a device is unlikely, but not infinitely so. (I gave a version of this argument in
is induction unformalizable?, which is linked to from Berry’s Paradox and universal distribution. Did you read it?)
Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability.
What do you mean by “decide the truth value”? Most statements aren’t valid or unsatisfiable, there is no truth value for them. We are not assuming any models here, just assigning plausibility to (statement) elements of language’s Lindenbaum algebra.
Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability.
Whatever model you have in mind, it will be categorized on one side of each statement of the language. We are assigning plausibility to statements, and hence classes of structures, not individual structures (which are like individual points for a continuous distribution).
Vladimir, ever since I joined this site I’ve been hearing many interesting not-quite-formal ideas from you, and as my understanding grows I can parse more and more of what you say. But you always seem to move on to the next idea before finishing the last one. I think you should spend way more effort on transforming your ideas into actual theorems with proofs and publishing them online. Sharing “intuitions” only gets us so far.
I have much less trouble reading math papers from unfamiliar fields than reading your informal arguments, because your arguments rely on unstated background assumptions much more than you seem to realize. Properly preparing your results for publication, even if they don’t get actually published somewhere peer-reviewed, should fix this problem.
I discuss things here because it’s fun (and sometimes I learn useful lessons from expressing them here, in addition to my private notes), not because I consider it effective means of communication. The not-quite-formal ideas are most of the time in fact not-quite-formal, rather than informally communicated formal ideas (often because I don’t understand the relevant math, a failure I’m working on). The dropped ideas are those I either found useless/meaningless/wrong or those that never came up in the discussion after some point.
Communicating informal ideas is too difficult, specifically because they assume tons of unstated background, background that you not only have to state, but convince people about. This is work both for the writer and for the reader. In addition, these informal ideas are not particularly valuable, which together with difficulty of communication makes the whole endeavor a waste of effort.
(At least on LW, common background gives a chance for some remarks to be understood, without that background having to be delivered explicitly.)
The plan is for all these hunches to eventually come together in a framework for decision theory, that should be transparently mathematical, and thus allow efficient little-hidden-background communication.
I’m afraid I still don’t quite understand your idea. Can you explain it a bit more?
For example, suppose I come across a black box that takes a string as input and outputs a 0 or 1. What does your idea say is the probability that it’s a halt-problem oracle, or a device that gives the truth value of statements in ZFC?
Or suppose I’m playing a game where I’ve been given a long string of bits and have to bet on the next one in the sequence. How do I use your idea to decide what to do?
(Feel free to pick your own examples if the above ones are not optimal for explaining your idea.)
What’s ambiguous with the definition? For example, unsatisfiable statements will get about as big plausibility as the valid ones, and for theories that are not finitely axiomatizable, plausibility is not defined (so you can’t ask about plausibility of some models, unless there is a categorical finite theory defining them). How to use this in decision-making is a special case of a more general open problem in ambient control.
Part of what confuses me is that you said we’re assigning plausibility to classes of structures, not individual structures, but it seems like we’d need to assign plausibility to individual structures in practice.
How to use this in decision-making is a special case of a more general open problem in ambient control.
Can’t you give an example using a situation where Bayesian updating is non-problematic, and just show how we might use your idea for the prior with standard decision theory?
If you can refer to an individual structure informally, then either there is a language that allows finitely describing it, or ability to refer to that structure is an illusion and in fact you are only referring to some bigger collection of things (property) of which the object you talk about is an element. If you can’t refer to a structure, then you don’t need plausibility for it.
Can’t you give an example using a situation where Bayesian updating is non-problematic, and just show how we might use your idea for the prior with standard decision theory?
This is only helpful is something works with tricky mathematical structures, and in all cases that seems to need to be preference. For example, you’d prefer to make decisions that are (likely!) consistent with a given theory (make it hold), then it helps if your decision and that theory are expressed in the same setting (language), and you can make decisions under logical uncertainty if you use the universal prior on statements. Normally, decision theories don’t consider such cases, so I’m not sure how to relate. Introducing observations will probably be a mistake too.
either there is a language that allows finitely describing it
But if you fix a language L for your universal prior, then there will be a more powerful metalanguage L’ that allows finitely describing some structure, which can’t be finitely described in the base language, right? So don’t we still have the problem of the universal prior not really being universal?
I can’t parse the second part of your response. Will keep trying...
But if you fix a language L for your universal prior, then there will be a more powerful metalanguage L’ that allows finitely describing some structure, which can’t be finitely described in the base language, right? So don’t we still have the problem of the universal prior not really being universal?
It can still talk about all structures, but sometimes won’t be able to point at a specific structure, only a class containing it. You only need a language expressive enough to describe everything preference refers to, and no more. (This seems to be the correct solution to ontology problem—describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.)
(Clarified the second part of the previous comment a bit.)
You only need a language expressive enough to describe everything preference refers to, and no more.
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
This seems to be the correct solution to ontology problem—describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
ETA: And sometimes I suspect even “what is the right utility function?” is really more like “what is the right decision theory?” than we currently believe. In other words there is objective morality after all, but we’re currently just too stupid or philosophically incompetent to figure out what it is.
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
The general idea seems right. If the existing languages are inadequate, they at least seem adequate for a full-featured prototype: figure out decision theory (and hence notion of preference) in terms of standard logic, then move on as necessary for extending expressive power. This should stop at some point, since this exercise at formality is aimed at construction of a program.
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
I don’t see clearly the distinction you’re making, so let me describe how I see it. Some design choices in constructing FAI would certainly be specific to our minds (values), but the main assumption to my approach to FAI is exactly that a large portion of design choices in FAI can be specified as a natural category in human brains, something we can point a simple mirror at and say “there!”, with the mirror doing most of the work in determining what goes into the FAI. I call the automated design choices “preference”, and the mirror (theory of mirror) “decision theory”, with the slot “notion of preference” that is to be filled in automatically. So, there is no question of which one of “decision theory” and “preference” is “essential”, both play a role. The worry is about the necessary size of the manually designed “decision theory” part, and whether it’s humanly possible to construct it.
Re: “Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability.”
It would never see the infinite description with the 0 probability, though—not enough space-time.
The evidence of the Oracle that the agent would get to see would be in the form of finite sensory inputs—and those would not be assigned zero probability. So: it could update on that evidence just fine—with no problems.
If the agent sees a tiny box with an Oracle inside it, that is just more finite sense-data about the state of the universe to update on—no problem—and no silly p=0 for an empirical observation.
Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability. A human would also think that such a device is unlikely, but not infinitely so. (I gave a version of this argument in is induction unformalizable?, which is linked to from Berry’s Paradox and universal distribution. Did you read it?)
What do you mean by “decide the truth value”? Most statements aren’t valid or unsatisfiable, there is no truth value for them. We are not assuming any models here, just assigning plausibility to (statement) elements of language’s Lindenbaum algebra.
Whatever model you have in mind, it will be categorized on one side of each statement of the language. We are assigning plausibility to statements, and hence classes of structures, not individual structures (which are like individual points for a continuous distribution).
Vladimir, ever since I joined this site I’ve been hearing many interesting not-quite-formal ideas from you, and as my understanding grows I can parse more and more of what you say. But you always seem to move on to the next idea before finishing the last one. I think you should spend way more effort on transforming your ideas into actual theorems with proofs and publishing them online. Sharing “intuitions” only gets us so far.
I have much less trouble reading math papers from unfamiliar fields than reading your informal arguments, because your arguments rely on unstated background assumptions much more than you seem to realize. Properly preparing your results for publication, even if they don’t get actually published somewhere peer-reviewed, should fix this problem.
I discuss things here because it’s fun (and sometimes I learn useful lessons from expressing them here, in addition to my private notes), not because I consider it effective means of communication. The not-quite-formal ideas are most of the time in fact not-quite-formal, rather than informally communicated formal ideas (often because I don’t understand the relevant math, a failure I’m working on). The dropped ideas are those I either found useless/meaningless/wrong or those that never came up in the discussion after some point.
Communicating informal ideas is too difficult, specifically because they assume tons of unstated background, background that you not only have to state, but convince people about. This is work both for the writer and for the reader. In addition, these informal ideas are not particularly valuable, which together with difficulty of communication makes the whole endeavor a waste of effort.
(At least on LW, common background gives a chance for some remarks to be understood, without that background having to be delivered explicitly.)
The plan is for all these hunches to eventually come together in a framework for decision theory, that should be transparently mathematical, and thus allow efficient little-hidden-background communication.
I’m afraid I still don’t quite understand your idea. Can you explain it a bit more?
For example, suppose I come across a black box that takes a string as input and outputs a 0 or 1. What does your idea say is the probability that it’s a halt-problem oracle, or a device that gives the truth value of statements in ZFC?
Or suppose I’m playing a game where I’ve been given a long string of bits and have to bet on the next one in the sequence. How do I use your idea to decide what to do?
(Feel free to pick your own examples if the above ones are not optimal for explaining your idea.)
What’s ambiguous with the definition? For example, unsatisfiable statements will get about as big plausibility as the valid ones, and for theories that are not finitely axiomatizable, plausibility is not defined (so you can’t ask about plausibility of some models, unless there is a categorical finite theory defining them). How to use this in decision-making is a special case of a more general open problem in ambient control.
Part of what confuses me is that you said we’re assigning plausibility to classes of structures, not individual structures, but it seems like we’d need to assign plausibility to individual structures in practice.
Can’t you give an example using a situation where Bayesian updating is non-problematic, and just show how we might use your idea for the prior with standard decision theory?
If you can refer to an individual structure informally, then either there is a language that allows finitely describing it, or ability to refer to that structure is an illusion and in fact you are only referring to some bigger collection of things (property) of which the object you talk about is an element. If you can’t refer to a structure, then you don’t need plausibility for it.
This is only helpful is something works with tricky mathematical structures, and in all cases that seems to need to be preference. For example, you’d prefer to make decisions that are (likely!) consistent with a given theory (make it hold), then it helps if your decision and that theory are expressed in the same setting (language), and you can make decisions under logical uncertainty if you use the universal prior on statements. Normally, decision theories don’t consider such cases, so I’m not sure how to relate. Introducing observations will probably be a mistake too.
But if you fix a language L for your universal prior, then there will be a more powerful metalanguage L’ that allows finitely describing some structure, which can’t be finitely described in the base language, right? So don’t we still have the problem of the universal prior not really being universal?
I can’t parse the second part of your response. Will keep trying...
It can still talk about all structures, but sometimes won’t be able to point at a specific structure, only a class containing it. You only need a language expressive enough to describe everything preference refers to, and no more. (This seems to be the correct solution to ontology problem—describe preference as being about mathematical structures (more generally, concepts/theories), and ignore the question of the nature of reality.)
(Clarified the second part of the previous comment a bit.)
Why do you think that any logical language (of the sort we’re currently familiar with) is sufficiently expressive for this purpose?
I’m not sure. One way to think about it is whether the question “what is the right prior?” is more like “what is the right decision theory?” or more like “what is the right utility function?” In What Are Probabilities, Anyway? I essentially said that I lean towards the latter, but I’m highly uncertain.
ETA: And sometimes I suspect even “what is the right utility function?” is really more like “what is the right decision theory?” than we currently believe. In other words there is objective morality after all, but we’re currently just too stupid or philosophically incompetent to figure out what it is.
The general idea seems right. If the existing languages are inadequate, they at least seem adequate for a full-featured prototype: figure out decision theory (and hence notion of preference) in terms of standard logic, then move on as necessary for extending expressive power. This should stop at some point, since this exercise at formality is aimed at construction of a program.
I don’t see clearly the distinction you’re making, so let me describe how I see it. Some design choices in constructing FAI would certainly be specific to our minds (values), but the main assumption to my approach to FAI is exactly that a large portion of design choices in FAI can be specified as a natural category in human brains, something we can point a simple mirror at and say “there!”, with the mirror doing most of the work in determining what goes into the FAI. I call the automated design choices “preference”, and the mirror (theory of mirror) “decision theory”, with the slot “notion of preference” that is to be filled in automatically. So, there is no question of which one of “decision theory” and “preference” is “essential”, both play a role. The worry is about the necessary size of the manually designed “decision theory” part, and whether it’s humanly possible to construct it.
Ok, I think I had misinterpreted your previous comment. I’ll have to think over your idea.
Maybe the human is a bad philosopher in this case and is simply wrong.
Re: “Consider a universal prior based on an arbitrary logical language L, and a device that can decide the truth value of any sentence in that language. Such a device has no finite description in L (according to Tarski’s undefinability theorem), so the universal prior based on L would assign it zero probability.”
It would never see the infinite description with the 0 probability, though—not enough space-time.
The evidence of the Oracle that the agent would get to see would be in the form of finite sensory inputs—and those would not be assigned zero probability. So: it could update on that evidence just fine—with no problems.
If the agent sees a tiny box with an Oracle inside it, that is just more finite sense-data about the state of the universe to update on—no problem—and no silly p=0 for an empirical observation.