we have an updating process which can change its mind about any particular thing; and that updating process itself is not the ground truth, but rather has beliefs (which can change) about what makes an updating process legitimate.
This should still be a strong formal theory, but one which requires weaker assumptions than usual
There seems to be a bit of a tension here. What you’re outlining for most of the post still requires a formal system with assumptions within which to take the fixed point, but then that would mean that it can’t change its mind about any particular thing. Indeed it’s not clear how such a totally self-revising system could ever be a fixed point of constraints of rationality: since it can revise anything, it could only be limited by the physically possible.
This is most directly an approach for solving meta-philosophy.
Related to the last point, your project would seem to imply some interesting semantics. Usually our semantics are based on the correspondence theory: we start thinking about what the world can be like, and then expressions get their meaning from their relations to these ways the world can be like, particularly through our use. (We can already see how this would take us down a descriptive path.) This leads to problems where you can’t explain the content what confused beliefs. Like for example most children believe for some time that their languages words for things are the real words for them. If you’re human you propably understand what I was talking about, but if some alien was puzzled about what the “real” there was supposed to mean, I don’t think I could explain it. Basically once you’ve unconfused yourself, you become unable to say what you believed.
Now if we’re foundationalists, we say that thats because you didn’t actually believe anything, and that that was just a linguistic token passed around your head but failing to be meaningful, because you didn’t implement The Laws correctly. But if we want to have a theory like yours, it treats this cognitively, and so such beliefs must meaningful in some sense. I’m very curious what this would look like.
More generally this is a massively ambitious undertaking. If you succeeded it would solve a bunch of other issues, not even from running it but just from the conceptual understanding of how it would work. For example in your last post on signalling you mentioned:
First of all, it might be difficult to define the hypothetical scenario in which all interests are aligned, so that communication is honest. Taking an extreme example, how would we then assign meaning to statements such as “our interests are not aligned”?
I think a large part of embedded agency has a similar problem, where we try to build our semantics on “If I was there, I would think”, and apply this to scenarios where we are essentially not there, because we’re thinking about our non-existence, or about bugs that would make us not think that way, or some such. So if you solved this it would propably just solve anthropics as well. On the one hand this is exciting, on the other its a reason to be sceptical. And all of this eerily reminds me of German Idealism. In any case I think this is very good as a post.
There seems to be a bit of a tension here. What you’re outlining for most of the post still requires a formal system with assumptions within which to take the fixed point, but then that would mean that it can’t change its mind about any particular thing. Indeed it’s not clear how such a totally self-revising system could ever be a fixed point of constraints of rationality: since it can revise anything, it could only be limited by the physically possible.
It’s sort of like the difference between a programmable computer vs an arbitrary blob of matter. A programmable computer provides a rigid structure which can’t be changed, but the set of assumptions imposed really is quite light. When programming language designers aim for “totally self-revising systems” (languages with more flexibility in their assumptions, such as Lisp), they don’t generally attack the assumption that the hardware should be fixed. (Although occasionally they do go as far as asking for FPGAs.)
(a finite approximation of) Solomonoff Induction can be said to make “very few assumptions”, because it can learn a wide variety of programs. Certainly it makes less assumptions than more special-case machine learning systems. But it also makes a lot more assumptions than the raw computer. In particular, it has no allowance for updating against the use of Bayes’ Rule for evaluating which program is best.
I’m aiming for something between the Solomonoff induction and the programmable computer. It can still have a rigid learning system underlying it, but in some sense it can learn any particular way of selecting hypotheses, rather than being stuck with one.
Now if we’re foundationalists, we say that thats because you didn’t actually believe anything, and that that was just a linguistic token passed around your head but failing to be meaningful, because you didn’t implement The Laws correctly. But if we want to have a theory like yours, it treats this cognitively, and so such beliefs must meaningful in some sense. I’m very curious what this would look like.
This seems like a rather excellent question which demonstrates a high degree of understanding of the proposal.
I think the answer from my not-necessarily-foundationalist but not-quite-pluralist perspective (a pluralist being someone who points to the alternative foundations proposed by different people and says “these are all tools in a well-equipped toolbox”) is:
The meaning of a confused concept such as “the real word for X” is not ultimately given by any rigid formula, but rather, established by long deliberation on what it can be understood to mean. However, we can understand a lot of meaning through use. Pragmatically, what “the real word for X” seems to express is that there is a correct thing to call something, usually uniquely determined, which can be discovered through investigation (EG by asking parents). This implies that other terms are incorrect (EG other languages, or made-up terms). “Incorrect” here means normatively incorrect, which is still part of our map; but to cash out what that means to a greater degree, it means you can EG scold people who use wrong terms, and you should teach them better terms, etc.
To sum up, meaning in this view is broadly more inferentialist and less correspondence-based: the meaning of a thing is more closely tied with the inferences around that thing, than with how that thing corresponds to a territory.
So if you solved this it would propably just solve anthropics as well.
I’m not seeing that implication at all! The way I see it, the framework “stands over” decision-theoretic issues such as anthorpics, providing no answers (only providing an epistemic arena in which uncertainty about them can be validly expressed rather than requiring some solution in order to define correct reasoning in the first place).
I now think you were right about it not solving anthropics. I interpreted afoundationalism insufficiently ambitiously; now that I have a proof-of-concept for normative semantics I can indeed not find it to presuppose an anthropics.
It’s sort of like the difference between a programmable computer vs an arbitrary blob of matter.
This is close to what I meant: My neurons keep doing something-like reinforcement learning, whether or not I theoretically believe thats valid. “I in fact can not think outside this” does adress the worry about a merely rational constraint.
On the other hand, we do want AI to eventually consider other hardware, and that might even be necessary for normal embedded agency, since we dont fully trust our hardware even when we dont want to normal-sense-change it.
To sum up, meaning in this view is broadly more inferentialist and less correspondence-based: the meaning of a think is more closely tied with the inferences around that thing, than with how that thing corresponds to a territory.
I broadly agree with inferentialism, but I don’t think that entirely adresses it. The mark of confused, rather than merely wrong, beliefs is that they dont really have a coherent use. So for example it might be that theres a path through possible scenarios leading back to the starting point, where if at every step I adjust my reaction in a way that seems appropriate to me, I end up with a different reaction when I’m back at the start. If you tried to describe my practices here you would just explicitly account for the framing dependence. But then it wouldn’t be confused! That framing-dependent concept you described also exists, but it seems quite different from the confused one. For the confused concept its essential that I consider it not dependent in this way. But if you try to include that in your description, by also describing the practices around my meta-beliefs about the concept, and the meta-meta beliefs, and so on, then you’d end up also describing the process with which I recognized it as confused and revised it. And then we’re back in the position of already-having-recognized that its bullshit.
When you were only going up individual meta-levels, the propositions logical induction worked with could be meaningful even if they were wrong, because they were part of processes outside the logical induction process, and those were sufficient to give them truth-conditions. Now you want to determine both what to believe and how those beliefs are to be used in one go, and it’s undermining that, because the “how beliefs are to be used” is what foundationalism kept fixed, and which gave them their truth conditions.
I’m not seeing that implication at all!
Well, this is a bit analogy-y, but I’ll try to explain. I think theres a semantic issue with anthropics (indeed, under inferentialism all confusion can be expressed as a semantic issue). Things like “the propability that I will have existed if I do X now” are unclear. For example a descriptivist way of understanding conditional propabilities is something like “The token C means conditional propability iff whenever you believe xCy = p, then you would belief P(y) = p if you came to believe x”. But that assumes not only that you are logically perfect but that you are there to have beliefs and answer for them. Now most of the time it’s not a problem if you’re not actually there, because we can just ask about if you were there (and you somehow got oxygen and glucose despite not touching anything, and you could see without blocking photons, etc but lets ignore that for now), even if you aren’t actually. But this can be a problem with anthropic situations. Normally when a hypothetical involves you, you can just imagine it from your prespective, and when it doesn’t involve you, you can imagine you were there. But if you’re trying to imagine a scenario that involves you but you can’t imagine it from your prespective, because you come into existence in it, or you have a mental defect in it, or something, then you have to imagine it from the third person. So you’re not really thinking about yourself, you’re thinking about a copy, which may be in quite a different epistemic situation. So if you can conceptually explain how to have semantics that accounts for my making mistakes, then I think that would propably be able to account for my not being there as well (in both cases, it’s just the virtuous epistemic process missing). And that would tell us how to have anthropic beliefs, and that would unknot the area.
There seems to be a bit of a tension here. What you’re outlining for most of the post still requires a formal system with assumptions within which to take the fixed point, but then that would mean that it can’t change its mind about any particular thing. Indeed it’s not clear how such a totally self-revising system could ever be a fixed point of constraints of rationality: since it can revise anything, it could only be limited by the physically possible.
Related to the last point, your project would seem to imply some interesting semantics. Usually our semantics are based on the correspondence theory: we start thinking about what the world can be like, and then expressions get their meaning from their relations to these ways the world can be like, particularly through our use. (We can already see how this would take us down a descriptive path.) This leads to problems where you can’t explain the content what confused beliefs. Like for example most children believe for some time that their languages words for things are the real words for them. If you’re human you propably understand what I was talking about, but if some alien was puzzled about what the “real” there was supposed to mean, I don’t think I could explain it. Basically once you’ve unconfused yourself, you become unable to say what you believed.
Now if we’re foundationalists, we say that thats because you didn’t actually believe anything, and that that was just a linguistic token passed around your head but failing to be meaningful, because you didn’t implement The Laws correctly. But if we want to have a theory like yours, it treats this cognitively, and so such beliefs must meaningful in some sense. I’m very curious what this would look like.
More generally this is a massively ambitious undertaking. If you succeeded it would solve a bunch of other issues, not even from running it but just from the conceptual understanding of how it would work. For example in your last post on signalling you mentioned:
I think a large part of embedded agency has a similar problem, where we try to build our semantics on “If I was there, I would think”, and apply this to scenarios where we are essentially not there, because we’re thinking about our non-existence, or about bugs that would make us not think that way, or some such. So if you solved this it would propably just solve anthropics as well. On the one hand this is exciting, on the other its a reason to be sceptical. And all of this eerily reminds me of German Idealism. In any case I think this is very good as a post.
It’s sort of like the difference between a programmable computer vs an arbitrary blob of matter. A programmable computer provides a rigid structure which can’t be changed, but the set of assumptions imposed really is quite light. When programming language designers aim for “totally self-revising systems” (languages with more flexibility in their assumptions, such as Lisp), they don’t generally attack the assumption that the hardware should be fixed. (Although occasionally they do go as far as asking for FPGAs.)
(a finite approximation of) Solomonoff Induction can be said to make “very few assumptions”, because it can learn a wide variety of programs. Certainly it makes less assumptions than more special-case machine learning systems. But it also makes a lot more assumptions than the raw computer. In particular, it has no allowance for updating against the use of Bayes’ Rule for evaluating which program is best.
I’m aiming for something between the Solomonoff induction and the programmable computer. It can still have a rigid learning system underlying it, but in some sense it can learn any particular way of selecting hypotheses, rather than being stuck with one.
This seems like a rather excellent question which demonstrates a high degree of understanding of the proposal.
I think the answer from my not-necessarily-foundationalist but not-quite-pluralist perspective (a pluralist being someone who points to the alternative foundations proposed by different people and says “these are all tools in a well-equipped toolbox”) is:
The meaning of a confused concept such as “the real word for X” is not ultimately given by any rigid formula, but rather, established by long deliberation on what it can be understood to mean. However, we can understand a lot of meaning through use. Pragmatically, what “the real word for X” seems to express is that there is a correct thing to call something, usually uniquely determined, which can be discovered through investigation (EG by asking parents). This implies that other terms are incorrect (EG other languages, or made-up terms). “Incorrect” here means normatively incorrect, which is still part of our map; but to cash out what that means to a greater degree, it means you can EG scold people who use wrong terms, and you should teach them better terms, etc.
To sum up, meaning in this view is broadly more inferentialist and less correspondence-based: the meaning of a thing is more closely tied with the inferences around that thing, than with how that thing corresponds to a territory.
I’m not seeing that implication at all! The way I see it, the framework “stands over” decision-theoretic issues such as anthorpics, providing no answers (only providing an epistemic arena in which uncertainty about them can be validly expressed rather than requiring some solution in order to define correct reasoning in the first place).
I now think you were right about it not solving anthropics. I interpreted afoundationalism insufficiently ambitiously; now that I have a proof-of-concept for normative semantics I can indeed not find it to presuppose an anthropics.
This is close to what I meant: My neurons keep doing something-like reinforcement learning, whether or not I theoretically believe thats valid. “I in fact can not think outside this” does adress the worry about a merely rational constraint.
On the other hand, we do want AI to eventually consider other hardware, and that might even be necessary for normal embedded agency, since we dont fully trust our hardware even when we dont want to normal-sense-change it.
I broadly agree with inferentialism, but I don’t think that entirely adresses it. The mark of confused, rather than merely wrong, beliefs is that they dont really have a coherent use. So for example it might be that theres a path through possible scenarios leading back to the starting point, where if at every step I adjust my reaction in a way that seems appropriate to me, I end up with a different reaction when I’m back at the start. If you tried to describe my practices here you would just explicitly account for the framing dependence. But then it wouldn’t be confused! That framing-dependent concept you described also exists, but it seems quite different from the confused one. For the confused concept its essential that I consider it not dependent in this way. But if you try to include that in your description, by also describing the practices around my meta-beliefs about the concept, and the meta-meta beliefs, and so on, then you’d end up also describing the process with which I recognized it as confused and revised it. And then we’re back in the position of already-having-recognized that its bullshit.
When you were only going up individual meta-levels, the propositions logical induction worked with could be meaningful even if they were wrong, because they were part of processes outside the logical induction process, and those were sufficient to give them truth-conditions. Now you want to determine both what to believe and how those beliefs are to be used in one go, and it’s undermining that, because the “how beliefs are to be used” is what foundationalism kept fixed, and which gave them their truth conditions.
Well, this is a bit analogy-y, but I’ll try to explain. I think theres a semantic issue with anthropics (indeed, under inferentialism all confusion can be expressed as a semantic issue). Things like “the propability that I will have existed if I do X now” are unclear. For example a descriptivist way of understanding conditional propabilities is something like “The token C means conditional propability iff whenever you believe xCy = p, then you would belief P(y) = p if you came to believe x”. But that assumes not only that you are logically perfect but that you are there to have beliefs and answer for them. Now most of the time it’s not a problem if you’re not actually there, because we can just ask about if you were there (and you somehow got oxygen and glucose despite not touching anything, and you could see without blocking photons, etc but lets ignore that for now), even if you aren’t actually. But this can be a problem with anthropic situations. Normally when a hypothetical involves you, you can just imagine it from your prespective, and when it doesn’t involve you, you can imagine you were there. But if you’re trying to imagine a scenario that involves you but you can’t imagine it from your prespective, because you come into existence in it, or you have a mental defect in it, or something, then you have to imagine it from the third person. So you’re not really thinking about yourself, you’re thinking about a copy, which may be in quite a different epistemic situation. So if you can conceptually explain how to have semantics that accounts for my making mistakes, then I think that would propably be able to account for my not being there as well (in both cases, it’s just the virtuous epistemic process missing). And that would tell us how to have anthropic beliefs, and that would unknot the area.