First, this is a great, really thoughtful comment.
My initial thought is that I’m doing a bad job in this post of pointing at the thing that really matters, which is why I think we may see different framings as useful. I see the problem of the criterion as a particular instance of exposing a deeper, fundamental meta-fact about our world, which is that uncertainty is fundamental to our existence and all knowledge is teleological. I don’t think all that comes through here because when I wrote this it was my first time really trying to explain fundamental uncertainty, so I only go so far in explaining it. I got a bit too excited a confused the problem of the criterion for fundamental uncertainty itself.
That (1) seems boring to Bayesians et al. seems fine to me because they already buy that things are fundamentally uncertain, although I think in practice most aspiring Bayesians don’t really know why, which bleeds into (2). I see (2) as important because knowledge does, in fact, ground out in some foundation, and we try hard to make sure that grounding is correct by requiring justifications for our grounding. Personally, the question I’m most interested in is (3), and then considering what implications that has for (1), (2), and (4), though I don’t really have much to say about (4) here other than that one should be fundamentally uncertain and that I think in practice many people trying to work out (4) get confused because they imagine they can achieve justified true belief, or more often with aspiring Bayesians justified true meta-belief even though all object-level beliefs are uncertain.
So my I have to admit my motivations when writing this post were a bit of a mix of yelling at rationalists I meet who are performing rationality but missing this deeper thing that I was also missing out on for a long time and trying to really clarify what this deeper thing is that’s going on.
I know you’re reading some of my other posts, but let me see if I can explain how I actually think about the problem of the criterion today:
We and every dynamic process can be thought of as made up of lots of feedback circuits, and most of those circuits are negative feedback circuits (positive feedback, where it exists, is often wrapped inside a negative feedback circuit because otherwise the universe would be boring and already tiled with something like paperclips). These circuits get information by observing the world and then generating a signal to control the circuit. The process of generating the signal is necessarily lossy, and it’s because we’re going to compress information when generating the signal. As a result, the circuit makes a “choice” about what to signal on, the signal is thus teleological because the signal depends on what purpose the signal has (purpose is an idea in cybernetics I won’t expand here, but basically it’s what the circuit is designed to do), so all the knowledge (information inside the circuits) we have is both uncertain due to compression (and maybe something something quantum mechanics observation, but I don’t think we need to go there) and teleological because circuits impose teleology through their design.
I’ll admit, this is all sort of a weird way to explain it because it hinges on this unusual cybernetic model of the world, so I try to explain myself in other ways that are more familiar to folks. Maybe that’s the wrong approach, but I think it’s reasonable to come at this thing outside in (which is what I do in my book) by addressing the surface level uncertainty first and then peeling back the layers to find the fundamental uncertainty. This post is weird because it’s a bit in the middle.
(I’ll also note there’s an even deeper version of how I think about fundamental uncertainty and teleology today, but I’m not sure I can put it in words because it comes from meditation experience and I can only talk about it in mysterious ways. I think trying to talk about that way of thinking about it is probably more confusing than helpful.)
IMHO the cybernetic picture isn’t weird or uncommon; naively, I expect it to get less pushback.
I see (2) as important because knowledge does, in fact, ground out in some foundation, and we try hard to make sure that grounding is correct by requiring justifications for our grounding.
I think this is what I most want to push back on. My own sense is that you are confused about this. On my understanding, you seem to simultaneously believe that the core foundationalist assumptions make sense, and also believe an impossibility argument which shows them to be inconsistent. This doesn’t make sense to me.
My formalization here is only one possible way to understand the infinite-regress problem (although I think it does a good job of capturing the essence of it) -- but, in this formalization, the contradiction is really direct, which makes it seem pretty silly.
I also think the contradictory axioms do capture two intuitions which, like, beginning philosophy majors might endorse.
So I think the infinite regress problem should be explained to beginning philosophers as a warning against these naive assumptions about justification. (And indeed, this is how I was taught.)
But that’s what it is to me. It seems to be something else for you. Like a paradox. You write of proving the impossibility of solution, rather than resolving the problem. You write that we should “hold the problem”. Like, in some sense it is still a problem even after it has been solved.
(Perhaps the seeming contradiction is merely due to the way the criterion problem conflates multiple problems; EG, the naturalistic question of where knowledge comes from is still a live question after the justification-infinite-regress problem has been resolved.)
It makes sense to me to try to spell out the consequences of the infinite-regress problem; they may be complex and non-obvious. But for me this should involve questioning the naive assumptions of justification, and figuring out what it is they were trying to do. On my analysis, a reasonable place to go from there is the tiling agents problem and Vingean reflection. This is a more sophisticated picture of the problems rational agents run into when philosophizing about themselves, because it admits that you don’t need to already know how you know in order to know—you are already whatever sort of machine you are, and the default outcome is that you keep running as you’ve run. You’re not at risk of all your knowledge evaporating if you can’t justify it. However, there is a big problem of self-trust, and how you can achieve things over time by cooperating with yourself. And there’s an even bigger problem if you do have the ability to self-modify; then your knowledge actually might evaporate if you can’t justify it.
But this problem is to a large degree solved by the lack of a probabilistic Lob’s theorem. This means we are in a much better situation with respect to self-trust, vingean reflection, and the tiling agents problem—so long as we can accept probabilistic fallibility rather than needing perfect self-trust.
So in effect, the answer to (a newer and significantly more sophisticated version of) the recursive justification problem is the same as the answer to skepticism which you agreed was right and proper—namely, embracing probabilism rather than requiring certainty.
I expect you’ll agree with this conclusion; the part where I think we have some sort of fuzzy disagreement is the part where you still say things like “I see (2) as important because knowledge does, in fact, ground out in some sort of foundation, and we try to make sure that grounding is correct by requiring justifications for our grounding.” I think there are some different possible ways to spell out what you meant here, but I don’t think the best ones connect with (2) very much, since (2) is based on some silly-in-hindsight intuitions about justification.
So I think maybe there’s something goin on here where I’m taking too much for granted that formal systems are the best way to figure out what to believe about the world to get an accurate picture of it, and so long as you’re figuring out what to believe using some formal system then you’re forced to ground it in some assumed foundation that is not itself justified. This seems important because it means there’s a lot of interesting stuff going on with those assumptions and how they get chosen such that they cause the rest of the system to produce beliefs that track reality.
But it sounds like you think this is a confused view. I admit I feel a bit differently about things than the way they’re presented in the post, but not entirely, as my comment reflects. I think when I wrote the post I really couldn’t see my way around how to know things without being able to formally justify them. Now I better understand there can be knowing without belief that knowledge is justified or true, but I do think there’s something to the idea that I am still too much captured by my own ontology and not fully seeing it as I see through it, and this causes me to get a bit mixed up in places.
The bit about self-trust is really interesting. To get a little personal and psychological, I think the big difference between me ~4 years ago and me now is that ~4 years ago I learned to trust myself in some big important way that goes deeper than surface level self-trust. On the surface this looks like trusting my feelings and experiences, but to a certain extent anyone can do this if you just give them feedback that says trust those things more than other things. But I’m talking about something a level deeper where I trust the process by which those things get generated to do what they do and that they’re doing their best (in the sense that the universe is deterministic and we’re all always doing our best/worst because it could have been no other way).
If I’m honest I’ve really struggled to understand what’s going on with Lob’s theorem and why it seems like a big deal to everyone. But it seems like from what you’re saying it’s related to the issue that got me started on things, which is how do you deal with the problem of verifying that another agent (which could be yourself at another time) shares your values and is thus in some way aligned with you. If that’s the case maybe I already grasp the fundamentals of this obstacle Lob seems to create and have just failed to realize that’s what folks were talking about when they talk about Lob with respect to aligning AI.
@abramdemski Wanted to say thanks again for engaging with my posts and pointing me towards looking again at Lob. It’s weird: now that I’ve taken so time to understand it, it’s just what in my mind was already the thing going on with Godel, just I wasn’t doing a great job of separating out what Godel proves and what the implications are. As presented on its own, Lob didn’t seem that interesting to me so I kept bouncing off it as something worth looking at, but now I realize it’s just the same thing I learned from GEB’s presentation of Peano arithmetic and Godel when I read it 20+ years ago.
When I go back to make revisions to the book, I’ll have to reconsider including Godel and Lob somehow in the text. I didn’t because I felt like it was a bit complicated and I didn’t really need to dig into it since I think there’s already a bit too many cases where people use Godel to overreach and draw conclusions that aren’t true, but it’s another way to explain these ideas. I just have to think about if Godel and Lob are necessary: that is, do I need to appeal to them to make my key points, or are these things that are better left as additional topics I can point folks at but not key to understanding the intuitions I want them to develop.
I’ve heard Lob remarked that he would never have published if he realized earlier how close his theorem was to just Godel’s second incompleteness theorem; but I can’t seem to entirely agree with Lob there. It does seem like a valuable statement of its own.
I agree, Godel is dangerously over-used, so the key question is whether it’s necessary here. Other formal analogs of your point include Tarski’s undefinability, and the realizablility / grain-of-truth problem. There are many ways to gesture towards a sense of “fundamental uncertainty”, so the question is: what statement of the thing do you want to make most central, and how do you want to argue/illustrate that statement?
First, this is a great, really thoughtful comment.
My initial thought is that I’m doing a bad job in this post of pointing at the thing that really matters, which is why I think we may see different framings as useful. I see the problem of the criterion as a particular instance of exposing a deeper, fundamental meta-fact about our world, which is that uncertainty is fundamental to our existence and all knowledge is teleological. I don’t think all that comes through here because when I wrote this it was my first time really trying to explain fundamental uncertainty, so I only go so far in explaining it. I got a bit too excited a confused the problem of the criterion for fundamental uncertainty itself.
That (1) seems boring to Bayesians et al. seems fine to me because they already buy that things are fundamentally uncertain, although I think in practice most aspiring Bayesians don’t really know why, which bleeds into (2). I see (2) as important because knowledge does, in fact, ground out in some foundation, and we try hard to make sure that grounding is correct by requiring justifications for our grounding. Personally, the question I’m most interested in is (3), and then considering what implications that has for (1), (2), and (4), though I don’t really have much to say about (4) here other than that one should be fundamentally uncertain and that I think in practice many people trying to work out (4) get confused because they imagine they can achieve justified true belief, or more often with aspiring Bayesians justified true meta-belief even though all object-level beliefs are uncertain.
So my I have to admit my motivations when writing this post were a bit of a mix of yelling at rationalists I meet who are performing rationality but missing this deeper thing that I was also missing out on for a long time and trying to really clarify what this deeper thing is that’s going on.
I know you’re reading some of my other posts, but let me see if I can explain how I actually think about the problem of the criterion today:
We and every dynamic process can be thought of as made up of lots of feedback circuits, and most of those circuits are negative feedback circuits (positive feedback, where it exists, is often wrapped inside a negative feedback circuit because otherwise the universe would be boring and already tiled with something like paperclips). These circuits get information by observing the world and then generating a signal to control the circuit. The process of generating the signal is necessarily lossy, and it’s because we’re going to compress information when generating the signal. As a result, the circuit makes a “choice” about what to signal on, the signal is thus teleological because the signal depends on what purpose the signal has (purpose is an idea in cybernetics I won’t expand here, but basically it’s what the circuit is designed to do), so all the knowledge (information inside the circuits) we have is both uncertain due to compression (and maybe something something quantum mechanics observation, but I don’t think we need to go there) and teleological because circuits impose teleology through their design.
I’ll admit, this is all sort of a weird way to explain it because it hinges on this unusual cybernetic model of the world, so I try to explain myself in other ways that are more familiar to folks. Maybe that’s the wrong approach, but I think it’s reasonable to come at this thing outside in (which is what I do in my book) by addressing the surface level uncertainty first and then peeling back the layers to find the fundamental uncertainty. This post is weird because it’s a bit in the middle.
(I’ll also note there’s an even deeper version of how I think about fundamental uncertainty and teleology today, but I’m not sure I can put it in words because it comes from meditation experience and I can only talk about it in mysterious ways. I think trying to talk about that way of thinking about it is probably more confusing than helpful.)
IMHO the cybernetic picture isn’t weird or uncommon; naively, I expect it to get less pushback.
I think this is what I most want to push back on. My own sense is that you are confused about this. On my understanding, you seem to simultaneously believe that the core foundationalist assumptions make sense, and also believe an impossibility argument which shows them to be inconsistent. This doesn’t make sense to me.
My formalization here is only one possible way to understand the infinite-regress problem (although I think it does a good job of capturing the essence of it) -- but, in this formalization, the contradiction is really direct, which makes it seem pretty silly.
I also think the contradictory axioms do capture two intuitions which, like, beginning philosophy majors might endorse.
So I think the infinite regress problem should be explained to beginning philosophers as a warning against these naive assumptions about justification. (And indeed, this is how I was taught.)
But that’s what it is to me. It seems to be something else for you. Like a paradox. You write of proving the impossibility of solution, rather than resolving the problem. You write that we should “hold the problem”. Like, in some sense it is still a problem even after it has been solved.
(Perhaps the seeming contradiction is merely due to the way the criterion problem conflates multiple problems; EG, the naturalistic question of where knowledge comes from is still a live question after the justification-infinite-regress problem has been resolved.)
It makes sense to me to try to spell out the consequences of the infinite-regress problem; they may be complex and non-obvious. But for me this should involve questioning the naive assumptions of justification, and figuring out what it is they were trying to do. On my analysis, a reasonable place to go from there is the tiling agents problem and Vingean reflection. This is a more sophisticated picture of the problems rational agents run into when philosophizing about themselves, because it admits that you don’t need to already know how you know in order to know—you are already whatever sort of machine you are, and the default outcome is that you keep running as you’ve run. You’re not at risk of all your knowledge evaporating if you can’t justify it. However, there is a big problem of self-trust, and how you can achieve things over time by cooperating with yourself. And there’s an even bigger problem if you do have the ability to self-modify; then your knowledge actually might evaporate if you can’t justify it.
But this problem is to a large degree solved by the lack of a probabilistic Lob’s theorem. This means we are in a much better situation with respect to self-trust, vingean reflection, and the tiling agents problem—so long as we can accept probabilistic fallibility rather than needing perfect self-trust.
So in effect, the answer to (a newer and significantly more sophisticated version of) the recursive justification problem is the same as the answer to skepticism which you agreed was right and proper—namely, embracing probabilism rather than requiring certainty.
I expect you’ll agree with this conclusion; the part where I think we have some sort of fuzzy disagreement is the part where you still say things like “I see (2) as important because knowledge does, in fact, ground out in some sort of foundation, and we try to make sure that grounding is correct by requiring justifications for our grounding.” I think there are some different possible ways to spell out what you meant here, but I don’t think the best ones connect with (2) very much, since (2) is based on some silly-in-hindsight intuitions about justification.
Hmm, I’ll have to think about this.
So I think maybe there’s something goin on here where I’m taking too much for granted that formal systems are the best way to figure out what to believe about the world to get an accurate picture of it, and so long as you’re figuring out what to believe using some formal system then you’re forced to ground it in some assumed foundation that is not itself justified. This seems important because it means there’s a lot of interesting stuff going on with those assumptions and how they get chosen such that they cause the rest of the system to produce beliefs that track reality.
But it sounds like you think this is a confused view. I admit I feel a bit differently about things than the way they’re presented in the post, but not entirely, as my comment reflects. I think when I wrote the post I really couldn’t see my way around how to know things without being able to formally justify them. Now I better understand there can be knowing without belief that knowledge is justified or true, but I do think there’s something to the idea that I am still too much captured by my own ontology and not fully seeing it as I see through it, and this causes me to get a bit mixed up in places.
The bit about self-trust is really interesting. To get a little personal and psychological, I think the big difference between me ~4 years ago and me now is that ~4 years ago I learned to trust myself in some big important way that goes deeper than surface level self-trust. On the surface this looks like trusting my feelings and experiences, but to a certain extent anyone can do this if you just give them feedback that says trust those things more than other things. But I’m talking about something a level deeper where I trust the process by which those things get generated to do what they do and that they’re doing their best (in the sense that the universe is deterministic and we’re all always doing our best/worst because it could have been no other way).
If I’m honest I’ve really struggled to understand what’s going on with Lob’s theorem and why it seems like a big deal to everyone. But it seems like from what you’re saying it’s related to the issue that got me started on things, which is how do you deal with the problem of verifying that another agent (which could be yourself at another time) shares your values and is thus in some way aligned with you. If that’s the case maybe I already grasp the fundamentals of this obstacle Lob seems to create and have just failed to realize that’s what folks were talking about when they talk about Lob with respect to aligning AI.
@abramdemski Wanted to say thanks again for engaging with my posts and pointing me towards looking again at Lob. It’s weird: now that I’ve taken so time to understand it, it’s just what in my mind was already the thing going on with Godel, just I wasn’t doing a great job of separating out what Godel proves and what the implications are. As presented on its own, Lob didn’t seem that interesting to me so I kept bouncing off it as something worth looking at, but now I realize it’s just the same thing I learned from GEB’s presentation of Peano arithmetic and Godel when I read it 20+ years ago.
When I go back to make revisions to the book, I’ll have to reconsider including Godel and Lob somehow in the text. I didn’t because I felt like it was a bit complicated and I didn’t really need to dig into it since I think there’s already a bit too many cases where people use Godel to overreach and draw conclusions that aren’t true, but it’s another way to explain these ideas. I just have to think about if Godel and Lob are necessary: that is, do I need to appeal to them to make my key points, or are these things that are better left as additional topics I can point folks at but not key to understanding the intuitions I want them to develop.
I’ve heard Lob remarked that he would never have published if he realized earlier how close his theorem was to just Godel’s second incompleteness theorem; but I can’t seem to entirely agree with Lob there. It does seem like a valuable statement of its own.
I agree, Godel is dangerously over-used, so the key question is whether it’s necessary here. Other formal analogs of your point include Tarski’s undefinability, and the realizablility / grain-of-truth problem. There are many ways to gesture towards a sense of “fundamental uncertainty”, so the question is: what statement of the thing do you want to make most central, and how do you want to argue/illustrate that statement?