IMHO the cybernetic picture isn’t weird or uncommon; naively, I expect it to get less pushback.
I see (2) as important because knowledge does, in fact, ground out in some foundation, and we try hard to make sure that grounding is correct by requiring justifications for our grounding.
I think this is what I most want to push back on. My own sense is that you are confused about this. On my understanding, you seem to simultaneously believe that the core foundationalist assumptions make sense, and also believe an impossibility argument which shows them to be inconsistent. This doesn’t make sense to me.
My formalization here is only one possible way to understand the infinite-regress problem (although I think it does a good job of capturing the essence of it) -- but, in this formalization, the contradiction is really direct, which makes it seem pretty silly.
I also think the contradictory axioms do capture two intuitions which, like, beginning philosophy majors might endorse.
So I think the infinite regress problem should be explained to beginning philosophers as a warning against these naive assumptions about justification. (And indeed, this is how I was taught.)
But that’s what it is to me. It seems to be something else for you. Like a paradox. You write of proving the impossibility of solution, rather than resolving the problem. You write that we should “hold the problem”. Like, in some sense it is still a problem even after it has been solved.
(Perhaps the seeming contradiction is merely due to the way the criterion problem conflates multiple problems; EG, the naturalistic question of where knowledge comes from is still a live question after the justification-infinite-regress problem has been resolved.)
It makes sense to me to try to spell out the consequences of the infinite-regress problem; they may be complex and non-obvious. But for me this should involve questioning the naive assumptions of justification, and figuring out what it is they were trying to do. On my analysis, a reasonable place to go from there is the tiling agents problem and Vingean reflection. This is a more sophisticated picture of the problems rational agents run into when philosophizing about themselves, because it admits that you don’t need to already know how you know in order to know—you are already whatever sort of machine you are, and the default outcome is that you keep running as you’ve run. You’re not at risk of all your knowledge evaporating if you can’t justify it. However, there is a big problem of self-trust, and how you can achieve things over time by cooperating with yourself. And there’s an even bigger problem if you do have the ability to self-modify; then your knowledge actually might evaporate if you can’t justify it.
But this problem is to a large degree solved by the lack of a probabilistic Lob’s theorem. This means we are in a much better situation with respect to self-trust, vingean reflection, and the tiling agents problem—so long as we can accept probabilistic fallibility rather than needing perfect self-trust.
So in effect, the answer to (a newer and significantly more sophisticated version of) the recursive justification problem is the same as the answer to skepticism which you agreed was right and proper—namely, embracing probabilism rather than requiring certainty.
I expect you’ll agree with this conclusion; the part where I think we have some sort of fuzzy disagreement is the part where you still say things like “I see (2) as important because knowledge does, in fact, ground out in some sort of foundation, and we try to make sure that grounding is correct by requiring justifications for our grounding.” I think there are some different possible ways to spell out what you meant here, but I don’t think the best ones connect with (2) very much, since (2) is based on some silly-in-hindsight intuitions about justification.
So I think maybe there’s something goin on here where I’m taking too much for granted that formal systems are the best way to figure out what to believe about the world to get an accurate picture of it, and so long as you’re figuring out what to believe using some formal system then you’re forced to ground it in some assumed foundation that is not itself justified. This seems important because it means there’s a lot of interesting stuff going on with those assumptions and how they get chosen such that they cause the rest of the system to produce beliefs that track reality.
But it sounds like you think this is a confused view. I admit I feel a bit differently about things than the way they’re presented in the post, but not entirely, as my comment reflects. I think when I wrote the post I really couldn’t see my way around how to know things without being able to formally justify them. Now I better understand there can be knowing without belief that knowledge is justified or true, but I do think there’s something to the idea that I am still too much captured by my own ontology and not fully seeing it as I see through it, and this causes me to get a bit mixed up in places.
The bit about self-trust is really interesting. To get a little personal and psychological, I think the big difference between me ~4 years ago and me now is that ~4 years ago I learned to trust myself in some big important way that goes deeper than surface level self-trust. On the surface this looks like trusting my feelings and experiences, but to a certain extent anyone can do this if you just give them feedback that says trust those things more than other things. But I’m talking about something a level deeper where I trust the process by which those things get generated to do what they do and that they’re doing their best (in the sense that the universe is deterministic and we’re all always doing our best/worst because it could have been no other way).
If I’m honest I’ve really struggled to understand what’s going on with Lob’s theorem and why it seems like a big deal to everyone. But it seems like from what you’re saying it’s related to the issue that got me started on things, which is how do you deal with the problem of verifying that another agent (which could be yourself at another time) shares your values and is thus in some way aligned with you. If that’s the case maybe I already grasp the fundamentals of this obstacle Lob seems to create and have just failed to realize that’s what folks were talking about when they talk about Lob with respect to aligning AI.
@abramdemski Wanted to say thanks again for engaging with my posts and pointing me towards looking again at Lob. It’s weird: now that I’ve taken so time to understand it, it’s just what in my mind was already the thing going on with Godel, just I wasn’t doing a great job of separating out what Godel proves and what the implications are. As presented on its own, Lob didn’t seem that interesting to me so I kept bouncing off it as something worth looking at, but now I realize it’s just the same thing I learned from GEB’s presentation of Peano arithmetic and Godel when I read it 20+ years ago.
When I go back to make revisions to the book, I’ll have to reconsider including Godel and Lob somehow in the text. I didn’t because I felt like it was a bit complicated and I didn’t really need to dig into it since I think there’s already a bit too many cases where people use Godel to overreach and draw conclusions that aren’t true, but it’s another way to explain these ideas. I just have to think about if Godel and Lob are necessary: that is, do I need to appeal to them to make my key points, or are these things that are better left as additional topics I can point folks at but not key to understanding the intuitions I want them to develop.
I’ve heard Lob remarked that he would never have published if he realized earlier how close his theorem was to just Godel’s second incompleteness theorem; but I can’t seem to entirely agree with Lob there. It does seem like a valuable statement of its own.
I agree, Godel is dangerously over-used, so the key question is whether it’s necessary here. Other formal analogs of your point include Tarski’s undefinability, and the realizablility / grain-of-truth problem. There are many ways to gesture towards a sense of “fundamental uncertainty”, so the question is: what statement of the thing do you want to make most central, and how do you want to argue/illustrate that statement?
IMHO the cybernetic picture isn’t weird or uncommon; naively, I expect it to get less pushback.
I think this is what I most want to push back on. My own sense is that you are confused about this. On my understanding, you seem to simultaneously believe that the core foundationalist assumptions make sense, and also believe an impossibility argument which shows them to be inconsistent. This doesn’t make sense to me.
My formalization here is only one possible way to understand the infinite-regress problem (although I think it does a good job of capturing the essence of it) -- but, in this formalization, the contradiction is really direct, which makes it seem pretty silly.
I also think the contradictory axioms do capture two intuitions which, like, beginning philosophy majors might endorse.
So I think the infinite regress problem should be explained to beginning philosophers as a warning against these naive assumptions about justification. (And indeed, this is how I was taught.)
But that’s what it is to me. It seems to be something else for you. Like a paradox. You write of proving the impossibility of solution, rather than resolving the problem. You write that we should “hold the problem”. Like, in some sense it is still a problem even after it has been solved.
(Perhaps the seeming contradiction is merely due to the way the criterion problem conflates multiple problems; EG, the naturalistic question of where knowledge comes from is still a live question after the justification-infinite-regress problem has been resolved.)
It makes sense to me to try to spell out the consequences of the infinite-regress problem; they may be complex and non-obvious. But for me this should involve questioning the naive assumptions of justification, and figuring out what it is they were trying to do. On my analysis, a reasonable place to go from there is the tiling agents problem and Vingean reflection. This is a more sophisticated picture of the problems rational agents run into when philosophizing about themselves, because it admits that you don’t need to already know how you know in order to know—you are already whatever sort of machine you are, and the default outcome is that you keep running as you’ve run. You’re not at risk of all your knowledge evaporating if you can’t justify it. However, there is a big problem of self-trust, and how you can achieve things over time by cooperating with yourself. And there’s an even bigger problem if you do have the ability to self-modify; then your knowledge actually might evaporate if you can’t justify it.
But this problem is to a large degree solved by the lack of a probabilistic Lob’s theorem. This means we are in a much better situation with respect to self-trust, vingean reflection, and the tiling agents problem—so long as we can accept probabilistic fallibility rather than needing perfect self-trust.
So in effect, the answer to (a newer and significantly more sophisticated version of) the recursive justification problem is the same as the answer to skepticism which you agreed was right and proper—namely, embracing probabilism rather than requiring certainty.
I expect you’ll agree with this conclusion; the part where I think we have some sort of fuzzy disagreement is the part where you still say things like “I see (2) as important because knowledge does, in fact, ground out in some sort of foundation, and we try to make sure that grounding is correct by requiring justifications for our grounding.” I think there are some different possible ways to spell out what you meant here, but I don’t think the best ones connect with (2) very much, since (2) is based on some silly-in-hindsight intuitions about justification.
Hmm, I’ll have to think about this.
So I think maybe there’s something goin on here where I’m taking too much for granted that formal systems are the best way to figure out what to believe about the world to get an accurate picture of it, and so long as you’re figuring out what to believe using some formal system then you’re forced to ground it in some assumed foundation that is not itself justified. This seems important because it means there’s a lot of interesting stuff going on with those assumptions and how they get chosen such that they cause the rest of the system to produce beliefs that track reality.
But it sounds like you think this is a confused view. I admit I feel a bit differently about things than the way they’re presented in the post, but not entirely, as my comment reflects. I think when I wrote the post I really couldn’t see my way around how to know things without being able to formally justify them. Now I better understand there can be knowing without belief that knowledge is justified or true, but I do think there’s something to the idea that I am still too much captured by my own ontology and not fully seeing it as I see through it, and this causes me to get a bit mixed up in places.
The bit about self-trust is really interesting. To get a little personal and psychological, I think the big difference between me ~4 years ago and me now is that ~4 years ago I learned to trust myself in some big important way that goes deeper than surface level self-trust. On the surface this looks like trusting my feelings and experiences, but to a certain extent anyone can do this if you just give them feedback that says trust those things more than other things. But I’m talking about something a level deeper where I trust the process by which those things get generated to do what they do and that they’re doing their best (in the sense that the universe is deterministic and we’re all always doing our best/worst because it could have been no other way).
If I’m honest I’ve really struggled to understand what’s going on with Lob’s theorem and why it seems like a big deal to everyone. But it seems like from what you’re saying it’s related to the issue that got me started on things, which is how do you deal with the problem of verifying that another agent (which could be yourself at another time) shares your values and is thus in some way aligned with you. If that’s the case maybe I already grasp the fundamentals of this obstacle Lob seems to create and have just failed to realize that’s what folks were talking about when they talk about Lob with respect to aligning AI.
@abramdemski Wanted to say thanks again for engaging with my posts and pointing me towards looking again at Lob. It’s weird: now that I’ve taken so time to understand it, it’s just what in my mind was already the thing going on with Godel, just I wasn’t doing a great job of separating out what Godel proves and what the implications are. As presented on its own, Lob didn’t seem that interesting to me so I kept bouncing off it as something worth looking at, but now I realize it’s just the same thing I learned from GEB’s presentation of Peano arithmetic and Godel when I read it 20+ years ago.
When I go back to make revisions to the book, I’ll have to reconsider including Godel and Lob somehow in the text. I didn’t because I felt like it was a bit complicated and I didn’t really need to dig into it since I think there’s already a bit too many cases where people use Godel to overreach and draw conclusions that aren’t true, but it’s another way to explain these ideas. I just have to think about if Godel and Lob are necessary: that is, do I need to appeal to them to make my key points, or are these things that are better left as additional topics I can point folks at but not key to understanding the intuitions I want them to develop.
I’ve heard Lob remarked that he would never have published if he realized earlier how close his theorem was to just Godel’s second incompleteness theorem; but I can’t seem to entirely agree with Lob there. It does seem like a valuable statement of its own.
I agree, Godel is dangerously over-used, so the key question is whether it’s necessary here. Other formal analogs of your point include Tarski’s undefinability, and the realizablility / grain-of-truth problem. There are many ways to gesture towards a sense of “fundamental uncertainty”, so the question is: what statement of the thing do you want to make most central, and how do you want to argue/illustrate that statement?