Note: I had originally intended to write a response post to this called “Building the Citadel of Truth”, basically arguing: “Yup, the Citadel of Truth sounds great. Let’s build it. Here are my thoughts about the constraints and design principles that would need to go into constructing it”
For various reasons I didn’t do that at the time (I think shortly afterwards I sort of burned out on the overall surrounding discourse). I might still do that someday.
Ideally, if it’s only OTHER people we’re worried about social harm from (i.e. non-aspiring-espistemic-rationalists), we still get to talk about the thing to build a fully integrated worldmodel. One property that a Citadel of Truth should have is actually keeping things private from the outside world. (This is a solveable logistical problem, although you do have to actually solve it. It might be good for LW to enable posts to be hidden from non-logged out users, perhaps requiring some karma threshold to see taboo posts)
The hardest of hard modes is “local politics”, where it’s not just that I’m worried about nebulous “outsiders” hurting me (or friends feeling pressure to disown me because they in turn face pressure from outsiders). Instead, the issue is politics inside the citadel. It seems like a quite desirable property to able to talk freely about which local orgs and people deserve money and prestige – but I don’t currently know of robust game mechanics that will actually, reliably enable this in any environment where I don’t personally know and trust each person.
Having multiple “inner citadels” of trust is sort of the de-facto way this is done currently, in my experience. Having clearer signposting on how to get trustworthy might be a net improvement.
Notably, proclaiming “I only care about truth, not politics” is not sufficient for me to trust someone in this domain. I think that’s a pretty bad litmus test.
It seems like a quite desirable property to able to talk freely about which local orgs and people deserve money and prestige – but I don’t currently know of robust game mechanics that will actually, reliably enable this in any environment where I don’t personally know and trust each person.
There should not be any “local orgs” inside the citadel; and if the people who participate in the citadel also happen to, together, constitute various other orgs… well, first of all, that’s quite a bad sign; but in any case discussions of them, and whether they deserve money and so on, should not take place inside the citadel.
If this is not obvious, then I have not communicated the concept effectively. I urge you to once again consider this part:
Any among us who have something to protect, in the world beyond the citadel, may wish to take the truths we find, and apply them to that outside world, and discuss these things with others who feel as they do. In these discussions, of plans and strategies for acting upon the wider world, the consequences of their words, for that world, may be of the utmost importance. But if so, to have such discussions, these planners will have to step outside the citadel’s walls. To talk in such a way in here—to speak, and to hold yourself and others to considering the consequences of their words upon the world—is to violate the citadel’s rule: that all that we do and say within, serves truth, and only truth.
For this reason, I am of the strong opinion that any Citadel of Truth is best built online, and not integrated strongly into any “meatspace” community, and certainly not built within, or “on top of”, or by, any existing such community.
EDIT: The point is, it’s a Citadel of Truth, not—repeat, not!—a Citadel of Discovering the Truth And Then Doing Desirable Things With It, Because That Was Our Goal All Along, Wasn’t It. If that is what you’re trying to build, then forget it; the whole thing is corrupted from the get-go, and will come to no good.
EDIT: The point is, it’s a Citadel of Truth, not—repeat, not!—a Citadel of Discovering the Truth And Then Doing Desirable Things With It, Because That Was Our Goal All Along, Wasn’t It. If that is what you’re trying to build, then forget it; the whole thing is corrupted from the get-go, and will come to no good.
Okay, yeah the thing I’m thinking about is definitely different from the thing you’re thinking about and I’ll refrain from referring to my thing as “The Citadel of Truth”.
[Edit: the thing-in-my-head still has a focus on “within the citadel-esque-thing, the primary sacred value is the truth, because to actually Use Truth to Do Desirable Things you need to actually to Focus On Truth For It’s Own Sake, and yes, this is a bit contradictory, and I’m not 100% sure how to resolve the contradiction.
But, a citadel that’s just focused on truth without paying attention to how that truth will actually get applied to anything, that doesn’t attempt to resolve the contradiction, doesn’t seem very interesting to me. That’s not the hard part]
There should not be any “local orgs” inside the citadel; and if the people who participate in the citadel also happen to, together, constitute various other orgs… well, first of all, that’s quite a bad sign; but in any case discussions of them, and whether they deserve money and so on, should not take place inside the citadel.
I do think this is plausibly quite relevant to The Thing I’m Thinking of, independent of whether it’s relevant to The Thing You’re Thinking Of. Will think on that a bit.
I’m left with sort of a confused “what problem is your conception of the Citadel actually trying to solve”, though?
The two main problems that the status quo face AFAICT (i.e. if you put down a flag and say “Truth!” and then some people show up and start talking, but nonetheless find that their talk isn’t always truthtracking), is:
There might be people Out There who dislike what you say, and harm or impose costs on you in some way
There might be people In the Conversation who have some kind of stake in the conversation, that are motivated to warp it.
I… think the first one is relatively straightforward. (There are two primary strategies I can see, of either “deciding not to care”, or “being somewhat private / obfuscated”. I think the latter is a better strategy but if you’re precommitting to “literally just focus truth with no optimization towards being able to use that truth later” I think the former strategy probably works fine)
For the second problem… well, if your solution is to filter/arrange things such that the citadel just doesn’t have Local Politics, then this problem doesn’t come up in the first place. The domains where this is coming up have to do with situations where Local Politics Is Already Here, and people wishing to be able to speak frankly despite that. The Citadel of Truth doesn’t seem like it solves that problem at all. It just posits Somewhere Else there being a Citadel where people who don’t have a stake in the Local Politics, having conversations that don’t end up affecting the Local Politics.
The two main problems that the status quo face AFAICT (i.e. if you put down a flag and say “Truth!” and then some people show up and start talking, but nonetheless find that their talk isn’t always truthtracking), is:
There might be people Out There who dislike what you say, and harm or impose costs on you in some way
There might be people In the Conversation who have some kind of stake in the conversation, that are motivated to warp it.
Note that these problems are not separate, but in fact are inextricably linked. This is because people Out There can come In Here (and will absolutely attempt to do so, in proportion to how successful your Citadel becomes), and also people In Here may decide to interact with social forces Out There.
… situations where Local Politics Is Already Here, and people wishing to be able to speak frankly despite that. The Citadel of Truth doesn’t seem like it solves that problem at all.
Indeed, it does not. Nor is it meant to.
I’m left with sort of a confused “what problem is your conception of the Citadel actually trying to solve”, though?
Figuring out the truth. Note, as per my other comment, that we currently do not have any institutions that have just that as their goal. Really—none. (If you think that this claim is obviously wrong, then, as usual: provide examples!)
Note that these problems are not separate, but in fact are inextricably linked. This is because people Out There can come In Here (and will absolutely attempt to do so, in proportion to how successful your Citadel becomes), and also people In Here may decide to interact with social forces Out There.
...
Note, as per my other comment, that we currently do not have any institutions that have just that as their goal. Really—none. (If you think that this claim is obviously wrong, then, as usual: provide examples!)
...
It is the hard part. It really, really is.
I’m not sure our models here are that different. What I’d argue (not sure if you’d disagree), is something like:
We have no institutions whose sole goal is figure out the truth, but the reason for this is that to be an “institution” (as opposed to some random collection of people just quietly figuring stuff out) you need some kind of mechanism for maintaining the institution, and this inevitably ends up instantiating it’s own version of Local Politics even it initially didn’t have such a thing.
I don’t have clear examples, no, but my guess is that there are, in fact, various small citadels throughout the world, but any citadel that’s successful enough for both of us to have heard of it, was necessarily successful enough to attract attention from Powers That Be.
Wikipedia and Academic Science both come to mind as institutions that have their own politics, but which I (suspect), still do okay-ish at generating little pocket-citadels that succeed at focusing on whatever subset of truthseeking they’ve specialized in – individual departments, projects, or research groups. The trouble lies on the outside world distinguishing which pockets are generating “real truth” and which are not (because any institution that became known as a distinguishing tool would probably become corrupted)
Perhaps one core disagreement here is about which problem is ‘actually impossible’?
I say, you don’t have the option of avoiding Local Politics, so the task is figuring out how to minimize the damage that local politics can do to epistemics (possibly aided by forking off private bubbles that are mostly inert to outsiders, thinking on their own, but reporting their findings periodically)
You say… something like ‘local politics is so toxic that the task must be figure out a way to avoid it’?
Well, roughly. I don’t think it’s possible to entirely avoid “local politics”, in a totally literal sense, because any interaction of people within any group will end up being ‘politics’ in some sense.
But, certainly my view is closer to the latter than to the former, yes. Basically, it’s just what I said in this earlier comment. To put it another way: if you already have “local politics”, you’re starting out with a disadvantage so crippling that there’s no point in even trying to build any “citadel of truth”.
… to actually Use Truth to Do Desirable Things you need to actually to Focus On Truth For It’s Own Sake, and yes, this is a bit contradictory, and I’m not 100% sure how to resolve the contradiction
I do not think there is any way to resolve the contradiction. It seems clear to me that just as no man may serve two masters, no organization may serve two goals. “What you are willing to trade off, may end up traded away”. And ultimately, you will sacrifice your pursuit of truth, if what you are actually pursuing is something else—because there will come a time when your actual goal turns out (in that situation, at that time, in that moment) to not be best served by pursuing Truth, for its own sake or otherwise.
And then your Citadel will not even be a Citadel of Truth And Something Else, but only a Citadel of Something Else, And Not Truth At All.
I think there’s still some highly technical apparent-contradiction-resolution to do in the other direction: in a monist physical universe, you can’t quite say, “only Truth matters, not consequences”, because that just amounts to caring about the consequence of there existing a physical system that implements correct epistemology: the map is part of the territory.
To be clear, I think almost everyone who brings this up outside the context of AI design is being incredibly intellectually dishonest. (“It’d be irrational to say that—we’d lose funding! And if we lose funding, then we can’t pursue Truth!”) But I want to avoid falling into the trap of letting the forceful rhetoric I need to defend against bad-faith appeals-to-consequences, obscure my view of actually substantive philosophy problems.
But, a citadel that’s just focused on truth without paying attention to how that truth will actually get applied to anything, that doesn’t attempt to resolve the contradiction, doesn’t seem very interesting to me. That’s not the hard part
It is the hard part. It really, really is.
If you doubt this, witness the fact that we currently have no such institutions.
Note: I had originally intended to write a response post to this called “Building the Citadel of Truth”, basically arguing: “Yup, the Citadel of Truth sounds great. Let’s build it. Here are my thoughts about the constraints and design principles that would need to go into constructing it”
For various reasons I didn’t do that at the time (I think shortly afterwards I sort of burned out on the overall surrounding discourse). I might still do that someday.
I touch upon the issues in this comment, which seems worth quoting here for now:
There should not be any “local orgs” inside the citadel; and if the people who participate in the citadel also happen to, together, constitute various other orgs… well, first of all, that’s quite a bad sign; but in any case discussions of them, and whether they deserve money and so on, should not take place inside the citadel.
If this is not obvious, then I have not communicated the concept effectively. I urge you to once again consider this part:
For this reason, I am of the strong opinion that any Citadel of Truth is best built online, and not integrated strongly into any “meatspace” community, and certainly not built within, or “on top of”, or by, any existing such community.
EDIT: The point is, it’s a Citadel of Truth, not—repeat, not!—a Citadel of Discovering the Truth And Then Doing Desirable Things With It, Because That Was Our Goal All Along, Wasn’t It. If that is what you’re trying to build, then forget it; the whole thing is corrupted from the get-go, and will come to no good.
Okay, yeah the thing I’m thinking about is definitely different from the thing you’re thinking about and I’ll refrain from referring to my thing as “The Citadel of Truth”.
[Edit: the thing-in-my-head still has a focus on “within the citadel-esque-thing, the primary sacred value is the truth, because to actually Use Truth to Do Desirable Things you need to actually to Focus On Truth For It’s Own Sake, and yes, this is a bit contradictory, and I’m not 100% sure how to resolve the contradiction.
But, a citadel that’s just focused on truth without paying attention to how that truth will actually get applied to anything, that doesn’t attempt to resolve the contradiction, doesn’t seem very interesting to me. That’s not the hard part]
I do think this is plausibly quite relevant to The Thing I’m Thinking of, independent of whether it’s relevant to The Thing You’re Thinking Of. Will think on that a bit.
I’m left with sort of a confused “what problem is your conception of the Citadel actually trying to solve”, though?
The two main problems that the status quo face AFAICT (i.e. if you put down a flag and say “Truth!” and then some people show up and start talking, but nonetheless find that their talk isn’t always truthtracking), is:
There might be people Out There who dislike what you say, and harm or impose costs on you in some way
There might be people In the Conversation who have some kind of stake in the conversation, that are motivated to warp it.
I… think the first one is relatively straightforward. (There are two primary strategies I can see, of either “deciding not to care”, or “being somewhat private / obfuscated”. I think the latter is a better strategy but if you’re precommitting to “literally just focus truth with no optimization towards being able to use that truth later” I think the former strategy probably works fine)
For the second problem… well, if your solution is to filter/arrange things such that the citadel just doesn’t have Local Politics, then this problem doesn’t come up in the first place. The domains where this is coming up have to do with situations where Local Politics Is Already Here, and people wishing to be able to speak frankly despite that. The Citadel of Truth doesn’t seem like it solves that problem at all. It just posits Somewhere Else there being a Citadel where people who don’t have a stake in the Local Politics, having conversations that don’t end up affecting the Local Politics.
Note that these problems are not separate, but in fact are inextricably linked. This is because people Out There can come In Here (and will absolutely attempt to do so, in proportion to how successful your Citadel becomes), and also people In Here may decide to interact with social forces Out There.
Indeed, it does not. Nor is it meant to.
Figuring out the truth. Note, as per my other comment, that we currently do not have any institutions that have just that as their goal. Really—none. (If you think that this claim is obviously wrong, then, as usual: provide examples!)
I’m not sure our models here are that different. What I’d argue (not sure if you’d disagree), is something like:
We have no institutions whose sole goal is figure out the truth, but the reason for this is that to be an “institution” (as opposed to some random collection of people just quietly figuring stuff out) you need some kind of mechanism for maintaining the institution, and this inevitably ends up instantiating it’s own version of Local Politics even it initially didn’t have such a thing.
I don’t have clear examples, no, but my guess is that there are, in fact, various small citadels throughout the world, but any citadel that’s successful enough for both of us to have heard of it, was necessarily successful enough to attract attention from Powers That Be.
Wikipedia and Academic Science both come to mind as institutions that have their own politics, but which I (suspect), still do okay-ish at generating little pocket-citadels that succeed at focusing on whatever subset of truthseeking they’ve specialized in – individual departments, projects, or research groups. The trouble lies on the outside world distinguishing which pockets are generating “real truth” and which are not (because any institution that became known as a distinguishing tool would probably become corrupted)
Perhaps one core disagreement here is about which problem is ‘actually impossible’?
I say, you don’t have the option of avoiding Local Politics, so the task is figuring out how to minimize the damage that local politics can do to epistemics (possibly aided by forking off private bubbles that are mostly inert to outsiders, thinking on their own, but reporting their findings periodically)
You say… something like ‘local politics is so toxic that the task must be figure out a way to avoid it’?
Does that sound right?
Well, roughly. I don’t think it’s possible to entirely avoid “local politics”, in a totally literal sense, because any interaction of people within any group will end up being ‘politics’ in some sense.
But, certainly my view is closer to the latter than to the former, yes. Basically, it’s just what I said in this earlier comment. To put it another way: if you already have “local politics”, you’re starting out with a disadvantage so crippling that there’s no point in even trying to build any “citadel of truth”.
I do not think there is any way to resolve the contradiction. It seems clear to me that just as no man may serve two masters, no organization may serve two goals. “What you are willing to trade off, may end up traded away”. And ultimately, you will sacrifice your pursuit of truth, if what you are actually pursuing is something else—because there will come a time when your actual goal turns out (in that situation, at that time, in that moment) to not be best served by pursuing Truth, for its own sake or otherwise.
And then your Citadel will not even be a Citadel of Truth And Something Else, but only a Citadel of Something Else, And Not Truth At All.
I think there’s still some highly technical apparent-contradiction-resolution to do in the other direction: in a monist physical universe, you can’t quite say, “only Truth matters, not consequences”, because that just amounts to caring about the consequence of there existing a physical system that implements correct epistemology: the map is part of the territory.
To be clear, I think almost everyone who brings this up outside the context of AI design is being incredibly intellectually dishonest. (“It’d be irrational to say that—we’d lose funding! And if we lose funding, then we can’t pursue Truth!”) But I want to avoid falling into the trap of letting the forceful rhetoric I need to defend against bad-faith appeals-to-consequences, obscure my view of actually substantive philosophy problems.
Everything else you said aside…
It is the hard part. It really, really is.
If you doubt this, witness the fact that we currently have no such institutions.