It is ~15% plausible to me that this is written in a way that includes me personally?
I hope not. I hope when you were counting your 30 people, that I’m not NEARLY important enough to show up in that kind of list these days but… maybe I still am? :-(
I wrote the stuff below, and I think maybe my thesis statement could be something like: “have you considered that the problem isn’t inter-generational trauma but rather some trauma from an unusually clear perception of actually existing circumstances that would traumatize most people if most people paid attention to it?”
Like… which part of the situation isn’t objectively pretty bad? Are you seeing the same world as me?
I&II. (EDITED) THIS IS NOT MY FIRST RODEO
...
I had some text here about “other political half-successes I’ve seen” but it suffices to say that this is not my first rodeo. It is normal for smart people to be ahead of the curve, and it is normal for them to be copied at lower fidelity, and it is normal for half-broken versions of what they were trying to say to be used to license some things they wanted and many things they disagree with.
This is hard to manage feelings about, kind of intrinsically.
Most of my regrets from these experiences, in retrospect, are about not hitting on the main big ideas that were most central, and which could have licensed shutting down DUMB VERSIONS of similar content.
Concepts like justice and fairness tend to be safer to transmit, especially if you include a demand for rigor.
Coherent Extrapolated Volition is imperfect, but I see a lot of wisdom in promoting it under the frame of “hard to ruin if picked up by lots of people” :-)
...
III. GOVERNANCE FUTURISM… MAYBE?
By 2013, I gave up hope on SingInst… because of how sad and disheartened I was that EVEN SingInst were probably ALSO going to predictably fail at saving the world…
Then I switched to trying to figure out, as a hobby, why organizations so often seem to start to systematically not do what they are nominally AIMING AT.
I don’t regret what I did at Google and the “institutional” line of thinking still doesn’t seem super obviously wrong to me.
Critch, I don’t think that you or Samo much disagree with many people (including me) that: NEW institutions will be decisive here because the old ones are simply INADEQUATE.
The focus on institutions doesn’t seem like it is actually the wrong focus? Still! After all these years!
Which is almost maybe hopeful? :-)
IV. CONGRESS CRITTERS ARE VALID MORAL PATIENTS
Let’s be clear, this kind of thing is NOT a problem I see in myself, but I could see how someone might seen it in me based on my circumstances:
The Fundamental Attribution Error (wiki/Fundamental_attribution_error) is a cognitive bias whereby you too often attribute someone else’s behavior to a fundamental (unchanging) aspect of their personality, rather than considering how their behavior might be circumstantial and likely to change.
When I interact with people F2F, I like practically all of them. Individual humans are like: THE BEST.
People in isolation, with no one watching, in an environment of safety, can think so clearly and so well while requiring so few calories. Also, all social mammals are wonderful to any other mammals that tickle their oxytocin circuits, and all mammals have these circuits, and its great <3
If I was at a cocktail party and bumped into a Congressman (assuming the goal was to make a good impression rather than hide or something) then I could probably tell a joke and get a laugh. I’m pretty sure it wouldn’t be hard at all to like the guy. Such people are often almost literally sparkly.
If I had enough situational mojo, I might even have the chance to raise an Important Issue Of The Day and watch his face fall, but then I could explain to him that I understand some of the real organizational and humanistic constraints he faces, and express sympathy for how his powers should be increased, but won’t be.
See the relief that he doesn’t expect me to expect him to Save Us All or some dumb impossible thing like that. He would respect the sympathetic understanding of the barriers that he faces in getting to solutions (even though he’s supposedly so powerful).
However...
V. LOOK AT THE CIRCUMSTANCES WITH CLEAR EYES
If you, Critch, agree that first pass the post voting for president is bad, that could be a circumstance that you think is likely to change… but I don’t think it will change.
Since it won’t change soon, we have to solve any given public goods problems WITHOUT A RELIABLY SANE CIVILIAN COMMANDER IN CHIEF.
This isn’t about the person, it is about the institutional circumstances.
Therefore we have to solve any given public goods problem WITHOUT A RELIABLY SANE CONGRESS MAKING SANE LAWS.
(Compare this to sympathizing with a congress person at a cocktail party: I’m not so gauche as to spontaneously bring up proportional representation with someone currently in congress, because that would NOT get a laugh. They would be confused, then probably opposed. In perhaps the same way religious people are certain that invisible dragons are also intangible to flour, pretty much all powerful people know that anything that would remove them from power is bad.)
VI. CAN THE VOTERS SAVE US? (SPOILER ALERT: NO)
If we were not beyond the reach of god, then Democracy Would Save Us just from having the label “democracy” as a descriptor.
Once you understand how governance should work, you look around and realize that we are in a fractally broken hellworld when it comes to governance.
It doesn’t fix anything that almost all the individual humans are basically nice, when most of them are incapable of noticing that the government is fractally broken.
They don’t realize that no non-trivial non-traditional public goods problems can be solved by the governance system we currently have. They aren’t bad people, but they prevent problem from being solved, which is bad.
VII. MAYBE CHINA?
Maybe the CCP has competence? Maybe we could auto-translate this whole website into Chinese and beg the CCP to save the world (since they live in the world too and presumably also want it to keep existing)?
But in case you didn’t notice, when covid happened the (unelected authoritarian) government of China made sure the airports OUT INTO THE WORLD kept operating, because they wanted the rest of the world to get infected and die of the disease if they didn’t have the competence to stop it.
Once somebody posts the recipe for practically feasible self-improving Goedel machines or AIs in form of code into which one can plug arbitrary utility functions, many users will equip such AIs with many different goals, often at least partially conflicting with those of humans. The laws of physics and the availability of physical resources will eventually determine which utility functions will help their AIs more than others to multiply and become dominant in competition with AIs driven by different utility functions. Which values are “good”? The survivors will define this in hindsight, since only survivors promote their values.
Humanity is really really bad at public goods problems.
We would need coordination to avoid the disasters from a lack coordination… so unless we get lucky somehow… it just ain’t gonna happen?
That is it.
That’s the entire problem.
Saying it again for clarity: doing public goods well is a public goods problem.
All those great shiny friendly humans who love dancing and have oxytocin circuits… are not already in high quality coordination-inducing “circumstances”.
I don’t personally believe in FOOM, but there doesn’t NEED to be a FOOM to end up with really really sad outcomes.
I think that it can and probably will happen in slow motion. The institutional inertia is too great and the problem is too complicated to fit in low-bandwidth channels.
If a solution is found it might well only occur in a single brain or maybe a tiny research group. That’s how proofs are generated. Proofs are often short, and proof checking is fast, so transmitting the idea should go pretty fast?
IX. BEING MILDLY IMPOLITE TO POWER
I’m only 15% certain I’m even being mentioned here?
There appears to be something like inter-generational trauma among people who think about AI x-risk — including some of the AI-focused parts of the EA and rationality communities — which is preventing the formation of valuable high-trust relationships with newcomers that could otherwise be helpful to humanity collectively making better decisions about AI.
But like… imagine people in a boardroom privately discussing something insanely bad, like GoF research, or a marketing campaign full of lies promoting an addictive carcinogen, and one person says “hey no, that would be insanely bad”, and then… maybe this statement would wake people up, and actually work to stop the insanely bad thing?
If that was universal, I think humans would NOT be doomed, but I suspect (from experience in board rooms) that it rarely happens, and when it happens it feels the same as telling a Congressman about proportional representation and how (probably) literally every seat in congress should be competitive in every election, including his.
It isn’t “polite to power”, but I think its true.
So I think I’m going to keep doing it even if I don’t win many friends or influence as many people that way. Basically, however it is that we all die eventually, from some insane catastrophe, I want my conscience to have been clean.
I remember that sickening feeling in my gut in February of 2020, when I realized that everything in the US was built on lies and flinching and we were totally going to fail at covid, that we had ALREADY failed at it, because of the lies, and because of sociological inertia.
We don’t use IQ tests to pick bureaucrats anymore. Also you can’t fire them for base line incompetence, only gross violations of procedure.
They didn’t know that “N95 masks have pores that are smaller than aerosolized viral particles, and therefore will obviously work to protect the respiratory tract of anyone who wears one properly”.
It wasn’t part of the procedures for them to know, apparently, and that’s why they weren’t fired, I guess?
A million Americans died, and not a single person at the FDA or CDC has been put on trial or resigned or apologized for it that I’ve heard of. It used to be that incentives matter, but maybe that stopped somehow?
Also many bureaucrats hold dumb ideologies about how elected officials firing bureaucrats is bad. It is insane. It was insane in 2019. It is still insane in 2022.
In 2019 I was trying to be politely quiet about it, but I am done with that.
IF “we” can’t tell enough truth, fast enough, to coordinate to defeat a murderous yet fundamentally insentient string of 30,000 nucleotides, that transcribes into protein at a rate of ~1 amino acid / second...
...THEN how the hell are “we” supposed to tame some rogue god or genie, with exabytes of cleverness, that thinks at the speed of light??
X. DOING “GOOD” IS PROBABLY GOOD
I want to stop lying-via-deference-to-power, you know? Maybe next time we’ll fail because too many people were too honest about too many objectively true mechanical facts that are relevant to the next problem?
But it seems like that’s not a likely failure mode.
And in the meantime, I’m going to keep along with my hobby of trying to design a benevolent 99.9% voluntarist (in bothsenses) online global government that doesn’t suck. (Voluntarism has some interesting properties. I’m not committed to it, but… maybe? War is bad. Civil war is worse. The tricky part is having things be bloodless and… well… “voluntary” I think? All you REALLY have to do, basically, is get “consent” from ~8 billion people?)
Also, I think I’d be reasonably happy working on this with anyone who is smart, with a will towards Good, who wants to help. Bonus: I encourage having opinions about why the polylaw system suggested in Terra Ignota is hilariously bad, but at least it is wrong in smaller and clearer and more interesting ways than the status quo! <3
This is a good old quote that resonates with how great every individual potentially is, and implicitly how terrible poorly organized swarms of people are… (note the word is “change” not necessarily “improve”):
Never doubt that a small group of thoughtful, committed, citizens can change the world. Indeed, it is the only thing that ever has.
Just a little feedback that your comments are… very long. I personally would read them more if you cut the bottom 50% from them after you finished. Or spent a little longer adding a bulletted summary at the top so I could choose to zoom into just the sections that interest me.
If you want terse, follow smart people on twitter, then when they talk to people in terse ways, you get to see it, terse responses are enforced, and so on <3
Being too verbose is one possible protective strategy for critiquing bad faith public speech. Then, as a second order optimization, the roman numeral thing is something that I think maybe Scott and I picked up from The Last Psychiatrist, as a way to make the length tolerable, and indicate “beats” in the emotional movement, when writing about subjects where bad faith can’t be ignored as something potentially salient to the object level discussion.
My most recent follow was based on this beauty, which aphoristically encodes seemingly-authentic (and possibly virtuous) self doubt plus awareness of vast semantic systems and deep time.
I’m curious what kind of blueprint / design docs / notes you have for the voluntarist global government. Do you have a website for this? Is there a governmental-design discord discussing this? What stage is this at? etc.
It is ~15% plausible to me that this is written in a way that includes me personally?
I hope not. I hope when you were counting your 30 people, that I’m not NEARLY important enough to show up in that kind of list these days but… maybe I still am? :-(
I wrote the stuff below, and I think maybe my thesis statement could be something like: “have you considered that the problem isn’t inter-generational trauma but rather some trauma from an unusually clear perception of actually existing circumstances that would traumatize most people if most people paid attention to it?”
Like… which part of the situation isn’t objectively pretty bad? Are you seeing the same world as me?
I&II. (EDITED) THIS IS NOT MY FIRST RODEO
...
I had some text here about “other political half-successes I’ve seen” but it suffices to say that this is not my first rodeo. It is normal for smart people to be ahead of the curve, and it is normal for them to be copied at lower fidelity, and it is normal for half-broken versions of what they were trying to say to be used to license some things they wanted and many things they disagree with.
This is hard to manage feelings about, kind of intrinsically.
Most of my regrets from these experiences, in retrospect, are about not hitting on the main big ideas that were most central, and which could have licensed shutting down DUMB VERSIONS of similar content.
Concepts like justice and fairness tend to be safer to transmit, especially if you include a demand for rigor.
Coherent Extrapolated Volition is imperfect, but I see a lot of wisdom in promoting it under the frame of “hard to ruin if picked up by lots of people” :-)
...
III. GOVERNANCE FUTURISM… MAYBE?
By 2013, I gave up hope on SingInst… because of how sad and disheartened I was that EVEN SingInst were probably ALSO going to predictably fail at saving the world…
Then I switched to trying to figure out, as a hobby, why organizations so often seem to start to systematically not do what they are nominally AIMING AT.
Part of this involved reading about institutions of various sorts, while working on the least dangerous most objectively morally good thing that I could find at Google in the meantime. (Credit assignment marginalia: Will Newsome as working on institutions before I was.)
I don’t regret what I did at Google and the “institutional” line of thinking still doesn’t seem super obviously wrong to me.
Critch, I don’t think that you or Samo much disagree with many people (including me) that: NEW institutions will be decisive here because the old ones are simply INADEQUATE.
The focus on institutions doesn’t seem like it is actually the wrong focus? Still! After all these years!
Which is almost maybe hopeful? :-)
IV. CONGRESS CRITTERS ARE VALID MORAL PATIENTS
Let’s be clear, this kind of thing is NOT a problem I see in myself, but I could see how someone might seen it in me based on my circumstances:
When I interact with people F2F, I like practically all of them. Individual humans are like: THE BEST.
People in isolation, with no one watching, in an environment of safety, can think so clearly and so well while requiring so few calories. Also, all social mammals are wonderful to any other mammals that tickle their oxytocin circuits, and all mammals have these circuits, and its great <3
If I was at a cocktail party and bumped into a Congressman (assuming the goal was to make a good impression rather than hide or something) then I could probably tell a joke and get a laugh. I’m pretty sure it wouldn’t be hard at all to like the guy. Such people are often almost literally sparkly.
If I had enough situational mojo, I might even have the chance to raise an Important Issue Of The Day and watch his face fall, but then I could explain to him that I understand some of the real organizational and humanistic constraints he faces, and express sympathy for how his powers should be increased, but won’t be.
See the relief that he doesn’t expect me to expect him to Save Us All or some dumb impossible thing like that. He would respect the sympathetic understanding of the barriers that he faces in getting to solutions (even though he’s supposedly so powerful).
However...
V. LOOK AT THE CIRCUMSTANCES WITH CLEAR EYES
If you, Critch, agree that first pass the post voting for president is bad, that could be a circumstance that you think is likely to change… but I don’t think it will change.
Since it won’t change soon, we have to solve any given public goods problems WITHOUT A RELIABLY SANE CIVILIAN COMMANDER IN CHIEF.
This isn’t about the person, it is about the institutional circumstances.
If you (generic reader) agree that Congress should have elections that make every single seat competitive and proportionally representative, that could be a circumstance that someone might think is likely to change… but I don’t think it will change very soon.
Therefore we have to solve any given public goods problem WITHOUT A RELIABLY SANE CONGRESS MAKING SANE LAWS.
(Compare this to sympathizing with a congress person at a cocktail party: I’m not so gauche as to spontaneously bring up proportional representation with someone currently in congress, because that would NOT get a laugh. They would be confused, then probably opposed. In perhaps the same way religious people are certain that invisible dragons are also intangible to flour, pretty much all powerful people know that anything that would remove them from power is bad.)
VI. CAN THE VOTERS SAVE US? (SPOILER ALERT: NO)
If we were not beyond the reach of god, then Democracy Would Save Us just from having the label “democracy” as a descriptor.
Over in dath ilan, they have something like Democracy, but also they have a planet full of one room schoolhouses, where professional Watchers mind kids every morning, and make sure the 11 year olds have fun while properly teaching the 7 year olds about supply and demand. Over here on Earth, we have voters so economically illiterate that they think that rent control will improve a housing crisis!
Do you think that the elementary school teachers we currently have will “change this circumstance” fast enough?
The economic illiteracy of voters could be a circumstance that is likely to change… but I don’t think it will change very soon.
Therefore we have to solve any given problem WITHOUT ECONOMICALLY SANE VOTERS VOTING FOR LEGISLATORS THAT PROMISE SANE POLICIES.
This isn’t about the students as humans. As humans, they are cute and and friendly and like to play games and listen to songs together.
Despite being wonderful as humans, there is a problem in their curricular circumstances and economic knowledge attainment.
Once you understand how governance should work, you look around and realize that we are in a fractally broken hellworld when it comes to governance.
It doesn’t fix anything that almost all the individual humans are basically nice, when most of them are incapable of noticing that the government is fractally broken.
They don’t realize that no non-trivial non-traditional public goods problems can be solved by the governance system we currently have. They aren’t bad people, but they prevent problem from being solved, which is bad.
VII. MAYBE CHINA?
Maybe the CCP has competence? Maybe we could auto-translate this whole website into Chinese and beg the CCP to save the world (since they live in the world too and presumably also want it to keep existing)?
But in case you didn’t notice, when covid happened the (unelected authoritarian) government of China made sure the airports OUT INTO THE WORLD kept operating, because they wanted the rest of the world to get infected and die of the disease if they didn’t have the competence to stop it.
(Also they are engaged in a genocide right now.) Competent: yes! Universally Benevolent? NO!
VIII. DON’T WE ALREADY KNOW WHAT THE DEFAULT IS?
Schmidhuber is a very smart (though unpopular) guy, and another very smart person on this very website went around asking AI researchers (including him) about Friendliness in AI long ago and Schmidhuber was stunningly unbiased and clear in his explanation of what was was likely to happen:
Avoiding a situation like this requires coordination, which is a public goods problem.
Humanity is really really bad at public goods problems.
We would need coordination to avoid the disasters from a lack coordination… so unless we get lucky somehow… it just ain’t gonna happen?
That is it.
That’s the entire problem.
Saying it again for clarity: doing public goods well is a public goods problem.
All those great shiny friendly humans who love dancing and have oxytocin circuits… are not already in high quality coordination-inducing “circumstances”.
I don’t personally believe in FOOM, but there doesn’t NEED to be a FOOM to end up with really really sad outcomes.
All it needs is for the top 3 or 10 or 37 competent human entities that are willing to go down this route (presumably corporations or state actors?) to compete with each other, at pumping the predictability out of the world, via stock markets or engagement algorithms or anything else they can come up with. (The story I linked to has ONE company, and they go quite fast. That part seems unrealistic to me. The board room conversations are not depicted. But the way they “harvest predictability out of the world” seems roughly like “how it is likely to go”.)
I think that it can and probably will happen in slow motion. The institutional inertia is too great and the problem is too complicated to fit in low-bandwidth channels.
If a solution is found it might well only occur in a single brain or maybe a tiny research group. That’s how proofs are generated. Proofs are often short, and proof checking is fast, so transmitting the idea should go pretty fast?
IX. BEING MILDLY IMPOLITE TO POWER
I’m only 15% certain I’m even being mentioned here?
But like… imagine people in a boardroom privately discussing something insanely bad, like GoF research, or a marketing campaign full of lies promoting an addictive carcinogen, and one person says “hey no, that would be insanely bad”, and then… maybe this statement would wake people up, and actually work to stop the insanely bad thing?
If that was universal, I think humans would NOT be doomed, but I suspect (from experience in board rooms) that it rarely happens, and when it happens it feels the same as telling a Congressman about proportional representation and how (probably) literally every seat in congress should be competitive in every election, including his.
It isn’t “polite to power”, but I think its true.
So I think I’m going to keep doing it even if I don’t win many friends or influence as many people that way. Basically, however it is that we all die eventually, from some insane catastrophe, I want my conscience to have been clean.
X. MAYBE IN HEAVEN I CAN HANG OUT WITH DIOGENES?
I remember that sickening feeling in my gut in February of 2020, when I realized that everything in the US was built on lies and flinching and we were totally going to fail at covid, that we had ALREADY failed at it, because of the lies, and because of sociological inertia.
We don’t use IQ tests to pick bureaucrats anymore. Also you can’t fire them for base line incompetence, only gross violations of procedure.
They didn’t know that “N95 masks have pores that are smaller than aerosolized viral particles, and therefore will obviously work to protect the respiratory tract of anyone who wears one properly”.
It wasn’t part of the procedures for them to know, apparently, and that’s why they weren’t fired, I guess?
A million Americans died, and not a single person at the FDA or CDC has been put on trial or resigned or apologized for it that I’ve heard of. It used to be that incentives matter, but maybe that stopped somehow?
Also many bureaucrats hold dumb ideologies about how elected officials firing bureaucrats is bad. It is insane. It was insane in 2019. It is still insane in 2022.
In 2019 I was trying to be politely quiet about it, but I am done with that.
IF “we” can’t tell enough truth, fast enough, to coordinate to defeat a murderous yet fundamentally insentient string of 30,000 nucleotides, that transcribes into protein at a rate of ~1 amino acid / second...
...THEN how the hell are “we” supposed to tame some rogue god or genie, with exabytes of cleverness, that thinks at the speed of light??
X. DOING “GOOD” IS PROBABLY GOOD
I want to stop lying-via-deference-to-power, you know? Maybe next time we’ll fail because too many people were too honest about too many objectively true mechanical facts that are relevant to the next problem?
But it seems like that’s not a likely failure mode.
And in the meantime, I’m going to keep along with my hobby of trying to design a benevolent 99.9% voluntarist (in both senses) online global government that doesn’t suck. (Voluntarism has some interesting properties. I’m not committed to it, but… maybe? War is bad. Civil war is worse. The tricky part is having things be bloodless and… well… “voluntary” I think? All you REALLY have to do, basically, is get “consent” from ~8 billion people?)
Also, I think I’d be reasonably happy working on this with anyone who is smart, with a will towards Good, who wants to help. Bonus: I encourage having opinions about why the polylaw system suggested in Terra Ignota is hilariously bad, but at least it is wrong in smaller and clearer and more interesting ways than the status quo! <3
This is a good old quote that resonates with how great every individual potentially is, and implicitly how terrible poorly organized swarms of people are… (note the word is “change” not necessarily “improve”):
Just a little feedback that your comments are… very long. I personally would read them more if you cut the bottom 50% from them after you finished. Or spent a little longer adding a bulletted summary at the top so I could choose to zoom into just the sections that interest me.
If you want terse, follow smart people on twitter, then when they talk to people in terse ways, you get to see it, terse responses are enforced, and so on <3
Being too verbose is one possible protective strategy for critiquing bad faith public speech. Then, as a second order optimization, the roman numeral thing is something that I think maybe Scott and I picked up from The Last Psychiatrist, as a way to make the length tolerable, and indicate “beats” in the emotional movement, when writing about subjects where bad faith can’t be ignored as something potentially salient to the object level discussion.
My most recent follow was based on this beauty, which aphoristically encodes seemingly-authentic (and possibly virtuous) self doubt plus awareness of vast semantic systems and deep time.
Okay :)
I’m curious what kind of blueprint / design docs / notes you have for the voluntarist global government. Do you have a website for this? Is there a governmental-design discord discussing this? What stage is this at? etc.
What kind of contribution(s) to this project would you say are most important right now?