I like the framework.
Conceptual nit: why do you include inhibitions as a type of incentive? It seems to me more natural to group them with internal motivations than external incentives. (I understand that they sit in the same position in the argument as external incentives, but I guess I’m worried that lumping them together may somehow obscure things.)
I actually agree with quite a bit of this. (I nearly included a line about pursuing excellence in terms of time allocation, but — it seemed possibly-redundant with some of the other stuff on not making the perfect the enemy of the good, and I couldn’t quickly see how to fit it cleanly into the flow of the post, so I left it and moved on …)
I think it’s important to draw the distinction between perfection and excellence. Broadly speaking, I think people often put too much emphasis on perfection, and often not enough on excellence.
Maybe I shouldn’t have led with the “Anything worth doing, is worth doing right” quote. I do see that it’s closer to perfectionist than excellence-seeking, and I don’t literally agree with it. Though one thing I like about the quote is the corollary: “anything not worth doing right isn’t worth doing” — again something I don’t literally agree with, but something I think captures an important vibe.
I do think people in academia can fail to find the corners they should be cutting. But I also think that they write a lot of papers that (to a first approximation) just don’t matter. I think that academia would be a healthier place if people invested more in asking “what’s the important thing here?” and focusing on that, and not trying to write a paper at all until they thought they could write one with the potential to be excellent.
No, multi-author submissions are welcome! (There’s space to disclose this on the entry form.)
Can you say more about why you believe this? At first glance, it seems to be like “fundamental instability” is much more tied to how AI development goes, so I would’ve expected it to be more tractable [among LW users].
Maybe “simpler” was the wrong choice of word. I didn’t really mean “more tractable”. I just meant “it’s kind of obvious what needs to happen (even if it’s very hard to get it to happen)”. Whereas with fundamental instability it’s more like it’s unclear if it’s actually a very overdetermined fundamental instability, or what exactly could nudge it to a part of scenario space with stable possibilities.
In a post-catastrophe world, it seems quite plausible to me that the rebounding civilizations would fear existential catastrophes and dangerous technologies and try hard to avoid technology-induced catastrophes.
I agree that it’s hard to reason about this stuff so I’m not super confident in anything. However, my inside view is that this story seems plausible if the catastrophe seems like it was basically an accident, but less plausible for nuclear war. Somewhat more plausible is that rebounding civilizations would create a meaningful world government to avoid repeating history.
Just a prompt to say that if you’ve been kicking around an idea of possible relevance to the essay competition on the automation of wisdom and philosophy, now might be the moment to consider writing it up—entries are due in three weeks.
My take is that in most cases it’s probably good to discuss publicly (but I wouldn’t be shocked to become convinced otherwise).
The main plausible reason I see for it potentially being bad is if it were drawing attention to a destabilizing technology that otherwise might not be discovered. But I imagine most thoughts are kind of going to be chasing through the implications of obvious ideas. And I think that in general having the basic strategic situation be closer to common knowledge is likely to reduce the risk of war.
(You might think the discussion could also have impacts on the amount of energy going into racing, but that seems pretty unlikely to me?)
The way I understand it could work is that democratic leaders with “democracy-aligned AI” would get more effective influence on nondemocratic figures (by fine-tuned persuasion or some kind of AI-designed political zugzwang or etc), thus reducing totalitarian risks. Is my understanding correct?
Not what I’d meant—rather, that democracies could demand better oversight of their leaders, and so reduce the risk of democracies slipping into various traps (corruption, authoritarianism).
My mainline guess is that information about bad behaviour by Sam was disclosed to them by various individuals, and they owe a duty of confidence to those individuals (where revealing the information might identify the individuals, who might thereby become subject to some form of retaliation).
(“Legal reasons” also gets some of my probability mass.)
OK hmm I think I understand what you mean.
I would have thought about it like this:
“our reference class” includes roughly the observations we make before observing that we’re very early in the universe
This includes stuff like being a pre-singularity civilization
The anthropics here suggest there won’t be lots of civs later arising and being in our reference class and then finding that they’re much later in universe histories
It doesn’t speak to the existence or otherwise of future human-observer moments in a post-singularity civilization
… but as you say anthropics is confusing, so I might be getting this wrong.
I largely disagree (even now I think having tried to play the inside game at labs looks pretty good, although I have sometimes disagreed with particular decisions in that direction because of opportunity costs). I’d be happy to debate if you’d find it productive (although I’m not sure whether I’m disagreeable enough to be a good choice).
I think point 2 is plausible but doesn’t super support the idea that it would eliminate the biosphere; if it cared a little, it could be fairly cheap to take some actions to preserve at least a version of it (including humans), even if starlifting the sun.
Point 1 is the argument which I most see as supporting the thesis that misaligned AI would eliminate humanity and the biosphere. And then I’m not sure how robust it is (it seems premised partly on translating our evolved intuitions about discount rates over to imagining the scenario from the perspective of the AI system).
Wait, how does the grabby aliens argument support this? I understand that it points to “the universe will be carved up between expansive spacefaring civilizations” (without reference to whether those are biological or not), and also to “the universe will cease to be a place where new biological civilizations can emerge” (without reference to what will happen to existing civilizations). But am I missing an inferential step?
I think that you’re right that people’s jobs are a significant thing driving the difference here (thanks), but I’d guess that the bigger impact of jobs is via jobs --> culture than via jobs --> individual decisions. This impression is based on a sense of “when visiting Constellation, I feel less pull to engage in the open-ended idea exploration vs at FHI”, as well as “at FHI, I think people whose main job was something else would still not-infrequently spend some time engaging with the big open questions of the day”.
I might be wrong about that ¯\_(ツ)_/¯
It’s not obvious that open source leads to faster progress. Having high quality open source products reduces the incentives for private investment. I’m not sure in which regimes that will play out that it’s overall accelerationist, but I sort of guess that it will be decelerationist during an intense AI race (where the investments needed to push the frontier out are enormous and significantly profit-motivated).