Eh, it’s sort of hard to talk about the overall future? That is roughly my confidence level of doom contingent on, like, Anthropic doing RSI starting in the next year. But that happening feels like something that is more like a 10-20% chance, and it’s harder to estimate what the doom probabilities will look like as we get further and further into the future (in part because something will mean we don’t get RSI soon, and it’s not obvious how that impacts further development).
Vaniver
Karma: 41,873
I think it began as the latter and became the former. (Like, when I worked there the situation seemed rosier than it does today.)
I think I buy some of this—some ‘moral progress’ is increasing wealth allowing us to afford more luxuries. The optimal amount of self-expression is higher when it doesn’t cost as much in terms of starvation.
But I think I’m mostly interested in a different sort of progress—the kind where someone’s idea of what ‘the good’ is changes. [In particular, when thinking about the deep future, it’s more relevant to ask population-ethics-style questions of “which populations would we rather exist?” than individual behavior questions like “what behavior is righteous in this case?”.]
There’s a concept that sometimes gets used of ‘technological completion’—that is, you don’t know every logical fact, but you have come across all of the relevant designs. You’re no longer designing better chips or cars or space probes, because you’ve found all of the instances on the design frontier.
So by “moral endpoint” I mostly mean which options should be chosen at technological completion. It would be weird if there was one obvious choice of what to fill the universe with, even if it’s not weird that there’s one best transistor design (or whatever).
MIRI doesn’t offer its employees a retirement plan. (OpenAI did, and this was viewed with some consternation by the more AGI-pilled employees.)
I think “the singularity is my retirement plan” is not a crazy position; it is mostly irrelevant to my personal financial situation, tho.
Unfortunately it is pretty challenging for these sorts of markets to work out because people who bet on doom can’t be paid out in situations where doom happens, and doomers who want to consume now and then pay back in worlds where they survive are probably bad counterparties who are not optimizing for their ability to pay back in worlds where they survive. (Eliezer’s bet with Bryan Caplan, for example, has Eliezer locking up money now (in order to be a good counterparty) which he then won’t be able to use, and if he wins the bet he also won’t be able to use that money. So it’s primarily symbolic.)
Suppose option A is something you could do that benefits you today (like playing video games), and option B is something that benefits someone else later (like cleaning up a park). How good option B should seem depends on how many people it will affect—if it’s a park that receives lots of visitors, there’s more benefit than if the park receives few visitors.
Thus lots of impact-generating behavior scales with p(win); the more likely the world is to exist tomorrow, the more it makes sense to save or invest instead of consume.
Atomic weapons are the first technology with the potential to end the world we’ve ever developed
I don’t think this is true, actually—but atomic weapons certainly had the potential to end New York City. It’s less obvious that someone would bomb Ithaca.
This is why I think it’s good for people to still have kids in the face of the AI thing.
I think this is true for most people but the contours are a bit detailed. I net think it’s true for me also, despite my personal situation being somewhat complicated. (I tried to have kids in 2017 and it was not obvious how long timelines were then, it didn’t work out and I am perhaps trying again soon.)
Hedging and Survival-Weighted Planning
Yes.
There’s a general issue here, which is that most situations are positive-sum instead of zero-sum. Asking which tradeoff to have on the production possibilities frontier is sometimes interesting, but most of the time the challenge is simply getting to the frontier (or pushing it further along).
The relevant chapter is called “Human Nature is Bad”, and is mostly focused on the empirical question (or the interpretational question of what it means to be good or bad). It is mostly not an argument “look, you’re Confucian, and so you need to do the Confucian thing”, which would not be very interesting or compelling, but I do think it points towards the coherence of the different parts. If you try to pick and choose elements from the buffet without considering their interactions with each other, you will end up with a worse meal.
Are there Multiple Moral Endpoints?
You’d also be interested in reading about the Intelligence Curse. Individual hackers would find themselves powerless against ASI-enabled cybersecurity.
It’s not obvious to me whether cybersecurity is offense-dominant or defense-dominant, but it wouldn’t surprise me too much if it’s defense-dominant, to the point that ‘cybersecurity’ is mostly an issue for civilian infrastructure instead of military infrastructure. But I do mean a generalized version of ‘hacking’, here, which includes things like tapping physical cables.
Democracy Dies With The Rifleman
Intelligence Dissolves Privacy
For example, some people around here are contemptuous of bioethics.
I do wonder how much of this is “contemptuous of bioethics done badly”. I can see the argument for “that means we need to do bioethics well, not continue to cede ground” but I do think it’s important to be honest about when fields are failing.
And I don’t remember Worth the Candle well enough to say one way or the other.
I think it’s… more like a wash? A lot of this depends on what you think about Anglecynn / its internal politics.
Yeah I’m probably trying to pack too many things in together. To expand on it:
I think there’s something that one can get from, for example, taking care of a garden, or a tank of shrimp, or whatever. Rationality helps a lot with it; you need to notice things, you often need to sweep away your preconceptions, you often need to rearrange how you orient to the world.
Harry really doesn’t demonstrate much of that; he couldn’t
keep alive a pet rock, after all.
And I think as you go thru the list of ratfic heroes, most of them also don’t have these sorts of responsibilities, or have them in a way that advances the plot instead of being the plot. (Miles Vorkosigan makes a lot of his feudal duties, but I think would very much not seem like a hero to a feudal audience, instead of something more like a tribal trickster deity.)
Part of this also is that it’s an ongoing relationship. You don’t get your pet to a good state and then declare mission accomplished; you instead have it occupying a bit of your attention, adjusting it as necessary. There’s a way it’s larger than what fits into your models in a way that is often breaking and expanding them, rather than being something that you can fit into your models and brilliant path your way around. (If Ender can, thru flexibility of mind, defeat the battle school, this is, in some sense, evidence that battle school was not a strong enough enemy for Ender.)
[Maybe another take on this is: ‘something to protect’ as the plot instead of the character’s motivation for getting good at punching is a pretty different type of story!]
I noticed this a long time ago and tried to write a ratfic that didn’t have this dynamic; I didn’t get particularly far, mostly because I don’t think I’m that great of a fiction writer.
I think a large part of this is which settings ratfic writers choose to write fanfic in. It is very easy to take JK Rowling’s Harry Potter setting and put ratfic in it, because it’s about as screwed up as the real world, and that calls for making major changes / doesn’t naturally call for fitting into the system.
I was writing a MLP fanfic, and the My Little Pony setting is way less screwed up; the protagonist, rather than being a frustrated genius who isn’t taken seriously by his parents or teachers, is a pampered prodigy who great things are expected of and whose education is being carefully attended to accordingly. If Twilight goes to Celestia with some complaint about how society is arranged, Celestia encourages her to write a memo to the relevant minister and then get into a policy debate which considers all of the relevant factors.
[Separately I tried writing Warhammer ratfic, which mostly turned into a meditation on how much it sucks to be in an epistemically hostile environment, and the Empire was already doing a mostly-optimal strategy given the existence of the ruinous powers. But that’s, like, a short story’s worth of content.]
I think another part of it is… lack of comfort with responsibility? In the narrow, local sense which I think makes for a good minister or romantic partner but is not the heroic responsibility of the CEO or God-Emperor or whatever.
The USG’s view is that mass surveillance of non-Americans is just fine, barring some specific agreement with the nation in question, and surveillance of Americans requires some sort of justification (like a warrant).
Note that lesswrong.com is open source, which can be easily found by googling “lesswrong.com github”.