I might write a top level post or shortform about this at some point. I find it baffling how casually people talk about dismantling the Sun around here. I recognize that this post makes no normative claim that we should do it, but it doesn’t say that it would be bad either, and expects that we will do it even if humanity remains in power. I think we probably won’t do it if humanity remains in power, we shouldn’t do it, and if humanity disassembles the Sun, it will probably happen for some very bad reason, like a fanatical dictatorship getting in power.
If we get some even vaguely democratic system that respects human rights at least a little, then many people (probably the vast majority) will want to live on Earth in their physical bodies and many will want to have children, and many of those children will also want to live on Earth and have children on their own. I find it unlikely that all subcultures that want this will die out on Earth in 10,000 years, especially considering the selections effects: the subcultures that prefer to have natural children on Earth are the ones that natural selection favors on Earth. So the scenarios when humanity dismantles the Sun probably involve a dictatorship rounding up the Amish and killing them while maybe uploading their minds somewhere, against all their protestation. Or possibly rounding up the Amish, and forcibly “increasing their intelligence and wisdom” by some artificial means, until they realize that their “coherent extrapolated volition” was in agreement with the dictatorship all along, and then killing off their bodies after their new mind consents. I find this option hardly any better. (Also, it’s not just the Amish you are hauling to the extermination camps kicking and screaming, but my mother too. And probably your mother as well. Please don’t do that.)
Also, I think the astronomical waste is probably pretty negligible. You can probably create a very good industrial base around the Sun with just some Dyson swarm that doesn’t take up enough light to be noticeable from Earth. And then you can just send out some probes to Alpha Centauri and the other neighboring stars to dismantle them if we really want to. How much time do we lose by this? My guess is at most a few years, and we probably want to take some years anyway to do some reflection before we start any giant project.
People sometimes accuse the rationalist community of being aggressive naive utilitarians, who only believe that the AGI is going to kill everyone, because they are only projecting themselves to it, as they also want to to kill everyone if they get power, so they can establish their mind-uploaded, we-are-the-grabby-aliens, turn-the-stars-into-computronium utopia a few months earlier that way. I think this accusation is mostly false, and most rationalists are in fact pretty reasonable and want to respect other people’s rights and so on. But when I see people casually discussing dismantling the Sun, with only one critical comment (Mikhail’s) that we shouldn’t do it, and it shows up in Solstice songs as a thing we want to do in the Great Transhumanist Future twenty years from now, I start worrying again that the critics are right, and we are the bad guys.
I prefer to think that it’s not because people are in fact happy about massacring the Amish and their own mothers, but because dismantling the Sun is a meme, and people don’t think through what it means. Anyway, please stop.
(Somewhat relatedly, I think it’s not obvious at all that if a misaligned AGI takes over the world, it will dismantle the Sun. It is more likely to do it than humanity would, but still, I don’t know how you could be any confident that the misaligned AI that first takes over will be the type of linear utilitarian optimizer that really cares about conquering the last stars at the edge of the Universe, so needs to dismantle the star in order to speed up its conquest with a few years.)
Without making any normative arguments: if you’re in a position (industrially and technologically) to disassemble the sun at all, or build something like a Dyson swarm, then it’s probably not too difficult to build an artificial system to light the Earth in such a way as to mimic the sun, and make it look and feel nearly identical to biological humans living on the surface, using less than a billionth of the sun’s normal total light output. The details of tides might be tricky, but probably not out of reach.
More seriously, accusing rationalists of hauling the Amish and their mothers to camps doesn’t seem quite fair. Like you said, most rationalists seem pretty nice and aren’t proposing involuntary rapid changes. And this post certainly didn’t.
You’d need to address the actual arguments in play to write a serious post about this. “Don’t propose weird stuff” isn’t a very good argument. You could argue that went very poorly with communism, or come up with some other argument. Actually I think rationalists have come up with some. It looks to me like the more respected rationalists are pretty cautious about doing weird drastic stuff just because the logic seems correct at the time. See the unilateralist curse and Yudkiwky’s and other’s pleas that nobody do anything drastic about AGI even though they think it’s very likely going to kill us all.
This stuff is fun to think about, but it’s planning the victory party before planning how to win the war.
How to put the future into kind and rational hands seems like an equally interesting and much more urgent project right now. I’d be fine with a pretty traditional utopian future or a very weird one, but not fine with joyless machines eating the sun, or worse yet all of the suns they can reach.
So, I’m with you on “hey guys, uh, this is pretty horrifying, right? Uh, what’s with the missing mood about that.”
The thing is that not doing it is also horrifying. i.e. see also All Possible Views About Humanity’s Future Are Wild. To not eat the sun is to throw away orders of magnitude more resources than anyone has ever thrown away before. Is it percentage-wise “a small fraction of the cosmos?”. Maybe. But, (quickly checks Claude, which wrote up a fermi code snippet before answering, I can share the work if you want to doublecheck yourself), a two year delay would be… 200 galaxies lost, longterm.
When you compare “the Amish get a Sun Replica that doesn’t change their experience”, the question “Is it worth throwing away 80 trillion stars to have the real thing” is, like, not a trivial question.
IMO there isn’t an option that isn’t at least a bit horrifying in some sense that one could have a missing mood about. And while I still feel unsettled about it, I think if I have to grieve something, makes more sense to grieve in the direction of “don’t throw away 80 trillion stars worth of resources.”
I think you’re also maybe just not appreciating how much would change in 10,000 years? Like, there is no single culture that has survived 10,000 years. (Maybe one of those small tribes in the amazon? I’d still bet on there having been a lot of cultural drift there but not confidently). The Amish are only a few hundred years old. I can imagine doing a lot of moral reflection and coming to the conclusion the sun shouldn’t be eaten until all human cultures have decided it’s the right thing to do, but I do really doubt that process takes 10,000 years.
I might write a top level post or shortform about this at some point. I find it baffling how casually people talk about dismantling the Sun around here. I recognize that this post makes no normative claim that we should do it, but it doesn’t say that it would be bad either, and expects that we will do it even if humanity remains in power. I think we probably won’t do it if humanity remains in power, we shouldn’t do it, and if humanity disassembles the Sun, it will probably happen for some very bad reason, like a fanatical dictatorship getting in power.
If we get some even vaguely democratic system that respects human rights at least a little, then many people (probably the vast majority) will want to live on Earth in their physical bodies and many will want to have children, and many of those children will also want to live on Earth and have children on their own. I find it unlikely that all subcultures that want this will die out on Earth in 10,000 years, especially considering the selections effects: the subcultures that prefer to have natural children on Earth are the ones that natural selection favors on Earth. So the scenarios when humanity dismantles the Sun probably involve a dictatorship rounding up the Amish and killing them while maybe uploading their minds somewhere, against all their protestation. Or possibly rounding up the Amish, and forcibly “increasing their intelligence and wisdom” by some artificial means, until they realize that their “coherent extrapolated volition” was in agreement with the dictatorship all along, and then killing off their bodies after their new mind consents. I find this option hardly any better. (Also, it’s not just the Amish you are hauling to the extermination camps kicking and screaming, but my mother too. And probably your mother as well. Please don’t do that.)
Also, I think the astronomical waste is probably pretty negligible. You can probably create a very good industrial base around the Sun with just some Dyson swarm that doesn’t take up enough light to be noticeable from Earth. And then you can just send out some probes to Alpha Centauri and the other neighboring stars to dismantle them if we really want to. How much time do we lose by this? My guess is at most a few years, and we probably want to take some years anyway to do some reflection before we start any giant project.
People sometimes accuse the rationalist community of being aggressive naive utilitarians, who only believe that the AGI is going to kill everyone, because they are only projecting themselves to it, as they also want to to kill everyone if they get power, so they can establish their mind-uploaded, we-are-the-grabby-aliens, turn-the-stars-into-computronium utopia a few months earlier that way. I think this accusation is mostly false, and most rationalists are in fact pretty reasonable and want to respect other people’s rights and so on. But when I see people casually discussing dismantling the Sun, with only one critical comment (Mikhail’s) that we shouldn’t do it, and it shows up in Solstice songs as a thing we want to do in the Great Transhumanist Future twenty years from now, I start worrying again that the critics are right, and we are the bad guys.
I prefer to think that it’s not because people are in fact happy about massacring the Amish and their own mothers, but because dismantling the Sun is a meme, and people don’t think through what it means. Anyway, please stop.
(Somewhat relatedly, I think it’s not obvious at all that if a misaligned AGI takes over the world, it will dismantle the Sun. It is more likely to do it than humanity would, but still, I don’t know how you could be any confident that the misaligned AI that first takes over will be the type of linear utilitarian optimizer that really cares about conquering the last stars at the edge of the Universe, so needs to dismantle the star in order to speed up its conquest with a few years.)
Without making any normative arguments: if you’re in a position (industrially and technologically) to disassemble the sun at all, or build something like a Dyson swarm, then it’s probably not too difficult to build an artificial system to light the Earth in such a way as to mimic the sun, and make it look and feel nearly identical to biological humans living on the surface, using less than a billionth of the sun’s normal total light output. The details of tides might be tricky, but probably not out of reach.
You’re such a traditionalist!
More seriously, accusing rationalists of hauling the Amish and their mothers to camps doesn’t seem quite fair. Like you said, most rationalists seem pretty nice and aren’t proposing involuntary rapid changes. And this post certainly didn’t.
You’d need to address the actual arguments in play to write a serious post about this. “Don’t propose weird stuff” isn’t a very good argument. You could argue that went very poorly with communism, or come up with some other argument. Actually I think rationalists have come up with some. It looks to me like the more respected rationalists are pretty cautious about doing weird drastic stuff just because the logic seems correct at the time. See the unilateralist curse and Yudkiwky’s and other’s pleas that nobody do anything drastic about AGI even though they think it’s very likely going to kill us all.
This stuff is fun to think about, but it’s planning the victory party before planning how to win the war.
How to put the future into kind and rational hands seems like an equally interesting and much more urgent project right now. I’d be fine with a pretty traditional utopian future or a very weird one, but not fine with joyless machines eating the sun, or worse yet all of the suns they can reach.
So, I’m with you on “hey guys, uh, this is pretty horrifying, right? Uh, what’s with the missing mood about that.”
The thing is that not doing it is also horrifying. i.e. see also All Possible Views About Humanity’s Future Are Wild. To not eat the sun is to throw away orders of magnitude more resources than anyone has ever thrown away before. Is it percentage-wise “a small fraction of the cosmos?”. Maybe. But, (quickly checks Claude, which wrote up a fermi code snippet before answering, I can share the work if you want to doublecheck yourself), a two year delay would be… 200 galaxies lost, longterm.
When you compare “the Amish get a Sun Replica that doesn’t change their experience”, the question “Is it worth throwing away 80 trillion stars to have the real thing” is, like, not a trivial question.
IMO there isn’t an option that isn’t at least a bit horrifying in some sense that one could have a missing mood about. And while I still feel unsettled about it, I think if I have to grieve something, makes more sense to grieve in the direction of “don’t throw away 80 trillion stars worth of resources.”
I think you’re also maybe just not appreciating how much would change in 10,000 years? Like, there is no single culture that has survived 10,000 years. (Maybe one of those small tribes in the amazon? I’d still bet on there having been a lot of cultural drift there but not confidently). The Amish are only a few hundred years old. I can imagine doing a lot of moral reflection and coming to the conclusion the sun shouldn’t be eaten until all human cultures have decided it’s the right thing to do, but I do really doubt that process takes 10,000 years.