Crossposting this follow-up thread, which I think clarifies the intended scope of the argument this is replying to:
Okay, so… making a final effort to spell things out.
What this thread is doing, is refuting a particular bad argument, quoted above, standard among e/accs, about why it’ll be totally safe to build superintelligence:
That the Solar System or galaxy is large, therefore, they will have no use for the resources of Earth.
The flaw in this reasoning is that, if your choice is to absorb all the energy the Sun puts out, or alternatively, leave a hole in your Dyson Sphere so that some non-infrared light continues to shine in one particular direction, you will do a little worse—have a little less income, for everything else you want to do—if you leave the hole in the Dyson Sphere. That the hole happens to point at Earth is not an argument in favor of doing this, unless you have some fondness in your preferences for something that lives on Earth and requires sunlight.
In other words, the size of the Solar System does not obviate the work of alignment; in the argument for how this ends up helping humanity at all, there is a key step where the ASI cares about humanity and wants to preserve it. But if you could put this quality into an ASI by some clever trick of machine learning (they can’t, but this is a different and longer argument) why do you need the Solar System to even be large? A human being runs on 100 watts. Without even compressing humanity at all, 800GW, a fraction of the sunlight falling on Earth alone, would suffice to go on operating our living flesh, if Something wanted to do that to us.
The quoted original tweet, as you will see if you look up, explicitly rejects that this sort of alignment is possible, and relies purely on the size of the Solar System to carry the point instead.
This is what is being refuted.
It is being refuted by the following narrow analogy to Bernard Arnault: that even though he has $170 billion, he still will not spend $77 on some particular goal that is not his goal. It is not trying to say of Arnault that he has never done any good in the world. It is a much narrower analogy than that. It is trying to give an example of a very simple property that would be expected by default in a powerful mind: that they will not forgo even small fractions of their wealth in order to accomplish some goal they have no interest in accomplishing.
Indeed, even if Arnault randomly threw $77 at things until he ran out of money, Arnault would be very unlikely to do any particular specific possible thing that cost $77; because he would run out of money before he had done even three billion things, and there are a lot more possible things than that.
If you think this point is supposed to be deep or difficult or that you are supposed to sit down and refute it, you are misunderstanding it. It’s not meant to be a complicated point. Arnault could still spend $77 on a particular expensive cookie if he wanted to; it’s just that “if he wanted to” is doing almost all of the work, and “Arnault has $170 billion” is doing very little on it. I don’t have that much money, and I could also spend $77 on a Lego set if I wanted to, operative phrase, “if I wanted to”.
This analogy is meant to support an equally straightforward and simple point about minds in general, which suffices to refute the single argument step quoted at the top of this thread: that because the Solar System is large, superintelligences will leave humanity alone even if they are not aligned.
I suppose, with enough work, someone can fail to follow that point. In this case I can only hope you are outvoted before you get a lot of people killed.
If you then look at the replies, you’ll see that of course people are then going, “Oh, it doesn’t matter that they wouldn’t just relinquish sunlight for no reason; they’ll love us like parents!”
Conversely, if I had tried to lay out the argument for why, no, ASIs will not automatically love us like parents, somebody would have said: “Why does that matter? The Solar System is large!”
If one doesn’t want to be one of those people, one needs the attention span to sit down and listen as one wacky argument for “why it’s not at all dangerous to build machine superintelligences”, is refuted as one argument among several. And then, perhaps, sit down to hear the next wacky argument refuted. And the next. And the next. Until you learn to generalize, and no longer need it explained to you each time, or so one hopes.
If instead on the first step you run off and say, “Oh, well, who cares about that argument; I’ve got this other argument instead!” then you are not cultivating the sort of mental habits that ever reach understanding of a complicated subject. For you will not sit still to hear the refutation of your second wacky argument either, and by the time we reach the third, why, you’ll have wrapped right around to the first argument again.
It is for this reason that a mind that ever wishes to learn anything complicated, must learn to cultivate an interest in which particular exact argument steps are valid, apart from whether you yet agree or disagree with the final conclusion, because only in this way can you sort through all the arguments and finally sum them.
I don’t feel comfortable changing the title of other people’s posts unilaterally, though I agree that a title change would be good.
To my own surprise, I wasn’t actually the one who crossposted this and came up with the title (my guess is it was Robby). I poked him about changing the title.
It was me. I initially suggested “Bernard Arnault won’t give you $77” as the title, Eliezer said “don’t bury the lead, just say ‘ASI will not leave just a little sunlight for Earth’”. After reading this thread I was thinking about alternate titles and was thinking about ones that would both convey the right thing and feel like a reasonably succinct/aesthetic/etc.
Maybe change “superintelligences will not spare Earth a little sunlight” to “unaligned superintelligences will not spare...”, or “unaligned AIs will not spare...”, given that it’s not addressing hypothetical AIs that “love us like parents”.
Crossposting this follow-up thread, which I think clarifies the intended scope of the argument this is replying to:
Maybe you should change the title of this post? It would also help if the post linked to the kinds of arguments he was refuting.
I don’t feel comfortable changing the title of other people’s posts unilaterally, though I agree that a title change would be good.
To my own surprise, I wasn’t actually the one who crossposted this and came up with the title (my guess is it was Robby). I poked him about changing the title.
It was me. I initially suggested “Bernard Arnault won’t give you $77” as the title, Eliezer said “don’t bury the lead, just say ‘ASI will not leave just a little sunlight for Earth’”. After reading this thread I was thinking about alternate titles and was thinking about ones that would both convey the right thing and feel like a reasonably succinct/aesthetic/etc.
I updated the title with one Eliezer seemed fine with (after poking Robby). Not my top choice, but better than the previous one.
Maybe change “superintelligences will not spare Earth a little sunlight” to “unaligned superintelligences will not spare...”, or “unaligned AIs will not spare...”, given that it’s not addressing hypothetical AIs that “love us like parents”.
I didn’t cross-post it, but I’ve poked EY about the title!
I just edited this into the OP.