Of course, this is still just a proxy measure… say that we’re “in a simulation”, or that there are already superintelligences in our environment who won’t let us eat the stars, or something like that—we still want to get as good a bargaining position as we possibly can, or to coordinate with the watchers as well as we possibly can, or in a more fundamental sense we want to not waste any of our potential, which I think is the real driving intuition here.
Agreed. I was being lazy and using “astronomical waste” as a pointer to this more general concept, probably because I was primed by people talking about “astronomical waste” a bunch recently.
Further clarifying and expanding on that intuition might be very valuable, both for polemical reasons and for organizing some thoughts on AI strategy.
Also agreed, but I currently don’t have much to add to what’s already been said on this topic.
The thing I’ve seen that looks closest to white-box metaphilosophical AI in the existing literature is Eliezer’s causal validity semantics, or more precisely the set of intuitions Eliezer was drawing on to come up with the idea of causal validity semantics.
Ugh, I found CFAI largely impenetrable when I first read it, and have the same reaction reading it now. Can you try translating the section into “modern” LW language?
Agreed. I was being lazy and using “astronomical waste” as a pointer to this more general concept, probably because I was primed by people talking about “astronomical waste” a bunch recently.
Also agreed, but I currently don’t have much to add to what’s already been said on this topic.
Ugh, I found CFAI largely impenetrable when I first read it, and have the same reaction reading it now. Can you try translating the section into “modern” LW language?
CFAI is deprecated for a reason, I can’t read it either.