A true superintelligence could wipe out humanity incredibly easily—but it could build a utopia nearly as easily. Even if it were almost entirely misaligned, just a sliver of human morality could make it decide to give humans a paradise beyond their wildest imaginings.
As long as the superintelligence’s values don’t contain any components that pull against components of human morality. But in case of almost-alignment there might indeed be some such components. Almost-alignment is where s-risks live.
As long as the superintelligence’s values don’t contain any components that pull against components of human morality. But in case of almost-alignment there might indeed be some such components. Almost-alignment is where s-risks live.
Yep, totally agree (and in fact I’m at a s-risk retreat right now). Definitely a “could make it decide” rather than a “will make it decide”.