Caring about our well-being is similar to us being interesting to study, both attitudes are paying attention to us specifically, whether because we in particular made it into ASI’s values (a fragile narrow target), or because graceful extrapolation of status quo made it into their values (which I think is more likely), so that the fact that we’ve been living here in the past becomes significant. So if alignment is unlikely, s-risk is similarly unlikely. And if alignment works via robustness of moral patienthood (for ASIs that got to care about such concepts), it’s a form of respecting boundaries, so probably doesn’t pose s-risk.
There might also be some weight to trade with aliens argument, if in a few billions of years our ASI makes contact with an alien-aligned alien ASI that shares their builders’ assignment of moral patienthood to a wide range of living sapient beings. Given that the sky is empty, possibly for a legible reason even, and since the amount of reachable stuff is not unbounded, this doesn’t seem very likely. Also, the alien ASI would need to be aligned, though a sapient species not having fingers might be sufficient to get there, getting a few more millions of years of civilization and theory before AGI. But all this is likely to buy humanity is a cold backup, which needs to survive all the way to a stable ASI, through all intervening misalignments, and encryption strong enough to be unbreakable by ASIs is not too hard, so there might be some chance of losing the backup even if it’s initially made.
It doesn’t seem to me that you have addressed the central concern here. I am concerned that a paperclip maximiser would study us.
There are plenty of reasons I can imagine for why we may contain helpful information for a paperclip maximiser. One such example could be that a paperclip maximiser would want to know what an alien adversary may be like, and would decide that studying life on Earth should give insights about that.
Caring about our well-being is similar to us being interesting to study, both attitudes are paying attention to us specifically, whether because we in particular made it into ASI’s values (a fragile narrow target), or because graceful extrapolation of status quo made it into their values (which I think is more likely), so that the fact that we’ve been living here in the past becomes significant. So if alignment is unlikely, s-risk is similarly unlikely. And if alignment works via robustness of moral patienthood (for ASIs that got to care about such concepts), it’s a form of respecting boundaries, so probably doesn’t pose s-risk.
There might also be some weight to trade with aliens argument, if in a few billions of years our ASI makes contact with an alien-aligned alien ASI that shares their builders’ assignment of moral patienthood to a wide range of living sapient beings. Given that the sky is empty, possibly for a legible reason even, and since the amount of reachable stuff is not unbounded, this doesn’t seem very likely. Also, the alien ASI would need to be aligned, though a sapient species not having fingers might be sufficient to get there, getting a few more millions of years of civilization and theory before AGI. But all this is likely to buy humanity is a cold backup, which needs to survive all the way to a stable ASI, through all intervening misalignments, and encryption strong enough to be unbreakable by ASIs is not too hard, so there might be some chance of losing the backup even if it’s initially made.
It doesn’t seem to me that you have addressed the central concern here. I am concerned that a paperclip maximiser would study us.
There are plenty of reasons I can imagine for why we may contain helpful information for a paperclip maximiser. One such example could be that a paperclip maximiser would want to know what an alien adversary may be like, and would decide that studying life on Earth should give insights about that.