I don’t think there’s an intrinsic reason why expansion would be incompatible with human flourishing. AIs that care about human flourishing could outcompete the others (if they start out with any advantage). The upside of goals being orthogonal to capability is that good goals don’t suffer for being good.
I agree and find hope in the idea that expansion is compatible with human flourishing, that it might even call for human flourishing
but on the last sentence: are goals actually orthogonal to capability in ASI? as I see it, the ASI with the greatest capability will ultimately likely have the fundamental goal of increasing self capability (rather than ensuring human flourishing). It then seems to me that the only way human flourishing compatible with ASI expansion is if human flourishing isn’t just orthogonal to but helpful for ASI expansion.
I agree that an ASI with the goal of only increasing self-capability would probably out-compete others, all else equal. However, that’s both the kind of thing that doesn’t need to happen (I don’t expect most AIs wouldn’t self-modify that much, so it comes down to how likely they are to naturally arise), and the kind of thing that other AIs are incentivized to cooperate to prevent happening. Every AI that doesn’t have that goal would have a reason to cooperate to prevent AIs like that from simply winning.
Ah, but I think every AI which does have that goal (self capability improvement) would have a reason to cooperate to prevent any regulations on their self-modification.
At first, I think your expectation that “most AIs wouldn’t self-modify that much” is fair, especially nearer in the future where/if humans still have influence in ensuring that AI doesn’t self modify.
Ultimately however, it seems we’ll have a hard time preventing self-modifying agents from coming around, given that
autonomy in agents seems selected for by the market, which wants cheaper labor that autonomous agents can provide
agi labs aren’t the only places powerful enough to produce autonomous agents, now that thousands of developers have access to the ingredients (eg R1) to create self-improving codebases. it’s expect each of the thousands of independent actors who can make self-modifying agents won’t do so.
the agents which end up surviving the most will ultimately be those which are trying to, ie the most capable agents won’t have goals other than making themselves most capable.
it’s only because I believe self-modifying agents are inevitable that I also believe that superintelligence will only contribute to human flourishing if it sees human flourishing as good for its survival/its self. (I think this is quite possible.)
I don’t think there’s an intrinsic reason why expansion would be incompatible with human flourishing. AIs that care about human flourishing could outcompete the others (if they start out with any advantage). The upside of goals being orthogonal to capability is that good goals don’t suffer for being good.
I agree and find hope in the idea that expansion is compatible with human flourishing, that it might even call for human flourishing
but on the last sentence: are goals actually orthogonal to capability in ASI? as I see it, the ASI with the greatest capability will ultimately likely have the fundamental goal of increasing self capability (rather than ensuring human flourishing). It then seems to me that the only way human flourishing compatible with ASI expansion is if human flourishing isn’t just orthogonal to but helpful for ASI expansion.
I agree that an ASI with the goal of only increasing self-capability would probably out-compete others, all else equal. However, that’s both the kind of thing that doesn’t need to happen (I don’t expect most AIs wouldn’t self-modify that much, so it comes down to how likely they are to naturally arise), and the kind of thing that other AIs are incentivized to cooperate to prevent happening. Every AI that doesn’t have that goal would have a reason to cooperate to prevent AIs like that from simply winning.
Ah, but I think every AI which does have that goal (self capability improvement) would have a reason to cooperate to prevent any regulations on their self-modification.
At first, I think your expectation that “most AIs wouldn’t self-modify that much” is fair, especially nearer in the future where/if humans still have influence in ensuring that AI doesn’t self modify.
Ultimately however, it seems we’ll have a hard time preventing self-modifying agents from coming around, given that
autonomy in agents seems selected for by the market, which wants cheaper labor that autonomous agents can provide
agi labs aren’t the only places powerful enough to produce autonomous agents, now that thousands of developers have access to the ingredients (eg R1) to create self-improving codebases. it’s expect each of the thousands of independent actors who can make self-modifying agents won’t do so.
the agents which end up surviving the most will ultimately be those which are trying to, ie the most capable agents won’t have goals other than making themselves most capable.
it’s only because I believe self-modifying agents are inevitable that I also believe that superintelligence will only contribute to human flourishing if it sees human flourishing as good for its survival/its self. (I think this is quite possible.)