(Supposing the agent can somehow model successors)
a) The activation action would be penalized by the new agent’s expected impact. The new agent’s impact budget might essentially be deducted all at once, plus whatever frictional costs are imposed by setup. Plus, making successors is instrumentally convergent, so this seems pretty unlikely.
Do note that it wouldn’t vicariously incur the anti-”survival incentive” incentive penalty. However, this still doesn’t seem to let it sneak in extra impact, if you think about it. Therefore, just making a normal maximizer is highly penalized, for resource, instrumental, and approval reasons.
b) Heavily penalized.
c) Unclear, this is one of the embedded agency questions.
d) Heavily penalized, discussed in the anti-”survival incentive” incentive examples.
(Supposing the agent can somehow model successors)
a) The activation action would be penalized by the new agent’s expected impact. The new agent’s impact budget might essentially be deducted all at once, plus whatever frictional costs are imposed by setup. Plus, making successors is instrumentally convergent, so this seems pretty unlikely.
Do note that it wouldn’t vicariously incur the anti-”survival incentive” incentive penalty. However, this still doesn’t seem to let it sneak in extra impact, if you think about it. Therefore, just making a normal maximizer is highly penalized, for resource, instrumental, and approval reasons.
b) Heavily penalized.
c) Unclear, this is one of the embedded agency questions.
d) Heavily penalized, discussed in the anti-”survival incentive” incentive examples.