“Why If Your AGI Doesn’t Take Over The World, Somebody Else’s Soon Will”
i.e. however good your safeguards are, it doesn’t help if:
another team can take your source code and remove safeguards (and why they might have incentives to do so)
Multiple discovery means that your AGI invention will soon be followed by 10 independent ones, at least one of which will lack necessary safeguards
EDIT: “safeguard” here means any design feature put in to prevent the AGI obtaining singleton status.
“Why If Your AGI Doesn’t Take Over The World, Somebody Else’s Soon Will”
i.e. however good your safeguards are, it doesn’t help if:
another team can take your source code and remove safeguards (and why they might have incentives to do so)
Multiple discovery means that your AGI invention will soon be followed by 10 independent ones, at least one of which will lack necessary safeguards
EDIT: “safeguard” here means any design feature put in to prevent the AGI obtaining singleton status.