It was to stop treating any solution that didn’t involve human control as axiomatically unacceptable, without regard to other outcomes.
The issue is that it’s unclear if it’s acceptable, so should be avoided if at all possible, pending more consideration. In principle there is more time for that than what’s relevant for any other concerns that don’t involve the risk of losing control in a less voluntary way. The revealed preference looks the same as finding it unacceptable to give up the potential for human control, but the argument is different, so long term implied behavior following from that argument is different. It might only take a million years to decide to give up control.
By this, are you not assuming that keeping humans in charge is extremely unlikely to result in a short-term catastrophe? You may not get a million years or even a hundred years.
By the way, I think the worst risk from human control isn’t extinction. The worse, and more likely, risk, is some kind of narrow, fanatical value system being imposed universally, very possibly by direct mind control. I’d expect “safeguards” to be set up to make sure that the world won’t drift away from that system… not even in a million years. And the collateral damage from the safeguards would probably be worse than the limitations imposed by the base value system.
I would expect the mind control to apply more to the humans “in charge” than to the rest.
I’m not making any claims about feasibility, I only dispute the claim that it’s known that permanently giving up the potential for human control is an acceptable thing to do, or that making such a call (epistemic call about what is known) is reasonable in the foreseeable future. To the extent it’s possible to defer this call, it should therefore be deferred (this is a normative claim, not a plan or a prediction of feasibility). If it’s not possible to keep the potential for human control despite this uncertainty, then it’s not possible, but that won’t be because the uncertainty got resolved to the extent that it could be humanly resolved.
The issue is that it’s unclear if it’s acceptable, so should be avoided if at all possible, pending more consideration. In principle there is more time for that than what’s relevant for any other concerns that don’t involve the risk of losing control in a less voluntary way. The revealed preference looks the same as finding it unacceptable to give up the potential for human control, but the argument is different, so long term implied behavior following from that argument is different. It might only take a million years to decide to give up control.
By this, are you not assuming that keeping humans in charge is extremely unlikely to result in a short-term catastrophe? You may not get a million years or even a hundred years.
By the way, I think the worst risk from human control isn’t extinction. The worse, and more likely, risk, is some kind of narrow, fanatical value system being imposed universally, very possibly by direct mind control. I’d expect “safeguards” to be set up to make sure that the world won’t drift away from that system… not even in a million years. And the collateral damage from the safeguards would probably be worse than the limitations imposed by the base value system.
I would expect the mind control to apply more to the humans “in charge” than to the rest.
I’m not making any claims about feasibility, I only dispute the claim that it’s known that permanently giving up the potential for human control is an acceptable thing to do, or that making such a call (epistemic call about what is known) is reasonable in the foreseeable future. To the extent it’s possible to defer this call, it should therefore be deferred (this is a normative claim, not a plan or a prediction of feasibility). If it’s not possible to keep the potential for human control despite this uncertainty, then it’s not possible, but that won’t be because the uncertainty got resolved to the extent that it could be humanly resolved.