By this, are you not assuming that keeping humans in charge is extremely unlikely to result in a short-term catastrophe? You may not get a million years or even a hundred years.
By the way, I think the worst risk from human control isn’t extinction. The worse, and more likely, risk, is some kind of narrow, fanatical value system being imposed universally, very possibly by direct mind control. I’d expect “safeguards” to be set up to make sure that the world won’t drift away from that system… not even in a million years. And the collateral damage from the safeguards would probably be worse than the limitations imposed by the base value system.
I would expect the mind control to apply more to the humans “in charge” than to the rest.
I’m not making any claims about feasibility, I only dispute the claim that it’s known that permanently giving up the potential for human control is an acceptable thing to do, or that making such a call (epistemic call about what is known) is reasonable in the foreseeable future. To the extent it’s possible to defer this call, it should therefore be deferred (this is a normative claim, not a plan or a prediction of feasibility). If it’s not possible to keep the potential for human control despite this uncertainty, then it’s not possible, but that won’t be because the uncertainty got resolved to the extent that it could be humanly resolved.
By this, are you not assuming that keeping humans in charge is extremely unlikely to result in a short-term catastrophe? You may not get a million years or even a hundred years.
By the way, I think the worst risk from human control isn’t extinction. The worse, and more likely, risk, is some kind of narrow, fanatical value system being imposed universally, very possibly by direct mind control. I’d expect “safeguards” to be set up to make sure that the world won’t drift away from that system… not even in a million years. And the collateral damage from the safeguards would probably be worse than the limitations imposed by the base value system.
I would expect the mind control to apply more to the humans “in charge” than to the rest.
I’m not making any claims about feasibility, I only dispute the claim that it’s known that permanently giving up the potential for human control is an acceptable thing to do, or that making such a call (epistemic call about what is known) is reasonable in the foreseeable future. To the extent it’s possible to defer this call, it should therefore be deferred (this is a normative claim, not a plan or a prediction of feasibility). If it’s not possible to keep the potential for human control despite this uncertainty, then it’s not possible, but that won’t be because the uncertainty got resolved to the extent that it could be humanly resolved.