After all, if we agree that there is a set of values, a set of behaviors that we would want to a superintelligence acting in humanity’s best interest to have, why wouldn’t I myself choose to hold these values and do these behaviors?
We have finite computational power, and time costs are not free.
Taking 10 decades to make every decision is itself suboptimal. If we agree that yes, this is what we would do given sufficient resources, but we don’t so we fall back on heuristics that are themselves suboptimal, but less suboptimal than the time cost of making the optimal decision...
The unspoken assumption here is that what is limiting us is not (just) the computation limits of our brains, but our philosophical understanding of humanity’s ethics.
That there is (or could be, anyway), a version of human-optimal-ethics that my brain could execute, and that would outperform existing heuristics.
But the hypothesis that the current heuristics we have are already somewhat-optimal for our biological brains (or at least not worth the effort trying to improve) is perfectly reasonable.
We have finite computational power, and time costs are not free.
Taking 10 decades to make every decision is itself suboptimal. If we agree that yes, this is what we would do given sufficient resources, but we don’t so we fall back on heuristics that are themselves suboptimal, but less suboptimal than the time cost of making the optimal decision...
I agree.
The unspoken assumption here is that what is limiting us is not (just) the computation limits of our brains, but our philosophical understanding of humanity’s ethics.
That there is (or could be, anyway), a version of human-optimal-ethics that my brain could execute, and that would outperform existing heuristics.
But the hypothesis that the current heuristics we have are already somewhat-optimal for our biological brains (or at least not worth the effort trying to improve) is perfectly reasonable.