I agree strongly that, as a problem for humans, assuming that the consequentialist model has all our real values is not a safe assumption.
I would go further, and say that this assumption is almost always going to be importantly wrong and result in loss of important values. Nor do I think this is a hypothetical failure mode, at all; I believe it is common in our circles.
Consequentialism over world-histories feels pretty safe to me. Consequentialism over world states seems pretty unsafe to me. Do you feel that even consequentialism over world histories is unsafe? What would be a potential value that couldn’t be captured by that model?
Well, I think it’s not very hard (even in our circles) to find people doing consequentialism badly, looking only at short-term / easily observable consequences (I think this is especially common among newer EA folk, and some wannabe-slytherin-types). It seemed likely Zvi meant a stronger version of the claim though, which I’m not sure how I’d operationalize.
I agree strongly that, as a problem for humans, assuming that the consequentialist model has all our real values is not a safe assumption.
I would go further, and say that this assumption is almost always going to be importantly wrong and result in loss of important values. Nor do I think this is a hypothetical failure mode, at all; I believe it is common in our circles.
Consequentialism over world-histories feels pretty safe to me. Consequentialism over world states seems pretty unsafe to me. Do you feel that even consequentialism over world histories is unsafe? What would be a potential value that couldn’t be captured by that model?
Name one example? :)
I’m here from the future to say “Sam Bankman-fried”.
Well, I think it’s not very hard (even in our circles) to find people doing consequentialism badly, looking only at short-term / easily observable consequences (I think this is especially common among newer EA folk, and some wannabe-slytherin-types). It seemed likely Zvi meant a stronger version of the claim though, which I’m not sure how I’d operationalize.