The answer that’s obvious to me is that my mental moral machinery—both the bit that says “specks of dust in the eye can’t outweigh torture, no matter how many there are” and the bit that says “however small the badness of a thing, enough repetition of it can make it arbitrarily awful” or “maximize expected sum of utilities”—wasn’t designed for questions with numbers like 3^^^3 in. In view of which, I profoundly mistrust any answer I might happen to find “obvious” to the question itself.
It is not humility to note that extrapolating models unimaginably far beyond their normal operating ranges is a fraught business. Just because we can apply a certain utility approximation to our monkeysphere, or even a few orders of magnitude above our monkeysphere, doesn’t mean the limiting behavior matches our approximation.
In other words, you’re meta-cogitation is
1 - do I trust my very certain intuition? or
2 - do I trust the heuristic from formal/mathematical thinking (that I see as useful partially and specifically to compensate for inaccuracies in our intuition)?
The answer that’s obvious to me is that my mental moral machinery—both the bit that says “specks of dust in the eye can’t outweigh torture, no matter how many there are” and the bit that says “however small the badness of a thing, enough repetition of it can make it arbitrarily awful” or “maximize expected sum of utilities”—wasn’t designed for questions with numbers like 3^^^3 in. In view of which, I profoundly mistrust any answer I might happen to find “obvious” to the question itself.
Isn’t this just appeal to humility? If not, what makes this different?
It is not humility to note that extrapolating models unimaginably far beyond their normal operating ranges is a fraught business. Just because we can apply a certain utility approximation to our monkeysphere, or even a few orders of magnitude above our monkeysphere, doesn’t mean the limiting behavior matches our approximation.
In other words, you’re meta-cogitation is 1 - do I trust my very certain intuition? or 2 - do I trust the heuristic from formal/mathematical thinking (that I see as useful partially and specifically to compensate for inaccuracies in our intuition)?