But you could just as easily frame this as “abstract reasoning about unfamiliar domains is hard therefore you should distrust doom arguments”.
But doesn’t this argument hold with the opposite conclusion, too? E.g. “abstract reasoning about unfamiliar domains is hard therefore you should distrust arguments about good-by-default worlds”.
No, failure in the face of strong active hostile optimization pressure is different from failure when acting to overcome challenges in a neutral universe that just obeys physical laws. Building prisons is a different challenge from building bridges. Especially in the case where you know that there will be agents both inside and outside the prison that very much want to get through. Security is inherently a harder problem. A puzzle which has been designed to be challenging to solve and to have a surprising unintuitive solution is thus a better match for designing a security system. I bet you’d have much higher success rate of ‘first plans’ if the game were one of those bridge building games.
But doesn’t this argument hold with the opposite conclusion, too? E.g. “abstract reasoning about unfamiliar domains is hard therefore you should distrust arguments about good-by-default worlds”.
Yepp, all of these arguments can weigh in many different directions, depending on your background beliefs. That’s my point.
No, failure in the face of strong active hostile optimization pressure is different from failure when acting to overcome challenges in a neutral universe that just obeys physical laws. Building prisons is a different challenge from building bridges. Especially in the case where you know that there will be agents both inside and outside the prison that very much want to get through. Security is inherently a harder problem. A puzzle which has been designed to be challenging to solve and to have a surprising unintuitive solution is thus a better match for designing a security system. I bet you’d have much higher success rate of ‘first plans’ if the game were one of those bridge building games.