Oh, and for the FDA/TSA/etc, apologies for not being up to writing a more scathing review. This has to do with issues mentioned here. I welcome your more critical analysis if you’d want to write it. I sort of wrote the minimal thing I could. I do want to stand by the statement that it isn’t highly fine-tuned utilitarian optimization by any stretch (IE it’s not at all like Vernor Vinge’s vision of over-optimization), and also, that many relevant people actively prefer for the TSA to exist (eg politicians and people on the street will often give defenses like “it helps avert these extreme risks”).
I’m not sure we disagree on anything. I don’t mean to imply that the TSA increases slack… Perhaps I over-emphasized the opposite-ness of my two scenarios. They are not opposite in every way.
Death by No Slack:
Utilitarian, puritan, detail-oriented (context-insensitive) thinking dominates everything. Goodhart’s curse is everywhere. Performance measurements. Tests. All procedures are evidence-based. Empirical history has a tendency to win the argument over careful reasoning (including in conversations about catastrophic risks, which means risks are often under-estimated). But we have longstanding institutions which have become highly optimized. Things have been in economic equilibrium for a long time. Everything is finely tuned. Things are close to their malthusian limit. People wouldn’t know how to deal with change.
Death by Red Tape:
Don’t swim 30 minutes after eating. Don’t fall asleep in a room with a fan going. The ultimate nanny state, regulating away anything which has been associated with danger in the past, unless it’s perceived as necessary for society to function, in which case it comes with paperwork and oversight. Climbing trees is illegal unless you have a license. If you follow all the rules you have a very high chance of dying peacefully in your bed. There’s a general sense that “follow all the rules” means “don’t do anything weird”. Any genuinely new idea gets the reaction “that sounds illegal”. Or “the insurance/bank/etc won’t like it”.
Hm, these scenarios actually have a lot of similarities...
Oh, and for the FDA/TSA/etc, apologies for not being up to writing a more scathing review. This has to do with issues mentioned here. I welcome your more critical analysis if you’d want to write it. I sort of wrote the minimal thing I could. I do want to stand by the statement that it isn’t highly fine-tuned utilitarian optimization by any stretch (IE it’s not at all like Vernor Vinge’s vision of over-optimization), and also, that many relevant people actively prefer for the TSA to exist (eg politicians and people on the street will often give defenses like “it helps avert these extreme risks”).
I’m not sure we disagree on anything. I don’t mean to imply that the TSA increases slack… Perhaps I over-emphasized the opposite-ness of my two scenarios. They are not opposite in every way.
Death by No Slack:
Utilitarian, puritan, detail-oriented (context-insensitive) thinking dominates everything. Goodhart’s curse is everywhere. Performance measurements. Tests. All procedures are evidence-based. Empirical history has a tendency to win the argument over careful reasoning (including in conversations about catastrophic risks, which means risks are often under-estimated). But we have longstanding institutions which have become highly optimized. Things have been in economic equilibrium for a long time. Everything is finely tuned. Things are close to their malthusian limit. People wouldn’t know how to deal with change.
Death by Red Tape:
Don’t swim 30 minutes after eating. Don’t fall asleep in a room with a fan going. The ultimate nanny state, regulating away anything which has been associated with danger in the past, unless it’s perceived as necessary for society to function, in which case it comes with paperwork and oversight. Climbing trees is illegal unless you have a license. If you follow all the rules you have a very high chance of dying peacefully in your bed. There’s a general sense that “follow all the rules” means “don’t do anything weird”. Any genuinely new idea gets the reaction “that sounds illegal”. Or “the insurance/bank/etc won’t like it”.
Hm, these scenarios actually have a lot of similarities...