For #2, not sure if this is a skeptic or an advocate point: why have a separate team at all? When designing a bridge you don’t have one team of engineers making the bridge, and a separate team of engineers making sure the bridge doesn’t fall down. Within openAI, isn’t everyone committing to good things happening, and not just strictly picking the lowest-hanging fruit? If alignment-informed research is better long-term, why isn’t the whole company the “safety team” out of simple desire to do their job?
We could make this more obviously skeptical by rephrasing it as a wisdom-of-the-crowds objection. You say we need people focused on alignment because it’s not always the lowest-hanging fruit. But other people aren’t dumb and want things to go well—are you saying they’re making a mistake?
And then you have to either say yes, they’re making a mistake because people are (e.g.) both internally and externally over-incentivized to do things that have flashy results now, or no, they’re not making a mistake, in fact having a separate alignment group is a good idea even in a perfect world because of (e.g.) specialization of basic research, or some combination of the two.
In my experience, you need separate teams doing safety research because specialization is useful—it’s easiest to make progress when both individuals and teams specialize a bit and develop taste and mastery of a narrow range of topics.
For #2, not sure if this is a skeptic or an advocate point: why have a separate team at all? When designing a bridge you don’t have one team of engineers making the bridge, and a separate team of engineers making sure the bridge doesn’t fall down. Within openAI, isn’t everyone committing to good things happening, and not just strictly picking the lowest-hanging fruit? If alignment-informed research is better long-term, why isn’t the whole company the “safety team” out of simple desire to do their job?
We could make this more obviously skeptical by rephrasing it as a wisdom-of-the-crowds objection. You say we need people focused on alignment because it’s not always the lowest-hanging fruit. But other people aren’t dumb and want things to go well—are you saying they’re making a mistake?
And then you have to either say yes, they’re making a mistake because people are (e.g.) both internally and externally over-incentivized to do things that have flashy results now, or no, they’re not making a mistake, in fact having a separate alignment group is a good idea even in a perfect world because of (e.g.) specialization of basic research, or some combination of the two.
In my experience, you need separate teams doing safety research because specialization is useful—it’s easiest to make progress when both individuals and teams specialize a bit and develop taste and mastery of a narrow range of topics.