This post highlights for me that we don’t have a good understanding of what things like “more rational” and “more sane” mean, in terms of what dimensions human minds tend to vary along as a result of nature, ordinary nurture, and specialized nurture of the kind CFAR is trying to do.
I mentioned some specific biases that seem especially likely to cause risk for an FAI team. Is that the kind of “understanding” you’re talking about, or something else?
Your post suggests that an FAI feasibility team would be made of the same people who would then (depending on their findings) go ahead and form an actual FAI team, but does that need to be the case?
I think probably there would just be an FAI research team that is told to continually reevaluate feasibility/safety as it goes. I just called it “FAI feasibility team” to emphasize that at the start its most important aim would be to evaluate feasibility and safety. Having an actual separate feasibility team might buy some additional overall sanity (but how, besides that attachment to being FAI researchers won’t be an issue since they won’t continue to be FAI researchers either way?). It seems like there would probably be better ways to spend the extra resources if we had them though.
I mentioned some specific biases that seem especially likely to cause risk for an FAI team. Is that the kind of “understanding” you’re talking about, or something else?
I think that falls under my parenthetical comment in the first paragraph. Understanding what rationality-type skills would make this specific thing go well is obviously useful, but it would also be great if we had a general understanding of what rationality-type skills naturally vary together, so that we can use phrases like “more rational” and have a better idea of what they refer to across different contexts.
It seems like there would probably be better ways to spend the extra resources if we had them though.
Maybe? Note that if people like Holden have concerns about whether FAI is too dangerous, that might make them more likely to provide resources toward a separate FAI feasibility team than toward, say, a better FAI team, so it’s not necessarily a fixed heap of resources that we’re distributing.
I mentioned some specific biases that seem especially likely to cause risk for an FAI team. Is that the kind of “understanding” you’re talking about, or something else?
I think probably there would just be an FAI research team that is told to continually reevaluate feasibility/safety as it goes. I just called it “FAI feasibility team” to emphasize that at the start its most important aim would be to evaluate feasibility and safety. Having an actual separate feasibility team might buy some additional overall sanity (but how, besides that attachment to being FAI researchers won’t be an issue since they won’t continue to be FAI researchers either way?). It seems like there would probably be better ways to spend the extra resources if we had them though.
I think that falls under my parenthetical comment in the first paragraph. Understanding what rationality-type skills would make this specific thing go well is obviously useful, but it would also be great if we had a general understanding of what rationality-type skills naturally vary together, so that we can use phrases like “more rational” and have a better idea of what they refer to across different contexts.
Maybe? Note that if people like Holden have concerns about whether FAI is too dangerous, that might make them more likely to provide resources toward a separate FAI feasibility team than toward, say, a better FAI team, so it’s not necessarily a fixed heap of resources that we’re distributing.