I think this article is an extremely-valuable kick-in-the-nuts for anyone who thinks they have alignment mostly solved, or even that we’re on the right track to doing so. I do, however, have one major concern. The possibility that, failing to develop a powerful AGI first will result in someone else developing something more dangerous x amount of time later, is a legitimate and serious concern. But I fear that the mentality of “if we won’t make it powerful now, we’re doomed”, if a mentality held by enough people in the AI space, might become a self-fulfilling prophecy for destruction. If Deepmind has the mentality that if they don’t develop AGI first, and make it powerful and intervening, FAIR will destroy us all 6 months later, and FAIR then adopts the same mentality, there’s now an incentive to develop AGI quickly and powerfully. Most incentives for most organizations would not be to immediately develop a severely powerful AGI. Trying to create a powerful AGI designed to stop all other AGIs from developing on the first try, out of fear that someone will develop something more dangerous if you don’t, might ironically be what gets us all killed. I think timelines and the number of organizations with a chance at developing AGI will be crucial here. If there is a long timeline before other companies can catch up, then waiting to deploy powerful AGI makes sense, instead working on weak AGIs first. If there is a short timeline, but only a few organizations that can catch up, then coordinating with them on safety would be less difficult. Even Facebook and other companies could potentially cave to enough organizational pressure. Would someone eventually develop a dangerous, powerful AGI, if no other powerful AGI is developed to prevent it? Yes. But it’s a matter of how long that can be delayed. If it’s weeks or months, we are probably doomed. If it’s years or decades, then we might have a chance.
I think this article is an extremely-valuable kick-in-the-nuts for anyone who thinks they have alignment mostly solved, or even that we’re on the right track to doing so. I do, however, have one major concern. The possibility that, failing to develop a powerful AGI first will result in someone else developing something more dangerous x amount of time later, is a legitimate and serious concern. But I fear that the mentality of “if we won’t make it powerful now, we’re doomed”, if a mentality held by enough people in the AI space, might become a self-fulfilling prophecy for destruction. If Deepmind has the mentality that if they don’t develop AGI first, and make it powerful and intervening, FAIR will destroy us all 6 months later, and FAIR then adopts the same mentality, there’s now an incentive to develop AGI quickly and powerfully. Most incentives for most organizations would not be to immediately develop a severely powerful AGI. Trying to create a powerful AGI designed to stop all other AGIs from developing on the first try, out of fear that someone will develop something more dangerous if you don’t, might ironically be what gets us all killed. I think timelines and the number of organizations with a chance at developing AGI will be crucial here. If there is a long timeline before other companies can catch up, then waiting to deploy powerful AGI makes sense, instead working on weak AGIs first. If there is a short timeline, but only a few organizations that can catch up, then coordinating with them on safety would be less difficult. Even Facebook and other companies could potentially cave to enough organizational pressure. Would someone eventually develop a dangerous, powerful AGI, if no other powerful AGI is developed to prevent it? Yes. But it’s a matter of how long that can be delayed. If it’s weeks or months, we are probably doomed. If it’s years or decades, then we might have a chance.