If the risk from AGI is significant (and whether you think p(doom) is 1% or 10% or 100% is it unequivocally significant) and imminent (and whether your timelines are 3 years or 30 years it is pretty imminent) the problem is that an institution as small as MIRI is a significant part of the efforts to mitigate this risk, not whether or not MIRI gave up.
(I recognize that some of the interest in MIRI is the result of having a relatively small community of people focused on the AGI x-risk problem and the early prominence in that community of a couple of individuals, but that really is just a restatement of the problem).
Yep! At least, I’d say “as small as MIRI and as unsuccessful at alignment work thus far”. I think small groups and individual researchers might indeed crack open the problem, but no one of them is highly likely on priors to do so (at least from my perspective, having seen a few people try the really obvious approaches for a few years). So we want there to exist a much larger ecosystem of people taking a stab at it, in the hope that some of them will do surprisingly well.
It could have been that easy to pull off; it’s ultimately just a technical question, and it’s hard to say how difficult such questions are to solve in advance, when they’ve literally never been attempted before.
But it was obviously never wise for humanity to take a gamble on it being that easy (and specifically easy for the one org that happened to start talking about the problem first). And insofar as we did take that gamble, it hasn’t paid out in “we now have a clear path to solving alignment”.
If the risk from AGI is significant (and whether you think p(doom) is 1% or 10% or 100% is it unequivocally significant) and imminent (and whether your timelines are 3 years or 30 years it is pretty imminent) the problem is that an institution as small as MIRI is a significant part of the efforts to mitigate this risk, not whether or not MIRI gave up.
(I recognize that some of the interest in MIRI is the result of having a relatively small community of people focused on the AGI x-risk problem and the early prominence in that community of a couple of individuals, but that really is just a restatement of the problem).
Yep! At least, I’d say “as small as MIRI and as unsuccessful at alignment work thus far”. I think small groups and individual researchers might indeed crack open the problem, but no one of them is highly likely on priors to do so (at least from my perspective, having seen a few people try the really obvious approaches for a few years). So we want there to exist a much larger ecosystem of people taking a stab at it, in the hope that some of them will do surprisingly well.
It could have been that easy to pull off; it’s ultimately just a technical question, and it’s hard to say how difficult such questions are to solve in advance, when they’ve literally never been attempted before.
But it was obviously never wise for humanity to take a gamble on it being that easy (and specifically easy for the one org that happened to start talking about the problem first). And insofar as we did take that gamble, it hasn’t paid out in “we now have a clear path to solving alignment”.