This doesn’t even address their stated reason/excuse for pushing straight for AGI.
I don’t have a link handy, but Altman has said that short timelines and a slow takeoff is a good scenario for AI safety. Pushing for AGI now raises the odds that, when we get it near it, it won’t get 100x smarter or more prolific rapidly. And I think that’s right, as far as it goes. It needs to be weighed against the argument for more alignment research before approaching AGI, but doing that weighing is not trivial. I don’t think there’s a clear winner.
Now, Altman pursuing more compute with his “7T investment” push really undercuts that argument being his sincere opinion, at least now (he said bit about that a while ago, maybe 5 years?).
But even if Altman was or is lying, that doesn’t make that thesis wrong. This might be the safest route to AGI. I haven’t seen anyone even try in good faith to weigh the complexities of the two arguments against each other.
Now, you can still say that this is evil, because the obviously better path is to do decades and generations of alignment work prior to getting anywhere near AGI. But that’s simply not going to happen.
One reason that goes overlooked is that most human beings are not utilitarians. Even if they realize we’re lowering the odds of future humans having an amazing, abundant future, they are pursuing AGI right now because it might prevent tham and many of those they love from dying painfully. This is terribly selfish from a utilitarian perspective, but reason does not cross the is/ought gap to make utilitarianism any more rational than selfishness. I think calling selfishness “evil” is ultimately correct, but it’s not obvious. And by that standard, most of humanity is currently evil.
And in this case, evil intentions still might have good outcomes. While OpenAI has no good alignment plan, neither does anyone else. Humanity is simply not going to pause all AI work to study alignment for generations, so plans that include substantial slowdown are not good plans. So fast timelines with a slow takeoff based on lack of compute might still be the best chance we’ve got. Again, I don’t know and I don’t think anyone else does, either.
“One reason that goes overlooked is that most human beings are not utilitarians”
I think this point is just straightforwardly wrong. Even from a purely selfish perspective, it’s reasonable to want to stop AI.
The main reason humanity is not going to stop seems mainly like coordination problems, or something close to learned helplessness in these kind of competitive dynamics.
I’m not sure that’s true. It’s true if you adopt the dominant local perspective “alignment is very hard and we need more time to do it”. But there are other perspectives: see “AI is easy to control” by Pope & Belrose, arguing that the success of RLHF means there’s a less than 1% risk of extinction from AI. I think this perspective is both subtly wrong and deeply confused in mistaking alignment with total x-risk, but the core argument isn’t obviously wrong. So reasonable people can and do argue for full speed ahead on AGI.
I agree with pretty much all of the counterarguments made by Steve Byrnes in his Thoughts on “AI is easy to control” by Pope & Belrose. But not all reasonable people will. And those who are also non-utilitarians (most of humanity) will be pursuing AGI ASAP for rational (if ultimately subtly wrong) reasons.
I think we need to understand and take this position seriously to do a good job of avoiding extinction as best we can.
Basically, I think whether or not one thinks whether alignment is hard or not is much more of the crux than whether or not they’re utilitarian.
Pesonally, I don’t find Pope & Belrose very convincing, although I do commend them for the reasonable effort—but if I did believe that AI is likely to go well, I’d probably also be all for it. I just don’t see how this is related to utilitarianism (maybe for all but a very small subset of people in EA).
This doesn’t even address their stated reason/excuse for pushing straight for AGI.
I don’t have a link handy, but Altman has said that short timelines and a slow takeoff is a good scenario for AI safety. Pushing for AGI now raises the odds that, when we get it near it, it won’t get 100x smarter or more prolific rapidly. And I think that’s right, as far as it goes. It needs to be weighed against the argument for more alignment research before approaching AGI, but doing that weighing is not trivial. I don’t think there’s a clear winner.
Now, Altman pursuing more compute with his “7T investment” push really undercuts that argument being his sincere opinion, at least now (he said bit about that a while ago, maybe 5 years?).
But even if Altman was or is lying, that doesn’t make that thesis wrong. This might be the safest route to AGI. I haven’t seen anyone even try in good faith to weigh the complexities of the two arguments against each other.
Now, you can still say that this is evil, because the obviously better path is to do decades and generations of alignment work prior to getting anywhere near AGI. But that’s simply not going to happen.
One reason that goes overlooked is that most human beings are not utilitarians. Even if they realize we’re lowering the odds of future humans having an amazing, abundant future, they are pursuing AGI right now because it might prevent tham and many of those they love from dying painfully. This is terribly selfish from a utilitarian perspective, but reason does not cross the is/ought gap to make utilitarianism any more rational than selfishness. I think calling selfishness “evil” is ultimately correct, but it’s not obvious. And by that standard, most of humanity is currently evil.
And in this case, evil intentions still might have good outcomes. While OpenAI has no good alignment plan, neither does anyone else. Humanity is simply not going to pause all AI work to study alignment for generations, so plans that include substantial slowdown are not good plans. So fast timelines with a slow takeoff based on lack of compute might still be the best chance we’ve got. Again, I don’t know and I don’t think anyone else does, either.
“One reason that goes overlooked is that most human beings are not utilitarians” I think this point is just straightforwardly wrong. Even from a purely selfish perspective, it’s reasonable to want to stop AI.
The main reason humanity is not going to stop seems mainly like coordination problems, or something close to learned helplessness in these kind of competitive dynamics.
I’m not sure that’s true. It’s true if you adopt the dominant local perspective “alignment is very hard and we need more time to do it”. But there are other perspectives: see “AI is easy to control” by Pope & Belrose, arguing that the success of RLHF means there’s a less than 1% risk of extinction from AI. I think this perspective is both subtly wrong and deeply confused in mistaking alignment with total x-risk, but the core argument isn’t obviously wrong. So reasonable people can and do argue for full speed ahead on AGI.
I agree with pretty much all of the counterarguments made by Steve Byrnes in his Thoughts on “AI is easy to control” by Pope & Belrose. But not all reasonable people will. And those who are also non-utilitarians (most of humanity) will be pursuing AGI ASAP for rational (if ultimately subtly wrong) reasons.
I think we need to understand and take this position seriously to do a good job of avoiding extinction as best we can.
Basically, I think whether or not one thinks whether alignment is hard or not is much more of the crux than whether or not they’re utilitarian.
Pesonally, I don’t find Pope & Belrose very convincing, although I do commend them for the reasonable effort—but if I did believe that AI is likely to go well, I’d probably also be all for it. I just don’t see how this is related to utilitarianism (maybe for all but a very small subset of people in EA).