So if I understand your main point, you argue that OpenAI LP incentivized new investments without endangering the safety, thanks to the capped returns. And that this tradeoff looks like one of the best possible, compared to becoming a for-profit or getting bought by a big for-profit company. Is that right?
I also think for most of the things I’m concerned about, psychological pressure to think the thing isn’t dangerous is more important; like, I don’t think we’re in the cigarette case where it’s mostly other people who get cancer while the company profits; I think we’re in the case where either the bomb ignites the atmosphere or it doesn’t, and even in wartime the evidence was that people would abandon plans that posed a serious chance of destroying humanity.
I agree with you that we’re in the second case, but that doesn’t necessarily means that there’s a fire alarm. And economic incentives might push you to go slightly further, where it looks like everything is still okay, but we reached transformative AI in a terrible way. [I don’t think it is actually the case for OpenAI right now, just answering to your point.]
Note also that economic incentives quite possibly push away from AGI towards providing narrow services (see Drexler’s various arguments that AGI isn’t economically useful, and so people won’t make it by default). If you are more worried about companies that want to build AGIs and then ask it what to do than you are about companies that want to build AIs to accomplish specific tasks, increased short-term profit motive makes OpenAI more likely to move in the second direction
Good point, I need to think more about that. A counterargument that springs to mind is that AGI research might push forward other kinds of AI, and thus bring transformative AI sooner even if it isn’t an AGI.
Out of the various mechanisms, I think the capped returns are relatively low ranking; probably the top on my list is the nonprofit board having control over decision-making (and implicitly the nonprofit board’s membership not being determined by investors, as would happen in a normal company).
So if I understand your main point, you argue that OpenAI LP incentivized new investments without endangering the safety, thanks to the capped returns. And that this tradeoff looks like one of the best possible, compared to becoming a for-profit or getting bought by a big for-profit company. Is that right?
I agree with you that we’re in the second case, but that doesn’t necessarily means that there’s a fire alarm. And economic incentives might push you to go slightly further, where it looks like everything is still okay, but we reached transformative AI in a terrible way. [I don’t think it is actually the case for OpenAI right now, just answering to your point.]
Good point, I need to think more about that. A counterargument that springs to mind is that AGI research might push forward other kinds of AI, and thus bring transformative AI sooner even if it isn’t an AGI.
Out of the various mechanisms, I think the capped returns are relatively low ranking; probably the top on my list is the nonprofit board having control over decision-making (and implicitly the nonprofit board’s membership not being determined by investors, as would happen in a normal company).