So the case for the grant wasn’t “we think it’s good to make OAI go faster/better”.
I agree. My intended meaning is not that the grant is bad because its purpose was to accelerate capabilities. I apologize that the original post was ambiguous
Rather, the grant was bad for numerous reasons, including but not limited to:
It appears to have had an underwhelming governance impact (as demonstrated by the board being unable to remove Sam).
It enabled OpenAI to “safety-wash” their product (although how important this has been is unclear to me.)
From what I’ve seen at conferences and job boards, it seems reasonable to assert that the relationship between Open Phil and OpenAI has lead people to work at OpenAI.
Less important, but the grant justification appears to take seriously the idea that making AGI open source is compatible with safety. I might be missing some key insight, but it seems trivially obvious why this is a terrible idea even if you’re only concerned with human misuse and not misalignment.
Finally, it’s giving money directly to an organisation with the stated goal of producing an AGI. There is substantial negative -EV if the grant sped up timelines.
This last claim seems very important. I have not been able to find data that would let me confidently estimate OpenAI’s value at the time the grant was given. However, wikipedia mentions that “In 2017 OpenAI spent $7.9 million, or a quarter of its functional expenses, on cloud computing alone.” This certainly makes it seem that the grant provided OpenAI with a significant amount of capital, enough to have increased its research output.
Keep in mind, the grant needs to have generated 30 million in EV just to break even. I’m now going to suggest some other uses for the money, but keep in mind these are just rough estimates and I haven’t adjusted for inflation. I’m not claiming these are the best uses of 30 million dollars.
The money could have funded an organisation the size of MIRI for roughly a decade (basing my estimate on MIRI’s 2017 fundraiser, using 2020 numbers gives an estimate of ~4 years).
Imagine the shift in public awareness if there had been an AI safety Superbowl ad for 3-5 years.
In your initial post, it sounded like you were trying to say:
This grant was obviously ex ante bad. In fact, it’s so obvious that it was ex ante bad that we should strongly update against everyone involved in making it.
I think that this argument is in principle reasonable. But to establish it, you have to demonstrate that the grant was extremely obviously ex ante bad. I don’t think your arguments here come close to persuading me of this.
For example, re governance impact, when the board fired sama, markets thought it was plausible he would stay gone. If that had happened, I don’t think you’d assess the governance impact as “underwhelming”. So I think that (if you’re in favor of sama being fired in that situation, which you probably are) you shouldn’t consider the governance impact of this grant to be obviously ex ante ineffective.
I think that arguing about the impact of grants requires much more thoroughness than you’re using here. I think your post has a bad “ratio of heat to light”: you’re making a provocative claim but not really spelling out why you believe the premises.
“This grant was obviously ex ante bad. In fact, it’s so obvious that it was ex ante bad that we should strongly update against everyone involved in making it.”
This is an accurate summary.
“arguing about the impact of grants requires much more thoroughness than you’re using here”
We might not agree on the level of effort required for a quick take. I do not currently have the time available to expand this into a full write up on the EA forum but am still interested in discussing this with the community.
“you’re making a provocative claim but not really spelling out why you believe the premises.”
I think this is a fair criticism and something I hope I can improve on.
I feel frustrated that your initial comment (which is now the top reply) implies I either hadn’t read the 1700 word grant justification that is at the core of my argument, or was intentionally misrepresenting it to make my point. This seems to be an extremely uncharitable interpretation of my initial post. (Edit: I am retracting this statement and now understand Buck’s comment was meaningful context. Apologies to Buck and see commentary by Ryan Greenblat below)
Your reply has been quite meta, which makes it difficult to convince you on specific points.
Your argument on betting markets has updated me slightly towards your position, but I am not particularly convinced. My understanding is that Open Phil and OpenAI had a close relationship, and hence Open Phil had substantially more information to work with than the average manifold punter.
I feel frustrated that your initial comment (which is now the top reply) implies I either hadn’t read the 1700 word grant justification that is at the core of my argument, or was intentionally misrepresenting it to make my point.
I think this comment is extremely important for bystanders to understand the context of the grant and it isn’t mentioned in your original short form post.
So, regardless of whether you understand the situation, it’s important that other people understand the intention of the grant (and this intention isn’t obvious from your original comment). Thus, this comment from Buck is valuable.
I also think that the main interpretation from bystanders of your original shortform would be something like:
OpenPhil made a grant to OpenAI
OpenAI is bad (and this was ex-ante obvious)
Therefore this grant is bad and the people who made this grant are bad.
Fair enough if this wasn’t your intention, but I think it will be how bystanders interact with this.
Less important, but the grant justification appears to take seriously the idea that making AGI open source is compatible with safety. I might be missing some key insight, but it seems trivially obvious why this is a terrible idea even if you’re only concerned with human misuse and not misalignment.
Hmmm, can you point to where you think the grant shows this? I think the following paragraph from the grant seems to indicate otherwise:
When OpenAI launched, it characterized the nature of the risks – and the most appropriate strategies for reducing them – in a way that we disagreed with. In particular, it emphasized the importance of distributing AI broadly;1 our current view is that this may turn out to be a promising strategy for reducing potential risks, but that the opposite may also turn out to be true (for example, if it ends up being important for institutions to keep some major breakthroughs secure to prevent misuse and/or to prevent accidents). Since then, OpenAI has put out more recent content consistent with the latter view,2 and we are no longer aware of any clear disagreements. However, it does seem that our starting assumptions and biases on this topic are likely to be different from those of OpenAI’s leadership, and we won’t be surprised if there are disagreements in the future.
“In particular, it emphasized the importance of distributing AI broadly;1 our current view is that this may turn out to be a promising strategy for reducing potential risks”
Yes, I’m interpreting the phrase “may turn out” to be treating the idea with more seriousness than it deserves.
Rereading the paragraph, it seems reasonable to interpret it as politely downplaying it, in which case my statement about Open Phil taking the idea seriously is incorrect.
So the case for the grant wasn’t “we think it’s good to make OAI go faster/better”.
I agree. My intended meaning is not that the grant is bad because its purpose was to accelerate capabilities. I apologize that the original post was ambiguous
Rather, the grant was bad for numerous reasons, including but not limited to:
It appears to have had an underwhelming governance impact (as demonstrated by the board being unable to remove Sam).
It enabled OpenAI to “safety-wash” their product (although how important this has been is unclear to me.)
From what I’ve seen at conferences and job boards, it seems reasonable to assert that the relationship between Open Phil and OpenAI has lead people to work at OpenAI.
Less important, but the grant justification appears to take seriously the idea that making AGI open source is compatible with safety. I might be missing some key insight, but it seems trivially obvious why this is a terrible idea even if you’re only concerned with human misuse and not misalignment.
Finally, it’s giving money directly to an organisation with the stated goal of producing an AGI. There is substantial negative -EV if the grant sped up timelines.
This last claim seems very important. I have not been able to find data that would let me confidently estimate OpenAI’s value at the time the grant was given. However, wikipedia mentions that “In 2017 OpenAI spent $7.9 million, or a quarter of its functional expenses, on cloud computing alone.” This certainly makes it seem that the grant provided OpenAI with a significant amount of capital, enough to have increased its research output.
Keep in mind, the grant needs to have generated 30 million in EV just to break even. I’m now going to suggest some other uses for the money, but keep in mind these are just rough estimates and I haven’t adjusted for inflation. I’m not claiming these are the best uses of 30 million dollars.
The money could have funded an organisation the size of MIRI for roughly a decade (basing my estimate on MIRI’s 2017 fundraiser, using 2020 numbers gives an estimate of ~4 years).
Imagine the shift in public awareness if there had been an AI safety Superbowl ad for 3-5 years.
Or it could have saved the lives of ~1300 children.
This analysis is obviously much worse if in fact the grant was negative EV.
In your initial post, it sounded like you were trying to say:
I think that this argument is in principle reasonable. But to establish it, you have to demonstrate that the grant was extremely obviously ex ante bad. I don’t think your arguments here come close to persuading me of this.
For example, re governance impact, when the board fired sama, markets thought it was plausible he would stay gone. If that had happened, I don’t think you’d assess the governance impact as “underwhelming”. So I think that (if you’re in favor of sama being fired in that situation, which you probably are) you shouldn’t consider the governance impact of this grant to be obviously ex ante ineffective.
I think that arguing about the impact of grants requires much more thoroughness than you’re using here. I think your post has a bad “ratio of heat to light”: you’re making a provocative claim but not really spelling out why you believe the premises.
“This grant was obviously ex ante bad. In fact, it’s so obvious that it was ex ante bad that we should strongly update against everyone involved in making it.”
This is an accurate summary.
“arguing about the impact of grants requires much more thoroughness than you’re using here”
We might not agree on the level of effort required for a quick take. I do not currently have the time available to expand this into a full write up on the EA forum but am still interested in discussing this with the community.
“you’re making a provocative claim but not really spelling out why you believe the premises.”
I think this is a fair criticism and something I hope I can improve on.
I feel frustrated that your initial comment (which is now the top reply) implies I either hadn’t read the 1700 word grant justification that is at the core of my argument, or was intentionally misrepresenting it to make my point. This seems to be an extremely uncharitable interpretation of my initial post. (Edit: I am retracting this statement and now understand Buck’s comment was meaningful context. Apologies to Buck and see commentary by Ryan Greenblat below)Your reply has been quite meta, which makes it difficult to convince you on specific points.
Your argument on betting markets has updated me slightly towards your position, but I am not particularly convinced. My understanding is that Open Phil and OpenAI had a close relationship, and hence Open Phil had substantially more information to work with than the average manifold punter.
I think this comment is extremely important for bystanders to understand the context of the grant and it isn’t mentioned in your original short form post.
So, regardless of whether you understand the situation, it’s important that other people understand the intention of the grant (and this intention isn’t obvious from your original comment). Thus, this comment from Buck is valuable.
I also think that the main interpretation from bystanders of your original shortform would be something like:
OpenPhil made a grant to OpenAI
OpenAI is bad (and this was ex-ante obvious)
Therefore this grant is bad and the people who made this grant are bad.
Fair enough if this wasn’t your intention, but I think it will be how bystanders interact with this.
Thank you, this explains my error. I’ve retracted that part of my response.
Hmmm, can you point to where you think the grant shows this? I think the following paragraph from the grant seems to indicate otherwise:
“In particular, it emphasized the importance of distributing AI broadly;1 our current view is that this may turn out to be a promising strategy for reducing potential risks”
Yes, I’m interpreting the phrase “may turn out” to be treating the idea with more seriousness than it deserves.
Rereading the paragraph, it seems reasonable to interpret it as politely downplaying it, in which case my statement about Open Phil taking the idea seriously is incorrect.