This seems like a great resource. I also like the way it’s presented. It’s very clean.
I’d appreciate more focus on the monetary return on investment large models provide their creators. I think that’s the key metric that will determine how far firms scale up these large models. Relatedly, I think it’s important to track advancements that improve model/training efficiency because they can change the expected ROI for further scaling models.
Thanks for the kind words and thoughtful comments.
You’re absolutely right that expected ROI ultimately determines scale of investment. I agree on your efficiency point too: scaling and efficiency are complements, in the sense that the more you have of one, the more it’s worth investing in the other.
I think we will probably include some measure of efficiency as you’ve suggested. But I’m not sure exactly what that will be, since efficiency measures tend to be benchmark-dependent so it’s hard to get apples-to-apples here for a variety of reasons. (e.g., differences in modalities, differences in how papers record their results, but also the fact that benchmarks tend to get smashed pretty quickly these days, so newer models are being compared on a different basis from old ones.) Did you have any specific thoughts about this? To be honest, this is still an area we are figuring out.
On the ROI side: while this is definitely the most important metric, it’s also the one with by far the widest error bars. The reason is that it’s impossible to predict all the creative ways people will use these models for economic ends — even GPT-3 by itself might spawn entire industries that don’t yet exist. So the best one could hope for here is something like a lower bound with the accuracy of a startup’s TAM estimate: more art than science, and very liable to be proven massively wrong in either direction. (Disclosure: I’m a modestly prolific angel investor, and I’ve spoken to — though not invested in — several companies being built on GPT-3′s API.)
There’s another reason we’re reluctant to publish ROI estimates: at the margin, these estimates themselves bolster the case for increased investment in scaling, which is concerning from a risk perspective. This probably wouldn’t be a huge effect in absolute terms, since it’s not really the sort of thing effective allocators weigh heavily as decision inputs, but there are scenarios where it matters and we’d rather not push our luck.
This seems like a great resource. I also like the way it’s presented. It’s very clean.
I’d appreciate more focus on the monetary return on investment large models provide their creators. I think that’s the key metric that will determine how far firms scale up these large models. Relatedly, I think it’s important to track advancements that improve model/training efficiency because they can change the expected ROI for further scaling models.
Thanks for the kind words and thoughtful comments.
You’re absolutely right that expected ROI ultimately determines scale of investment. I agree on your efficiency point too: scaling and efficiency are complements, in the sense that the more you have of one, the more it’s worth investing in the other.
I think we will probably include some measure of efficiency as you’ve suggested. But I’m not sure exactly what that will be, since efficiency measures tend to be benchmark-dependent so it’s hard to get apples-to-apples here for a variety of reasons. (e.g., differences in modalities, differences in how papers record their results, but also the fact that benchmarks tend to get smashed pretty quickly these days, so newer models are being compared on a different basis from old ones.) Did you have any specific thoughts about this? To be honest, this is still an area we are figuring out.
On the ROI side: while this is definitely the most important metric, it’s also the one with by far the widest error bars. The reason is that it’s impossible to predict all the creative ways people will use these models for economic ends — even GPT-3 by itself might spawn entire industries that don’t yet exist. So the best one could hope for here is something like a lower bound with the accuracy of a startup’s TAM estimate: more art than science, and very liable to be proven massively wrong in either direction. (Disclosure: I’m a modestly prolific angel investor, and I’ve spoken to — though not invested in — several companies being built on GPT-3′s API.)
There’s another reason we’re reluctant to publish ROI estimates: at the margin, these estimates themselves bolster the case for increased investment in scaling, which is concerning from a risk perspective. This probably wouldn’t be a huge effect in absolute terms, since it’s not really the sort of thing effective allocators weigh heavily as decision inputs, but there are scenarios where it matters and we’d rather not push our luck.
Thanks again!