Recall here the numbers here are substantially fake. The best they can tell you is roughly “this is very big” or “this is very small”. If you want to go much beyond that, you are reading sheep entrails. They are not coming from OpenAI but web traffic measurements. Such measurements are notoriously both noisy and highly biased and the biases change over time, and so unsurprisingly, at the time, OAers were saying it had overestimated the actual users by like 100%.
The numbers have lots of ways to be misleading. For example, because Chinese DL was, and still is, so inferior, there was a whole cottage industry of Chinese companies pirating accounts and black markets in credentials. This also applied to all the Third World or embargoed or difficult countries OA has denied access to, either for paying accounts or just period. Then you have people abusing it or sexing with it and getting banned and figuring out how to create new accounts, or moving on.
Just a huge amount of whac-a-mole going on. This obviously causes issues for interpreting any user metrics: a huge decrease in user count might actually reflect a huge increase in users, and vice-versa, depending on how the security arms race is going.
Summer vacation. If you look at the graph, you’ll notice a remarkable correlation with the Western academic calendar… Someone confidently proclaiming that ChatGPT use is crashing before seeing the September–November 2023 numbers is giving hostage to fortune.
EDIT: as of 1 October, SimilarWeb and other sources are reporting increasing traffic.
Many, many alternatives spinning up, many of which are using OA as the backend, particularly after the large price drops. These would count in a naive web-traffic approach as OA ‘losing users’, rather than gaining them.
More broadly, if they are going to true competitors like Claude-2 and do not count as OA users by any definition, that’s still damaging to Goia’s thesis that ‘generative AI is useless’ - sure, maybe it’s not great for OA but it shows that the users are getting value out of generative AI, and just that they found a better way to get that value.
Personally, I would say that a simple test of OA users/activity would be to look at how they act. OA is presumably getting regular large shipments of GPUs installed into datacenters as fast as MS money can buy them; if OA usage is flat for several months—never mind crashing! - then they should be ‘enjoying’ a glut of GPUs by now and acting accordingly. Does OA look like it has an embarrassment of GPUs? Or does it look like it is struggling to add capacity as necessary to keep up with constant user growth, and holding back major improvements because it can’t afford them, and focusing on optimizing models (even to the detriment of quality) to get more out of its existing GPUs?
Some other possible explanations for why ChatGPT usage has decreased:
The quality of the product has declined over time
People are using its competitors instead
There’s a lot one could say about this claim:
Recall here the numbers here are substantially fake. The best they can tell you is roughly “this is very big” or “this is very small”. If you want to go much beyond that, you are reading sheep entrails. They are not coming from OpenAI but web traffic measurements. Such measurements are notoriously both noisy and highly biased and the biases change over time, and so unsurprisingly, at the time, OAers were saying it had overestimated the actual users by like 100%.
The numbers have lots of ways to be misleading. For example, because Chinese DL was, and still is, so inferior, there was a whole cottage industry of Chinese companies pirating accounts and black markets in credentials. This also applied to all the Third World or embargoed or difficult countries OA has denied access to, either for paying accounts or just period. Then you have people abusing it or sexing with it and getting banned and figuring out how to create new accounts, or moving on.
Just a huge amount of whac-a-mole going on. This obviously causes issues for interpreting any user metrics: a huge decrease in user count might actually reflect a huge increase in users, and vice-versa, depending on how the security arms race is going.
Summer vacation. If you look at the graph, you’ll notice a remarkable correlation with the Western academic calendar… Someone confidently proclaiming that ChatGPT use is crashing before seeing the September–November 2023 numbers is giving hostage to fortune.
EDIT: as of 1 October, SimilarWeb and other sources are reporting increasing traffic.
Many, many alternatives spinning up, many of which are using OA as the backend, particularly after the large price drops. These would count in a naive web-traffic approach as OA ‘losing users’, rather than gaining them.
More broadly, if they are going to true competitors like Claude-2 and do not count as OA users by any definition, that’s still damaging to Goia’s thesis that ‘generative AI is useless’ - sure, maybe it’s not great for OA but it shows that the users are getting value out of generative AI, and just that they found a better way to get that value.
Personally, I would say that a simple test of OA users/activity would be to look at how they act. OA is presumably getting regular large shipments of GPUs installed into datacenters as fast as MS money can buy them; if OA usage is flat for several months—never mind crashing! - then they should be ‘enjoying’ a glut of GPUs by now and acting accordingly. Does OA look like it has an embarrassment of GPUs? Or does it look like it is struggling to add capacity as necessary to keep up with constant user growth, and holding back major improvements because it can’t afford them, and focusing on optimizing models (even to the detriment of quality) to get more out of its existing GPUs?
Thanks for this. Very useful.