Being highly skeptical of this GPT-3 “research” myself, let me make a meta-contrarian argument in favor of ways that we could do more constructive GPT-3 research, without letting the perfect be the enemy of the good.
One way is to try and develop semi-replicable “techniques” for training GPT-3, and quantifying their reliability.
So for example, imagine somebody comes up with a precise technical method for prompting GPT-3 to correctly classify whether or not parentheses are balanced or not, and also for determining stop conditions at which point the run will be terminated.
If its overall accuracy was better than chance, even when used by multiple independent investigators, then its reliability could be quantified. Its broader validity would be harder to determine. But I think this would be a step in the direction of turning the study of GPT-3′s capabilities into more of a science.
Additional challenges would remain: lack of peer review, lack of meaningful incentives for integrity, lack of funding to drive sufficient attention, and so on.
Hopefully that at least gives some perspective on how far we all are from anything approaching the scientific study of GPT-3.
Being highly skeptical of this GPT-3 “research” myself, let me make a meta-contrarian argument in favor of ways that we could do more constructive GPT-3 research, without letting the perfect be the enemy of the good.
One way is to try and develop semi-replicable “techniques” for training GPT-3, and quantifying their reliability.
So for example, imagine somebody comes up with a precise technical method for prompting GPT-3 to correctly classify whether or not parentheses are balanced or not, and also for determining stop conditions at which point the run will be terminated.
If its overall accuracy was better than chance, even when used by multiple independent investigators, then its reliability could be quantified. Its broader validity would be harder to determine. But I think this would be a step in the direction of turning the study of GPT-3′s capabilities into more of a science.
Additional challenges would remain: lack of peer review, lack of meaningful incentives for integrity, lack of funding to drive sufficient attention, and so on.
Hopefully that at least gives some perspective on how far we all are from anything approaching the scientific study of GPT-3.