You’ve probably already used them before, websites like GPTZero, ZeroGPT, Grammarly, Quilbot, all have their own AI Detectors. AI Detectors can be a combination of Pre-trained LLMS, Statistical Models, and Ml models using NLP(Natural Language Processing). The model will analyze linguistic patterns, sentence structures, and statistical measures like perplexity, the predictability of text, and burstiness, the sentence variety, to distinguish AI-generated content.
The Flaws
False Positives: Human-written text can be misclassified as AI-generated. This generally happens when there is minimal perplexity and burstiness. This translates to a simple and monotonous text which although AI like, might not have been written by AI.
False Negatives: Models like GPT-4 can produce text that passes as human-written, leading to undetected AI content. Recently Anthropic released Claude 3.5 Sonnet, but additionally and more recently they released their writing style function where you can add a certain writing style based on sets of text which it can replicate. AI will continue to improve its writing capabilities especially as scaling grows, so false negatives will be more apparent as time goes on.
Over-Reliance on Metrics: Measures like perplexity and burstiness can misclassify creative or technical writing. Like I mentioned before these are two of the main metrics all AI detectors use to distinguish AI writing. Text with low or consistent perplexity and burstiness are commonly marked as AI writing. But almost all academic writing also happens to fit that criteria, which is why academic research papers can easily be marked as AI when they are clearly not.
Transparency Issues: Many detectors lack clear explanations for flagged content, reducing trust and usability. It is important for us to understand that AI/ML models are essentially black boxes that are not interpretable to us humans whatsoever. In fact there are entire topics dedicated to making AI responses more interpretable such as Mechanistic Interoperability and Chain of Thought Reasoning(I won’t go in detail but feel free to check them out they are fascinating). All in all not even the developers know with %100 certainty why their model says what they say.
How should you use AI Detectors
do not use AI to write full essays in the first place(for now). If you only sprinkle in AI there is no reason as to why someone should be able to make a reasonable accusation of your writing being AI. But even better just write the essay yourself and don’t stress since you are a honest writer.
Use multiple detectors. AI detectors generally share the same evaluation techniques but there are key differences between each detector that make it unique. Use maybe 3 or 4 ‘reputable’ detectors that give good performance/accuracy and evaluate the same piece of text using them. You only need to worry if multiple/majority are returning extreme values. For example if 2 return 100% and the other 2 return 0% that is still highly suspicions of AI writing. If they all return 50-60 percent there’s not much need to worry about anything. But even then there is still good chance of false positive/negative, just a little less than using just 1 detector.
How should Admission Officers use AI Detectors
AI detectors are at a stage of development where they still give a significant amount of false positive/negative predictions. If you base your evaluations of someone’s personal statement off of an AI detector you are being extremely irresponsible. Judging how much AI is in someone’s essay based on your own judgment can be even more irresponsible. And combining the two will not improve your accuracy I assure you. I would suggest only worrying about the quality of the essay and keep AI out of your mind(for now). Generally AI written personal statements are bad to begin with. This is likely what’s best for both the AO and the applicant. Applicants will not have to worry about any bullshit evaluation an AI detector gives, and you as an AO will have one less thing to worry about when evaluating essays.
On AI Detectors Regarding College Applications
What are AI Detectors
You’ve probably already used them before, websites like GPTZero, ZeroGPT, Grammarly, Quilbot, all have their own AI Detectors. AI Detectors can be a combination of Pre-trained LLMS, Statistical Models, and Ml models using NLP(Natural Language Processing). The model will analyze linguistic patterns, sentence structures, and statistical measures like perplexity, the predictability of text, and burstiness, the sentence variety, to distinguish AI-generated content.
The Flaws
False Positives: Human-written text can be misclassified as AI-generated. This generally happens when there is minimal perplexity and burstiness. This translates to a simple and monotonous text which although AI like, might not have been written by AI.
False Negatives: Models like GPT-4 can produce text that passes as human-written, leading to undetected AI content. Recently Anthropic released Claude 3.5 Sonnet, but additionally and more recently they released their writing style function where you can add a certain writing style based on sets of text which it can replicate. AI will continue to improve its writing capabilities especially as scaling grows, so false negatives will be more apparent as time goes on.
Over-Reliance on Metrics: Measures like perplexity and burstiness can misclassify creative or technical writing. Like I mentioned before these are two of the main metrics all AI detectors use to distinguish AI writing. Text with low or consistent perplexity and burstiness are commonly marked as AI writing. But almost all academic writing also happens to fit that criteria, which is why academic research papers can easily be marked as AI when they are clearly not.
Transparency Issues: Many detectors lack clear explanations for flagged content, reducing trust and usability. It is important for us to understand that AI/ML models are essentially black boxes that are not interpretable to us humans whatsoever. In fact there are entire topics dedicated to making AI responses more interpretable such as Mechanistic Interoperability and Chain of Thought Reasoning(I won’t go in detail but feel free to check them out they are fascinating). All in all not even the developers know with %100 certainty why their model says what they say.
How should you use AI Detectors
do not use AI to write full essays in the first place(for now). If you only sprinkle in AI there is no reason as to why someone should be able to make a reasonable accusation of your writing being AI. But even better just write the essay yourself and don’t stress since you are a honest writer.
Use multiple detectors. AI detectors generally share the same evaluation techniques but there are key differences between each detector that make it unique. Use maybe 3 or 4 ‘reputable’ detectors that give good performance/accuracy and evaluate the same piece of text using them. You only need to worry if multiple/majority are returning extreme values. For example if 2 return 100% and the other 2 return 0% that is still highly suspicions of AI writing. If they all return 50-60 percent there’s not much need to worry about anything. But even then there is still good chance of false positive/negative, just a little less than using just 1 detector.
How should Admission Officers use AI Detectors
AI detectors are at a stage of development where they still give a significant amount of false positive/negative predictions. If you base your evaluations of someone’s personal statement off of an AI detector you are being extremely irresponsible. Judging how much AI is in someone’s essay based on your own judgment can be even more irresponsible. And combining the two will not improve your accuracy I assure you. I would suggest only worrying about the quality of the essay and keep AI out of your mind(for now). Generally AI written personal statements are bad to begin with. This is likely what’s best for both the AO and the applicant. Applicants will not have to worry about any bullshit evaluation an AI detector gives, and you as an AO will have one less thing to worry about when evaluating essays.