Can You Trust AI Detectors?
Large language models (LLMs) such as ChatGPT have been with us for over a year now. For legal copywriters and content marketing specialists, the fear of LLMs putting us out of business has mostly disappeared. For specialist writing such as law, it is obvious to those who care about the quality of their content that LLMs are not up to scratch. Just as a general copywriter cannot write legal copy that meets the standards of most Solicitors and Barristers, LLMs produce bland, unreferenced, and inaccurate copy which is totally below the benchmark required of professional legal writing. However, legal copywriters and content marketers now face another challenge, one I believe is far more insidious – being accused of using LLM tools to produce content. Many clients have taken to running copy through AI detectors and questioning their copywriter about whether or not they actually wrote the content.
The problem with this seemingly logical action by a client (after all, no one wants to pay £200 odd for an article that the writer has simply taken 10 minutes to put through an LLM and lightly edited or risk Google changing its algorithms to downgrade their website because it contains AI content) is that it destroys trust on both sides. Once the seed of ‘cheating’ has been placed in the clients head, it is extremely difficult to dislodge. After all, an algorithm says that some or all of the copy has been created by an LLM; therefore, it must be true. On the writers side, being accused that piece of work they have spent hours creating is AI generated is quite frankly, insulting. And given most freelance specialist writers are a) self-employed, and b) have plenty of work to be getting on with, there is a risk that they will simply drop the client. In such a situation, everyone loses.
How accurate are AI detectors?
Let’s cut to the chase, how accurate are AI detectors? The short answer is, not very. According to an investigation by the Washington Post, AI detectors like Turnitin and GPTZero suffer from false positives that can accuse innocent writers of using LLM tools. Worse still, if your copywriter is a non-native English speaker, the chances of their work being flagged as AI generated is even greater.
Most of the studies conducted on the accuracy of AI detectors have been done in the academic sphere, as students are regularly accused of AI cheating. Study after study shows when applied to human-written control responses, AI detection tools exhibited inconsistencies, producing false positives and uncertain classifications. Furthermore, Pegoraro et al. (2023) found that detecting ChatGPT-generated text (the most commonly used LLM) is highly challenging, with the most efficient tool achieving a success rate of less than 50%.
Advice for law firms worried about copywriters using LLMs
The fact is that those requiring specialist content such as legal, medical, or scientific have no need to worry about their copywriter using LLM tools to create content because these tools simply cannot produce the standard and accuracy of copy required. Most legal copywriters (or at least the ones you should be using) have a law degree and access to free and subscription-based case law, legislation, and commentary. And most importantly, they became legal writers because they love researching and writing. Why would they use a tool that takes away the very aspect of their work that brings them joy?
Final words
Just as LLMs are not adequate when it comes to writing specialist copy, AI detectors are fundamentally flawed and bring up false positives. You have probably heard that if you ask an AI detector to check the American Constitution, it comes up at over 90% probability of being written by AI. I did my own test and I can confirm that this is true. Now, unless James Madison had access to a time machine, we can assume that this is a false positive.
The lesson is that we need to start trusting our own judgment again. Do we really want to live in a world where what a computer says is gospel? I certainly don’t and I cannot imagine you do either. Otherwise, what is the point of human beings at all.
The Legal Copywriting Company is dedicated to helping law firms and barristers achieve their content marketing goals by creating engaging, SEO friendly copy for their websites and marketing materials. To find out more, please call 01691 839661 or email corinne@www.thelegalcopywritingcompany.co.uk.