At Appen, we address the demand for high-quality data by sourcing, vetting, and training AI specialists across multiple domains – such as STEM, coding, and finance – so they can produce high-quality AI training data for our customers. We released our AI Detector feature in the Appen AI Data Platform in late December 2024 and, like many features we develop and implement in ADAP, the AI Detector addresses key concerns expressed by our customers; in this case, how to maintain control over the origin of their data. Appen’s AI Detector not only weeds out bad actors abusing the crowdsourcing system but also ensures that the data gathered with human-in-the-loop is genuinely crafted by humans. As a result, AI Detector addresses pressing challenges around AI data quality and regulation.
Quality at the core
Data is essential for model development and refinement, from reinforcement learning to specialized fine-tuning, but acquiring the necessary volume and quality of data poses challenges. For example, recent studies have demonstrated how models become corrupted when trained extensively on synthetic data, falling into what is known as the “Curse of Recursion.” Appen’s AI Detector mechanism is a safeguard, ensuring continuous monitoring of human output before delivering it to our customers which improves model performance, rather than risking collapse.
Regulations and compliance
The risk of not being able to monitor whether data is human-generated or not extends beyond quality concerns. Ensuring that AI training and evaluation data is human-verified is critical to maintaining the integrity and fairness of high-risk AI systems. The EU AI Act outlines detailed requirements for training and evaluation data in high-risk models and therefore it is essential that model builders in these sectors have transparent and reliable data supply chains. This policy emphasizes the need for data to be representative and error-free, which can only be guaranteed with human oversight, to mitigate the risks of AI systems perpetuating inaccurate, discriminatory, and even dangerous behavior.
How AI Detector works
Addressing quality and compliance issues is not a new challenge. Academia, for instance, has been hard at work on solutions to prevent students and researchers from submitting AI-generated work. While many tools aim to identify AI-generated text by analyzing linguistic patterns, such as the use of specific words and grammatical structures, Appen’s AI Detector relies on behavioral signals, evaluating risk at the author-level. This method factors in multiple signals, forming a body of evidence that ensures a fairer evaluation and a higher accuracy in assessing whether a submission was generated by a human.
Our AI Detector is designed to ensure our customers receive the high-quality data they need for their models by empowering our teams with a data-driven solution to AI-detection. Based on results from our benchmark studies, if we detect three submissions from the same contributor with a 92% or greater likelihood of being AI-generated, we flag all three units and the contributor. At this point, there is a 99% probability that one of these three units is AI-generated. The project manager then reviews these flagged submissions and makes an informed decision on the next steps.
Interested in implementing AI Detector in your Appen projects? Learn more in our success center article or speak with an Appen expert today.