Success Stories:

Leading Search Engine Expands Internationally with Vendor-Neutral Quality Analysts

The Situation

A leading multilingual search engine provider had worked with Appen for a few years on search relevance evaluation. In that time, Appen had proven to be a reliable source for high-quality data. The firm partnered with Appen to develop quality metrics, improve rating guidelines, and increase the success of evaluators in production.

When the provider decided to invest in building out its search engine across 25 markets, it determined it did not have the in-house expertise to provide quality feedback to all the vendors it was working with. It needed a vendor neutral team of quality analysts who would keep the evaluators across all vendors in sync.

The Solution

The search engine firm turned to Appen for its proven expertise in quality management to provide a team of quality analysts. Appen identified its strongest, most experienced evaluators and auditors as candidates for inclusion in the new team. Each candidate had to prove not only their ability to apply the rating guidelines consistently and accurately, but also had to demonstrate strong communication skills, objectivity, and critical thinking. The initial team consisted of two and later expanded to seven.

To ensure the neutrality of the quality analyst team, Appen kept the management of the QA team completely separate from the management of the associated programs. A vendor manual was also developed outlining the expectations for all vendors so that preference or favoritism was not given to one vendor over another.


The search engine provider significantly improved the reliability of its search results and the quality of its data collection due to the vendor-neutral quality analyst team provided by Appen, even though data came from multiple vendors. The team ensured that all vendors received targeted feedback and that they were held to the same standards, raising the level of quality across the board. The program developed by Appen allows the client to get consistent performance despite using multiple vendors.

Key Success Factors

Appen’s team of seasoned judges and auditors provided the foundation for this program and is the reason for its continued success. The team provides objective feedback across multiple vendors and manages multiple communication streams to provide a single voice back to the client. Guidelines are also updated regularly to reduce confusion among the judges and to ensure that all judges are rating in the same fashion. With this clear and objective approach, Appen reduces the noise in the data. This has led to a strong program with stable metrics that continues to thrive today.

A leading multilingual search engine provider ready to invest in building out its search engine across 25 markets teamed up with Appen due to proven track record as a reliable source for quality management and high-quality data.



Find out how Appen

can help you meet your goals