Leading Search Engine Expands Internationally with Vendor-Neutral Quality Analysts
The Company
A leading multilingual search engine provider ready to invest in building out its search engine across 25 markets teamed up with us, thanks to proven track record as a reliable source for quality management and high-quality data.
The Challenge
A leading multilingual search engine provider had worked with us for a few years on search relevance evaluation. In that time, we had proven to be a reliable source for high-quality data. The firm partnered with us to develop quality metrics, improve rating guidelines, and increase the success of evaluators in production.
When the provider decided to invest in building out its search engine across 25 markets, it determined it did not have the in-house expertise to provide quality feedback to all the vendors it was working with. It needed a vendor neutral team of quality analysts who would keep the evaluators across all vendors in sync.
The Solution
The search engine firm turned to us for its proven expertise in quality management to provide a team of quality analysts. We identified its strongest, most experienced evaluators and auditors as candidates for inclusion in the new team. Each candidate had to prove not only their ability to apply the rating guidelines consistently and accurately, but also had to demonstrate strong communication skills, objectivity, and critical thinking. The initial team consisted of two and later expanded to seven.
To ensure the neutrality of the quality analyst team, we kept the management of the QA team completely separate from the management of the associated programs. A vendor manual was also developed outlining the expectations for all vendors so that preference or favoritism was not given to one vendor over another.
The Result
The search engine provider significantly improved the reliability of its search results and the quality of its data collection due to the vendor-neutral quality analyst team we provided, even though data came from multiple vendors. The team ensured that all vendors received targeted feedback and that they were held to the same standards, raising the level of quality across the board. The program we developed allows the client to get consistent performance despite using multiple vendors.
Our team of seasoned judges and auditors provided the foundation for this program and is the reason for its continued success. The team provides objective feedback across multiple vendors and manages multiple communication streams to provide a single voice back to the client. Guidelines are also updated regularly to reduce confusion among the judges and to ensure that all judges are rating in the same fashion. With this clear and objective approach, we reduce the noise in the data. This has led to a strong program with stable metrics that continues to thrive today.