Appen provides resources, training, and project management for search engine evaluation, supporting a large-scale data collection effort to help improve the quality of the Bing search engine in multiple markets.


A successful search engine user experience is primarily determined by the quality of the search results. High-quality search results are accurate, timely, comprehensive, free of spam and relevant to the search query’s intent. To improve the quality of the results of the Bing search engine, Microsoft Corporation is continually improving its technology. New ideas are tested on large-scale data sets developed by evaluators who review and rate query and result pairs. The Bing team develops general and specialized search engine experiences for the U.S. and international markets so Bing results are culturally relevant.


During an initial trial project, Appen distinguished itself as a proactive and agile partner for Microsoft in the U.S. market. In addition to assembling and training a team of linguistic resources for the project within weeks and quickly surpassing the established quality bar, Appen also provided recommendations for improving the evaluation process.

As a result of this successful first project, Microsoft expanded Appen’s involvement. Appen now processes millions of pieces of search data every month in more than a dozen markets worldwide with ever-improving quality.

Appen helped the Bing team expand by rapidly ramping up new markets. In early 2010, for example, Appen hired and trained dozens of data annotators within two months. Despite changing priorities throughout the project, not only was Appen able to meet all deadlines, but the group also added infrastructure to make the project sustainable and to constantly improve quality.

Appen developed a proprietary data analysis and reporting tool to ensure efficient and consistent reporting in all markets. The training department designs and delivers hundreds of training hours every month to annotators all over the globe. The project management team, organizationally structured to accommodate growth, manages the day-to-day logistics, and program managers focus on operations across markets.

As the Bing team continues to experiment with new ideas for search experiences, Appen assists by interactively working with Microsoft teams to develop, test with human judges, and suggest guideline changes to improve data quality.


Appen provides data collection support, allowing developers to benchmark performance and quickly enter and grow new markets. Appen’s organizational structure, with in-house recruiting and training departments, is built to accommodate rapid growth and adapt to changes.

Whether it is adjusting resource allocations or moving into new markets rapidly, Appen is able to meet the Bing team’s worldwide needs in an ever-changing environment.

Appen is a proactive partner that delivers above expectations and suggests process improvements in order to maximize every client’s success. Appen combines these processes with an extensive global network of specialized resources, a flexible staffing strategy, and superior project and program management to deliver high-quality data sets.