Improved Search Quality From Microsofts' Bing In Multiple Markets

The Company

Microsoft Corporation is an American multinational technology company with headquarters in Redmond, Washington. It develops, manufactures, licenses, supports and sells computer software, consumer electronics, personal computers, and related services. Its best known software products are the Microsoft Windows line of operating systems, the Microsoft Office suite, and the Internet Explorer and Edge Web browsers. Its flagship hardware products are the Xbox video game consoles and the Microsoft Surface lineup of touchscreen personal computers.

The Challenge

A successful search engine user experience is primarily determined by the quality of the search results. High-quality search results are accurate, timely, comprehensive, free of spam and relevant to the search query intent. To improve the quality of the results of the Bing search engine, Microsoft is continually improving its technology. New ideas are tested on large-scale data sets developed by evaluators who review and rate query and result pairs. The Bing team develops general and specialized search engine experiences for the U.S. and international markets so Bing results are culturally relevant.

The Solution

During an initial trial project, we were a proactive and agile partner for Microsoft in the U.S. market. In addition to assembling and training a team of linguistic resources for the project within weeks and quickly surpassing the established quality bar, we also provided recommendations for improving the evaluation process.As a result of this successful first project, Microsoft expanded our involvement. We since have processed millions of pieces of search data every month in more than a dozen markets worldwide with ever-improving quality.We helped the Bing team expand by rapidly ramping up new markets. In early 2010, for example, we hired and trained dozens of data annotators within two months. Despite changing priorities throughout the project, not only were we able to meet all deadlines, but the group also added infrastructure to make the project sustainable and to constantly improve quality.We developed a proprietary data analysis and reporting tool to ensure efficient and consistent reporting in all markets. The training department designs and delivers hundreds of training hours every month to annotators all over the globe. The project management team, organizationally structured to accommodate growth, manages the day-to-day logistics, and program managers focus on operations across markets.As the Bing team continues to experiment with new ideas for search experiences, we assist by interactively working with Microsoft teams to develop, test with human judges, and suggest guideline changes to improve data quality.

The Result

We provide data collection support, allowing developers to benchmark performance and quickly enter and grow new markets. Our organizational structure, with in-house recruiting and training departments, is built to accommodate rapid growth and adapt to changes.Whether it is adjusting resource allocations or moving into new markets rapidly, we are able to meet the Bing team’s worldwide needs in an ever-changing environment.We are a proactive partner that delivers above expectations and suggests process improvements in order to maximize every client’s success. We combine these processes with an extensive global network of specialized resources, a flexible staffing strategy, and superior project and program management to deliver high-quality data sets.