Crowd’s Collective Wisdom vs. Experts: Who Makes IBM Watson Smarter?

Understanding natural language is one of the great aspirations of artificial intelligence. While solving it could have exciting implications for society, it’s going to take an unprecedented volume of training data to make it happen. This is especially true in the interpretation of health data from a seemingly infinite array of sources — and a big reason why we’re so excited to see CrowdTruth.org leveraging CrowdFlower, now Appen, to train IBM Watson. Watson represents the state of the art in computational linguistics and computer vision put into action. It uses its never-before-seen understanding of language and imagery to comb through volumous datasets to unearth helpful information and make predictions (such as tips for disease diagnosis). According to Lora Aroyo, Principal Investigator of CrowdTruth, Watson acts as “a cognitive prosthetic to extend the decision making capabilities of an expert,” such as a doctor, who will utilize it as a tool for recommendations on how best to analyze a patient’s condition. In parallel, data enrichment platforms have become a valuable resource for data scientists looking to automate and scale the cleaning, data labeling, and enrichment of data using human intelligence for machine learning — ie., training data creation. While Watson continuously engages in active learning, its intelligence is strengthened by the quality of the training data it takes in from crowd contributors on data enrichment platforms such as Appen. IBM Watson Watson achieved noteriety when it won Jeaopardy a few years back. Image via Atomic Taco

What’s next for data enrichment and Watson?

Lora Aroyo from VU University Amsterdam, Chris Welty from the IBM Watson Research Center and Robert-Jan Sips of IBM Netherlands are leading the charge with CrowdTruth. Their work focuses on the labeling of training data that combines both subject matter experts and crowd contributors to strengthen Watson’s machine learning algorithms. What they’ve discovered is groundbreaking.   CrowdTruth Team The CrowdTruth Team CrowdTruth has found that expert annotators, highly paid health professionals working on training data creation, agree only 30% of the time. While the “popular crowd vote” covers 95% of expert agreement. Why? Experts don’t pay close attention to granularity in linguistic expression. Crowd contributors do. What this means, is that the collective intelligence of an unbiased crowd is as good, if not better, than expensive experts sitting in a room. The diversity of their annotations help Watson understand the details that expert trainers gloss over. I recently spoke with Lora Aroyo, who summarized CrowdTruth’s approach to training data creation: “The crowd of anonymous workers, typically unbiased from possible domain expertise, is processing text examples from a pure linguistic perspective, and in this way captures the diversity of interpretations provided by expert annotators.” In years past, the only path to clean training data was a laborious grant process and onboarding an army of contractors which was costly, time consuming and far from scalable. But today, researchers can simply switch Appen’s massive, on-demand workforce on-and-off to structure and extract knowledge from vast quantities of medical texts, images and videos. So much so, that CrowdTruth is proving that these layman data labelers are more effective at interpreting semantic content than paid professionals. The reason being, is that experts interpret data with pre-formed biases. Wheras crowd contributors, especially if you’re asking a bunch of them per data point, as CrowdTruth is, interpret data in a way that accounts for the spectrum of possible interpretations. Thus, Watson ingests a broader and more reliable vector to draw from as it computes predicitons. CrowdTruth’s training data complements medical expert inputs with the primary purpose of extracting insights from medical documents. By introducing people-powered data enrichment into Watson’s machine learning workflow, the system can supplement the need for expert trainers and more rapidly extract crucial knowledge from the likes of Wikipedia articles, patient case reports and beyond.

What Are the Implications of the CrowdTruth Research?

This framework will contribute to Watson’s ability to perform medical text analysis that promises to advance clinical research and the personalization of medical care in unprecedented ways. Imagine Watson being able to interpret a patient’s current symptoms and help prescribe the right treatment at blazingly fast speeds. How will do it? By synthesizing a doctor’s inputs alongside an automated analysis of a patient’s decades-old medical record, seasonal flu data for the area, and the universe of other medical knowledge amassed in Watson’s brain. What does that mean? Sick people are treated faster and better. On the clinical side, pharmaceutical companies can accelerate the development of drugs and researchers can dive deeper into their study of disease and genetics. In short, better medicine and more efficient clinical outcomes. By combining the expertise of medical professionals with the power of artificial intelligence, we very well could see these advances realized in our lifetimes. It’s an exciting prospect and we’re thrilled Appen is at the bleeding edge.

Dig Deeper:

If you’d like to dig deeper into CrowdTruth, check out the slideshow below, take a look at their GitHub, browse CrowdTruth’s team papers and presentations or take some time to read The Three Sides of CrowdTruth, that was recently published in the inaugural issue of the Journal of Human Computation.
 

More Articles Like This

All Articles

Blog

lady looking at computer with word bubbles on the left

Unraveling the Link Between Translations and Gender Bias in LLMs

Inclusiveness and harmlessness are two conditions integral to AI applications’ success and adoption. That, we cannot deny. But, inclusiveness and harmlessness are cultural aspects of our language and human relations. Not considering these specificities leads, at best, to quirky outputs but at worst, to biased and toxic situations. Here’s an example I recently explored comparing AI applications Chat GPT 3.5 …  

Blog

natural language processing

Natural Language Processing - Connecting The World through Language With AI

As young children, before we can even walk or talk, we’re listening. We’re hearing the sounds and vocalizations made by other people. We’re attaching those combinations of sounds to meanings, such as ‘mother’ and ‘door’, and learning to read the facial expressions of those around us to deepen our understanding of blocks of words. Then we go to school, start …  

Blog

Enhancing AI Efficiency: Streamlining AI Training Data with Workflows 

As the world becomes more and more data-driven, companies are finding it increasingly difficult to manage their ever-growing datasets. Data annotation is a crucial process in many industries, including machine learning, computer vision, and natural language processing. This is especially true for Large Language Models (LLMs), which require massive amounts of labeled text data to learn and improve. As the …  

Get in touch with Sales

Join our team!

Join our Crowd

Have a question? We’d love to help.

ImageImage
Join Our Crowd
@ 2023 Appen Limited
Legal Policies
Data Subject Access Request
@ 2023 Appen Limited
Legal Policies
Data Subject Access Request
Website for deploying AI with world class training data
Language
Chinese
Japanese