Evaluation 101: State of AI Key Takeaway 3
Products and programs get launched with bugs all the time--mobile updates to your favorite apps are a perfect example of a deploy and optimize approach. Often there are minor glitches that have little affect on the product or user experience, but sometimes there’s a bug that’s so apparent it takes the world by storm. People wonder how mistakes like these ever get passed quality assurance, and the answer is a simple one: not enough data or insufficient model evaluation and testing. If you don’t test your model, there's no real way to know if it functions properly when used by consumers.
A common misconception is models only need to be tested once to ensure they work properly. To truly be sure your model is as close to flawless as possible, it needs to be reevaluated every time an update is made. According to Our 2022 State of AI and Machine Learning Report, last year, 86% of organizations updated their models at least quarterly, with a directional increase to 91% this year. These constant updates indicate a need for a bigger focus on human-in-the-loop evaluation for AI models.
Our third key takeaway in our State of AI and Machine Learning report is focused on evaluation and how human-in-the-loop model evaluation is still essential in this day and age.
Model Evaluation Basics
There’s strong consensus around the importance of human-in-the-loop machine learning with 81% of respondents stating it's very or extremely important and 97% reporting that human-in-the-loop evaluation is important for accurate model performance. This is so vital for the success of machine learning that it’s actually the fourth and final step in our lifecycle for AI data.
Once a model is fully deployed, it’s nearly entirely autonomous except when additional validation and re-training is needed. Since new data points have to be added to create additional outputs, most models require re-evaluation on a near consistent basis.
While AI models are intended to automate problem solving and response for all scenarios, the entire process can be corrupted if the program learns incorrectly or is trained with bad data. That's where humans enter. Humans check over an annotated dataset and make sure it’s producing the desired outcome--often outcomes that mirror human decision making. If the outcome is correct, no action is needed. If the outcome is wrong, however, new data must be fed into the program and the incorrect data removed. The model then needs to be tested again until the right outcome is displaying. Once a model learns incorrectly, it will continue down that path until an outside force (aka humans) interject and create a course-correcting path.
Machines will make mistakes and will never, truly, replace humans as evaluation and training is a critical step to refining AI models.
Model Evaluation Challenges
For something as valuable to the success of ML models as model evaluation, it doesn’t receive the support it deserves. In our research for the 2022 State of AI report, we uncovered the fourth and final stage of the AI lifecycle receives the least budget allocation. Model evaluation is the stage that identifies inconsistencies in model outputs or if a program is or isn't functioning correctly. Non-functioning programs that make it to market will likely need to be reprogrammed which has a greater impact to budget than if proper model evaluation were included in the initial plan.
Another impactful challenge is the need to find a data partner that can provide the right quality and expertise to deliver the desired results for the AI model. In fact, 83% of those surveyed wish they could find a single partner to support all stages of the lifecycle. Not only can having one partner ensure the model is trained correctly the first time through, but it can provide much needed time and cost savings.
At Appen we’re proud that “Our unique ability to support all data-centric stages of the AI lifecycle for all data modalities positions Appen as the ideal external data provider.” Chief Product Officer, Sujatha Sagiraju
Learn More About Data for the AI Lifecycle
Model evaluation is critical to AI model success and industry experts share their thoughts in our 8th annual State of AI and Machine Learning Report. You can read the report today to better understand the current industry trends and challenges in relation to sourcing data, as well as read our other four key takeaways. For further information, watch our on-demand webinar, where we discussed in-depth all topics covered in our State of AI Report.