Every organization strives to become a disruptor, rather than be disrupted themselves. Not surprisingly, they see the longed-for land of milk and honey in AI — and rightly so. According to McKinsey, the estimated gains are significant. By building machine learning (ML) into processes, leading organizations increase process efficiency by 30% or more while increasing revenues from five percent to 10%. And, indeed, customer use cases often confirm the immediate impact of ML models in predictive analysis and customer-propelled purchase scenarios.
However, it's not that simple. Several blockers prevent organizations from overcoming the difficulties encountered when industrializing AI. According to Gartner, “Gartner research has found that only half of AI projects make it from pilot into production, and those that do take an average of nine months to do so.”* This figure speaks for itself.
In the previous blog post from this series, we saw how crucial it was to define a governance framework. Being clear about rules, requirements, and processes, as well as establishing individuals’ roles and responsibilities, supports effectiveness in the model's value chain. Now that we've defined the "why" and the "what" in our first pillar, let's focus on the "how" in Pillar #2 — the challenges and limitations of scaling AI (and how to remove friction along the way).
Put Models Into Production Through an Agile, Trusted, & Controlled Workflow
Now, it is time to operationalize your AI project, i.e., schedule the model lifecycle stages according to the rules defined in the governance framework by members. Unfortunately, some frictions still exist. So, as we group the model industrialization process into four main steps (prepare, build, deploy, and monitor), we’ll also focus on significant frictions encountered along the way and ways teams can remove them.
We spent too much time with bad data and misleading expectations." (Friction in the Preparation Stage, #1)
Set Clear Expectations From the Start to Go in the Right Direction
In the preparation phase, the business SME, data engineers, and data analysts should clarify the business intent of the model, look for the right datasets, and prepare them for development use. The business SME will explain the business intent in a model description, describing what we plan to do, who's working on it, and sponsoring it — all according to defined governance rules. No one wants to move faster if the model is going in the wrong direction, but this often happens in many organizations.
The data engineer will collect and profile data from the primary sources available through data discovery. Plus, they will prepare it for better use. To improve efficiency, a data engineer will delegate this step to a business/data analyst as long as they have self-service data preparation tools to offer to non-technical experts.
The Team Should Rely on Clear Expectations and Better Data Foundations
Putting data preparation in the hands of non-technical people frees experts from repetitive and tedious data preparation tasks. This way, they can focus on more challenging tasks while equipping business or data analysts with the tools to prepare data efficiently. The result is faster, more efficient, and more agile teams.
They could also make it faster using features. Feature stores are a central place to build, manage, and share features across different teams in the organization. Feature stores can be critical to helping organizations save time and resources on rebuilding the same features and ensuring consistency and accuracy.
Only two of us at the organization have the ability to design the model and check it for robustness and bias — and we're spending too much time on it." (Friction in the Build Phase, #2)
Focus on a Universal Platform and Leverage Automation
Your experts' time is often far too valuable to be wasted on tasks that other capable and more available people could do. Having a platform capable of meeting the needs of the most demanding coders while involving non-coders in the model building process remains an undeniable asset. Working on the same platform also allows teams to gain project visibility by making datasets available to a group and not just to an individual. This ambivalence is not easy to find.
Often, AI platforms remain in experts' hands without putting less-technical people in the loop. However, success and gains in operationalization come from getting them to work together in this "build" phase, leveraging the power of automation. Having automated capabilities and quality checks will significantly accelerate the design phase and better prepare the model for deployment by ensuring effortless deployment.
Bad models proliferate without control. We struggle to deploy models into production without creating incremental value." (Friction in the Deployment Phase, #3)
Ensure the Model & Analytics Are Safely Deployed With Minimal Effort and Maximum Value
In Pillar #1, we saw how critical it is to define the roles and responsibilities from the get-go. How does this materialize in a model industrialization process? Placing a sign-off cursor before deployment will authorize this person to deploy the right models and stop those that do not meet ethical and technical rules. This approval is often put aside in the deployment stage. However, it is increasingly critical to curb the proliferation of unreliable or unbiased models.
You must also facilitate the deployment in order to avoid calling IT for help. This responsibility remains under the supervision of an ML engineer (or, if necessary, a data engineer in agreement with IT) to orchestrate the industrialization of the models without unnecessarily burdening the production with low-performance models.
The deployment also means sharing and consumption. We will cover a wide array of consumption scenarios — from analytical dashboards to decision-making models or feeding web services via apps; the more the model is consumed and shared, the greater its value.
We waste too much time assessing model performance with no extra help." (Friction in the Monitoring Phase, #4)
Want to remove that friction? Leverage automated monitoring capabilities to decide to keep the best performing models in production. Alerts and controls should be part of your model's workflow to decide if, what, and when to retrain your models. Having the right indicators to measure the performance of a model continuously must be part of an automated process that puts people at the heart of the decision-making process.
- Does the model behave as expected?
- Is it drifting?
- Is the training data better than the deployed model?
It would be best to have these answers to decide what to do next. Sometimes data scientist teams develop models on different tools or other places, and they should do benchmarking, monitoring, and performance assessment on all existing models regardless of model origins. You should monitor them with the same criteria, whether designed differently or on other data warehouses like Snowflake or AWS.
MLOps team members must automate the indicators to decide to keep the models in production or retrain them if necessary. Even if the operation remains in the hands of an Ops manager or ML engineer, you should be able to have equivalent indicators adapted to the needs of the risk or compliance manager.
In Summary
In this blog post, we covered some frictions and steps of the model workflow built upon the rules and responsibilities of the model lifecycle as defined in Pillar #1. We have highlighted frictions and proposed recommendations for improvement along the model lifecycle. In our last and final blog post, we will see why and how to implement a centralized approach to tracking all models through their lifecycle stages.
*Gartner Press Release "Gartner Identifies Four Trends Driving Near-Term Artificial Intelligence Innovation," September 7, 2021 https://www.gartner.com/en/newsroom/press-releases/2021-09-07-gartner-identifies-four-trends-driving-near-term-artificial-intelligence-innovation GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved.