In today's rapidly evolving business landscape, harnessing the power of machine learning (ML) models has become a critical factor in gaining a competitive edge. However, merely deploying models isn't enough; ensuring their continued relevance and impact demands a comprehensive approach to monitoring and management. This is where functional monitoring steps in – a dynamic process that not only evaluates model performance but also translates it into actionable insights for business sponsors and stakeholders.
Plus, as the number of projects deployed to production increases, the sustainability of models gains paramount importance. A model’s accuracy means little if it fails to drive meaningful business outcomes. This underscores the need for sustained model management, encompassing regular checks for drift, adaptation to shifting data, and alignment with evolving business goals.
Functional Monitoring
Functional monitoring is used to convey key functionality of the business model’s performance to the business sponsors/owners. From a business perspective, functional monitoring is critical because it provides an opportunity to demonstrate the end-results of your predictive model and how it impacts the product. The kind of functional information that can be conveyed is variable and depends largely on the industry and use case. Examples of the kind of data displayed can include the number of contacts in a case, the number of broken flows in a system, and measurements of performance drifts.
Real-World Usage
Some applications of functional monitoring based on industry or functional need include:
- Fraud: The number of predicted fraudulent events, the evolution of the prediction’s likelihood, the number of false positive predictions, and rolling fraud figures;
- Churn Reduction: The number of predicted churn events, key variables for churn prediction, and the efficiency of marketing strategies towards churners (e.g., opening rates of emails);
- Pricing: Key variables of the pricing model, pricing drift, pricing variation over time, pricing variation across products, evolution of margin evaluations per day/year, and average transformation ratios.
In most situations, business sponsors must have the capability to detect early signs of drift. This is possible by enabling sponsors to easily review the model’s characteristics, view its history, determine the drift’s validity, and then take appropriate action. For example, a marketing campaign with more customers from a certain age group (e.g., 20-30 years old) could result in an inaccurate transformation ratio prediction due to the relative inconsistency of that group’s consumer behavior.
In addition, business sponsors must be provided with access to high-level technical errors. For example, if a pricing model is lacking data from a specific category, the business owner needs to be notified of the missing data so that they are aware of factors that impact their strategies.
Why Is This Important?
Knowledge transparency must be constantly shared and evangelized throughout an organization at every opportunity. A lapse in communication can compromise the importance and the value of using machine learning technology within your organization.
The Solution
“One day, our CEO spotted a funny recommendation on the company website. We realized that part of the rebuild chain had been broken for five days without anyone noticing. Well, we decided to keep this to ourselves.”
A successful communication strategy lies at the heart of any effective organization; such a strategy typically combines multiple channels:
1. Channel for the quick and continuous communication of events — these are channels where events are seamlessly communicated to team members, such as:
- New model in production; outliers in production
- Drop or increase in model performance over the last 24 hours
2. E-mail-based channel with a daily report. Such a report should be a succinct summary of key data, such as:
- A subject with core metrics
- Top n customers matching specific model criteria
- Three model metrics (e.g., a technical metric, high-level long-term metric, and a short-term business metric)
3. A web-based dashboard with drill-down capability; other channels should always include links to the dashboard in order to drive usage.
4. A real-time notification platform, such as Slack, is a popular option that provides flexible subscription options to stakeholders. If building a monitoring dashboard, visualization tools such as Tableau and Qlik are popular as well.
Sustainable Model Lifecycle Management
Previously in this article, we’ve discussed issues such as model monitoring and business sponsor involvement, but as the number of analytics workflows deployed to production increases exponentially, the issue of sustainability grows in urgency and importance.
We can simplify the journey from a prototyping analytics capability to robust productionized analytics with the following steps:
- Deploying models and entire workflows to the production environment in a fast and effective manner;
- Monitoring and managing these models in terms of drift, and retraining them either regularly or according to a predefined trigger; and
- Ensuring that the models in production continue to serve their purpose as well as possible given changes in data and business needs.
This last point is one that most organizations haven’t struggled with or even really encountered, but it’s vital to keep in mind now, because sustaining the lifecycle of models in production is the price of successfully deploying and managing them.
Why Is This Important?
Model management is often concerned with the performance of models, and the key metrics are generally related to the accuracy of scored datasets. But the usefulness of a model is measured in terms of business metrics -- that is, if a model has excellent accuracy, but it has no business impact, how could it be considered useful? An example could be a churn prediction model, which accurately predicts churn but provides no insight into how to reduce that churn.
Even with measures of accuracy, sustainability becomes an issue. Regular manual checks for drift, even if conducted monthly and in the most efficient manner, will soon become unwieldy as the number of models that needs to be checked multiplies. When you add monitoring for business metrics, the workload and complexity is even more daunting.
And finally, data is constantly shifting. Data sources are being changed, new ones are added, and new insights develop around this data. This means that models need to be constantly updated and refined in ways that simple retraining doesn’t address, and this is where the bulk of your team’s effort on sustainability will need to be focused.
The Solution
In order to manage the lifecycle of models in a sustainable way, as well as to extend the lifecycle of these models, you need to be able to:
- Manage all of your models from a central place, so that there is full visibility into model performance. Have a central location where you measure and track the drift of models via an API, and to the fullest extent possible provide for automated retraining and updating of these models;
- Build webapps and other tools to evaluate models against specific business metrics, so that everyone from the data scientists designing the models to end users of analytics products are aligned on the goals of the models; and
- Free up the time of data scientists and data engineers to focus on making models better and not only on addressing drift and lagging performance of existing models.
Charting a Sustainable Path Forward
Sustaining the lifecycle of ML models necessitates a central management hub for holistic visibility, automated retraining, and performance tracking. Building tools to assess models against business metrics ensures alignment across stakeholders. Ultimately, freeing up the resources of data scientists and engineers enables them to focus on model improvement rather than battling drift and lagging performance.
In a landscape defined by agility and innovation, functional monitoring emerges as a linchpin for unlocking the true potential of ML models. It not only ensures their relevance and impact but also empowers businesses to make informed decisions and adapt to evolving realities. As industries continue to evolve, the symbiotic relationship between functional monitoring and business success will undoubtedly remain a driving force in the realm of machine learning.