MLOps is a term that is gaining a lot of traction, but definitions of the concept remain blurry and many misleading myths are circulating. In “The 7 Myths of MLOps,” a recent Dataiku Product Days session, we have aimed to set the record straight with some reality checks and myth-busting. This blog recaps the useful information revealed in the session.
What Is MLOps Exactly?
The concept of MLOps is a fairly young one and definitions vary so, for the sake of clarity, let’s keep it simple and land on this basic understanding:
MLOps is a set of best practices that aims to deploy and maintain machine learning models in production reliably and efficiently.”
By definition, without MLOps, AI projects risk never being deployed, and as a consequence never generate any actual business value. So organizations investing in AI and data need to properly leverage MLOps practices to have a chance of generating returns. With that said, before they can properly implement MLOps, organizations might need to first understand what is often misunderstood about MLOps. So, without further ado, let's bust some myths and see how Dataiku supports MLOps!
Myth 1: MLOps Is All About Models
When we talk about data projects and AI, we tend to focus on the model itself as being the outcome of the entire project. However, the reality is that modeling is just one specific subset of all of the work that has to be done in order to successfully launch data projects.
The overall project consists of many different but crucial parts beyond just ML— configuration, data collection, feature extraction, data verification, analysis tools, process management tools, machine resource management, serving infrastructure, and monitoring.
MLOps encompasses everything from data to models to systems in production, and data operations and MLOps actually need to work together in order for a model to be pushed to production and operationalized.
Myth busted: MLOps is not just about models. It is about so much more!
Myth 2: You Can Just Copy From Lab to Prod
The reason that so many projects are not successful is that organizations fail to anticipate that pipelines need to function not just in the environment of the teams who design the models but in the production environment as well.
Most often design teams will focus on the performance of their models, not on the portability of their work in the specific context of the production environment. When the projects are taken to production teams, an overwhelming (or unrealistic) amount of work is leftover. The model then ends up at the end of the queue and the relevancy of the project is compromised.
In addition to that, deploying ML projects into production is a critical task in its own right. Data scientists might start by experimenting with the data in their own environments, preparing it, and tackling field engineering, but when they go to put the model into production, a barrier exists. When the project enters the world of production systems, all of the artifacts that went into the production of the model must move over too in order to eliminate these barriers. Then, after the model is fully trained and up-to-date, real-life scoring and predictions will demand data to be cleaned at each and every step.
Myth busted: It is not just copying from lab to prod. It is about facilitating and streamlining the move across each specific environment.
Dataiku specifically assists this effort through the full end-to-end capabilities of data project management, facilitating the move of complete projects — from data connections and preparation to training and deployment.
Myth 3: Once You Are Done, You Are Done
It seems like once a model is put in production it is all set. Right? Actually, that is wrong.
The majority of MLOps is not about initial deployment, but rather about keeping the projects healthy in their state of production over time. You must think beyond deployment and into actual model lifecycle management. Even if the design process only takes a few weeks, the model will most likely continue to run for years, and it needs to continue to provide sustainable and relevant insights over time. MLOps, in the long run, becomes more about keeping projects healthy and providing lasting high-quality results.
Myth busted: MLOps will never end!
Don’t let that stress you out though. Dataiku provides many capabilities to support the full lifecycle of the project, continuously. For instance, Dataiku provides environments where a model has multiple versions and various development testing across production stages.
Myth 4: Accuracy Is the Only Important Measure
The reality is that MLOps has lots of other important metrics beyond accuracy. Refer to the image below to see the other metrics that are necessary for model health, beyond accuracy:
Myth busted: MLOps isn’t about accuracy alone. Accuracy actually comes second to pipeline and service health and data drift detection.
Dataiku offers robust monitoring for data pipelines and data drift as well as for accuracy, so you can choose what metrics work best for your specific project needs!
Myth 5: If It Breaks, We’ll Figure It Out
Teams just starting with MLOps tend to think that a model breaking is a rare occurrence. In reality, models break often(change in an upstream system, change in data, model drift…), and you cannot effectively fix these broken models if you do not have a premeditated plan ready to go.
As seen above, one plan you can have in place is to have backup models waiting in the wings to take the stage when your original model fails. Another critical point is that you need sufficient tools to identify the breaks. Finally, and very importantly, you must have a way to fix and update your models without disrupting downstream services so that you continue to serve results as needed changes are made!
Myth busted: MLOps is not about the impossible task of having unbreakable models. It is about planning ahead to make a break a non-eventful.
Dataiku monitors various metrics in the new model registry to see where you have issues. Then, you can dig into logs to see the origin of the issue. Dataiku also enables you to retrain and deploy quickly.
Myth 6: MLOps Is Engineering, You Don’t Need to Understand the Models
Many might believe that the production team can just take the models and push them into production without really understanding the purpose and process of the project. This can lead to decisions that are relevant from a technical standpoint, but that can induce changes in model behavior or biases in the system. They actually need to understand the underlying behavior and expectations in order to fix issues and avoid deploying any unexpected or inaccurate models.
Myth busted: Any operation on a data project, including all MLOps related ones, requires an understanding of the broader picture of the models.
Transparency and shared understanding across the project lifecycle is essential to ensure trustable operations. So, how does an organization create this transparency? Dataiku is a perfect example of what you should look for from design to production, with a clear view of every single step of the flow. Dataiku generates automatic model documentation so that you can understand model goals, stakeholders, transformations, and other important information with a complete view of the modeling process readily available to share. Additionally, explainability reports offered in Dataiku feature interactive models and explanations. These explainability reports give production teams a better overall understanding of how the model operates.
Myth 7: MLOps and AI Governance Are Redundant
It is a common misconception that MLOps is a part of AI Governance, and while it is accurate to say that the two are related, you should know that they are different functions.
MLOps is focused on service-level specification and usage in order to deliver high-quality results, whereas AI Governance is focused on managing risk and ensuring compliance. Both MLOps and AI Governance teams want healthy projects, but the two perspectives are entirely different.
Here is a chart for a better understanding of the relationship between the two:
Myth busted: MLOps and AI Governance are not redundant. They are actually both unique and essential functions.
Key Takeaways
To summarize:
- First, MLOps is about more than models as data and infrastructure are critical components of healthy modeling as well.
- MLOps is an ongoing process that requires project documentation and strategies for shared understanding across teams.
- Finally, AI Governance and MLOps are two separate but both necessary components for scaling AI.
Hopefully, after reading this blog, you feel like you have a more complete, clear, and comprehensive understanding of what MLOps is, the ways that MLOps fits into your AI goals, and how Dataiku supports these initiatives.