Observability: Moving Beyond Model Monitoring

Dataiku Product, Tech Blog François Sergot

Observability is a topic that has been floating around for a while, but are you able to clearly outline the differences between observability and model monitoring as well as understand the value of the actions behind this system of logic? In this blog, discover the distinctions and learn about the processes involved, or check out the full Dataiku Product Days session for a more in-depth description of the examples on which this blog is based:

→ Watch the Full Session

Monitoring is concerned mainly with specific metrics while observability moves beyond this scope and is a global approach that expands through and encapsulates your entire data ecosystem. 

By definition, observability is the measurement of how well the internal states of a system can be inferred from knowledge of its external outputs. The objective here is not only to know how the system is working but also to promptly identify and analyze any issues present. This involves looking into the entire business process from end to end among multiple systems and being able to point out and follow any issues that may pop up. 

observability

The Internal States of the System 

As a vast topic, we need to look at observability from various angles to understand it. In this context, we will look at it from ITOps, DataOps, and MLOps points of view.

ITOps 

Starting from the infrastructure, this is a layer where observability is the most mature and where there are existing tools. In this space, Dataiku can act as the observer or the observee.

As the observer, Dataiku can ingest many types of data and logs, so that its capacities can be used to detect issues, build dashboards and generate calls to action. Dataiku can effectively help you label and leverage logs, reconciling information coming from Kubernetes and other sources into a cohesive system. The platform also allows you to synchronize and cross-analyze applications and logs clearly to discover how your infrastructure is functioning as a whole. 

As the observee, Dataiku offers many endpoints to allow you to fetch not only its up status but also its detailed internal metrics and the logs of its activity.

dataiku health check

Here is an example of a specific monitoring instance in Dataiku: 

monitoring example

DataOps

Our second breakdown of observability is DataOps, dealing with monitoring your data pipeline. Here we pay special attention to the magic triad: scenario, metrics, and checks. Dataiku offers a lot of ways to compute and surface metrics including custom metrics to build your own reports. 

metrics

Dataiku dataset metrics is a very powerful addition to standard metrics. It is a versatile tool to build a central dashboard using any of the internal dataset models or custom metrics.

But, remember, observability is not metrics alone! When you have a problem on your hands, logs and traces, as well as reports and action, are key. In these areas, Dataiku natively offers navigation to your dataset or entire data pipeline structure as well as full visually organized logs on execution.

customers scoring

Then, to report and act on discoveries, you can automate the processing and alerting using Dataiku scenarios, and organize this work into valuable insights using dashboards and workspaces.

Here is a scenario example: 

scenario example

MLOps

Finally, let’s look at the MLOps view — exploring how well models meet project goals and serve business needs. 

As an end-to-end solution, Dataiku has its own capabilities for each of the MLOps stages. Monitoring and observability are one of these main steps. Be it for Dataiku visual models, custom models exported models, or even foreign models, you can do all the model observability in Dataiku leveraging our Model Evaluation Store.

model observability

Here’s a quick look at model evaluation in Dataiku where we go from simple and actionable high-level drift metrics into a more complete and detailed understanding of how models are behaving.

API

evaluation

evaluation

Taking Action 

Now that we have covered many aspects of observability in Dataiku, it’s time to move these elements into action. Check out the latest capabilities in the newest version of Dataiku to kickstart the process: https://www.dataiku.com/product/product-updates

You May Also Like

Alteryx to Dataiku: Working With Datasets

Read More

Demystifying Multimodal LLMs

Read More

I Have AWS, Why Do I Need Dataiku?

Read More

Why Data Quality Matters in the Age of Generative AI

Read More