In the previous post, we started to address how to close one of the biggest organizational chasms in implementing an analytics and AI Governance solution — the why. Through our experience supporting dozens of clients across industries, geographies, and governance maturities with Dataiku Govern, we see organizations fall within five major reasons for needing governance: scaling strategy, risk/compliance, ethics, value growth, and operational efficiency.
This post will focus on the next two steps in the process, the what and how. Looking at your why from the previous post, we’ll start breaking down what needs to be done and then deliver a framework for how you might derive value from your solution. Let’s dive in!
What Needs to Be Done: Lay the Foundation for Robust Governance Processes
Hopefully at this point you have a clear understanding of your why and now we can start thinking about what needs to be done to effectively build robust analytics and AI Governance. Going back to the puzzle analogy in the previous post, the why allowed you to view the puzzle you are trying to put together, like the image on the box and your individual pieces all flipped right side up.
The what is like the processes you establish to find the pieces that fit together. Colors match, images coalesce and, as a result, edges and shapes have more value. Ultimately, the image (the why) helps you discern ways of piecing the puzzle together (the what): It serves as your north star and helps you generate an approach that serves what you ultimately want to accomplish.
Learning from our experiences, we’ve created a maturity staging structure that helps to qualify the state of play on the what. Below, the “Needs” roughly translate to the why (or the puzzle box image) and “Process” speaks to the what (or approaches to putting the pieces together). Assign your organization to one of these three stages:
Source: Dataiku
At any of the stages outlined above, it’s likely you won’t need to start from scratch! Even at Stage 1, data scientists might have had a few conversations with their manager to qualify projects informally; so part of building the what includes documenting that process and finding the right stakeholders.
If your organization is at Stage 2, you might consider starting with a list of the existing processes that are currently in flight to oversee and control your analytics and AI pipeline. Even if you have an established and defined analytics and AI Governance process at Stage 3, you might still need some help fine-tuning the process and implementing it successfully. With a tool like Dataiku Govern, most of the processes can be housed in a single platform which cuts down significantly on time, documentation, and hand-offs between tools.
What Are Your Governance Processes of Interest?
To clarify the existing governance processes you are trying to operationalize, centralize, or automate, here are some questions to align your what to your why:
- How many steps do you currently have in your process and do these steps make sense in relation to meeting your requirement(s)? Are there any gaps?
- What kind of information is required in your process to meet your requirement(s)?
- Do you have or need any review or/and sign-off stages to meet your requirement(s)?
- Who owns the process, each step, and the review or/and sign-off stages to meet your requirement(s)?
The answers to the above questions will look different for different organizations. In our experience, we have seen a number of important answers that might resonate. Here is a non-exhaustive list of processes that have been raised in response to discovering the what:
1. Compliance Reporting
2. Sensitive Data Qualification
3. Technical Model Validation
4. Project & Model Registry
These answers are your what; how are the processes going to be improved? How can existing, yet sub-optimal, processes be operationalized in Dataiku Govern to meet your why? Let’s find out!
How to Do It: Drive Value From Clear Business Objectives
In the last blog post, you set your vision for why your business needs a governance solution. In the previous section, you listed what processes need to be improved to meet your business objectives. In this section, we will dive into how you can realize the value in your process.
Back to our puzzle, this is where you physically put the puzzle together. Your grandmother is coming over today and the two of you have a plan. You’ve already flipped the pieces image-side up and started finding the edge pieces, you assign her to all of the blue pieces while you take on the reds. This process should be much faster than your baseline of trying every piece, every side until you find a match. I bet this streamlined process saves 30 hours of puzzling, and that’s significant value!
We’ll build on the what from the section above, bring in your current pain points to set a baseline, and then assign how those pains can be solved. The value gained is the gap between your existing process and the time, efficiency and cost benefits of addressing the pains with the ideal governance solution.
Existing Process Outcomes
For each of the processes listed in the previous section, clearly state the pains you hope to address by improving the process. It will help you understand how to generate improvements and establish ground zero to calculate value. For the examples above, the output might look like:
1. Compliance Reporting is discussed in meetings, often at a high level, but it is not standardized in documentation, nor are the steps needed to understand and mitigate risk clear.
2. Sensitive Data Qualification is happening via email and is hard to track if it wasn’t sent to you directly or if a third-party software is used to qualify. On the other hand, access permissions are handled ad hoc and sometimes not at all.
3. Technical Model Validation currently happens through asynchronous reviews and confirmed either verbally or with emails (see issue above) and systematic documentation is not happening for auditing or tracking purposes.
4. Project & Model Registry is maintained, perhaps with varying kinds of information about models or metadata, and lives in an unmonitored spreadsheet.
Ideal Process Outcome
To ensure your business objective is clear enough, note how each pain point can be alleviated for each process. This is where the value gain can be calculated. We’ve included some links and verbiage specific to Dataiku, but you don’t need to be a current user to see how alleviating these pain points brings value to your organization.
1. Compliance or Risk Factors are clearly stated in the project qualification and systematically tracked throughout a project workflow. Reviews and sign-offs are required before a model is pushed into production.
a. Documentation can be uploaded and customized to meet compliance requirements, there’s a clear timeline and responsibility stated each step of the way.
2. Sensitive Data Qualification is documented, centralized, and signed off on in a visible, accessible platform. As required, permissions are developed so the right people have access to such data and non-qualified data scientists are unable to access or use such data.
a. Audits documentation is centrally located, proper access can be monitored.
3. Technical Model Validation is automated through scenarios and model drift is automatically pulled into the platform for monitoring and sign off.
a. No longer need to ensure the right people are CC’d on the evaluation emails, no need to search through emails, or link them into additional documentation. The data is visible to anyone who has access to the project in the Dataiku Govern node.
4. Model Registry is automatically generated for your portfolio of projects and models and important metadata is continuously updated. Details in the view can drill down and expand to include sponsors, sign-offs, and data like drift metrics.
a. No reminders to fill out the spreadsheet, eliminates uncertainty if the list is updated, the process is now visible and transparent.
Bridging the Gap and What Lies Beyond Your Gap Analysis
Over the course of these three exercises, you’ve made some great progress in reducing your time to value. Your puzzle is complete — congratulations!
You’ve created your why — driving governance from clear business objectives. These business objectives helped you concretely list what your team will be solving for the processes that will support your why. The final step was to break down the what into actionable steps on how you might find value in your solution. This is where your value comes from, the difference between your current state and the art of possible when it comes to fully utilizing AI Governance capabilities to solve your business needs.
You are now able to avoid long back-and-forth between teams in the hope of clarifying governance objectives. You will also be able to make faster and more informed decisions on how to test the capabilities of interest, which ultimately brings value back to the organization through well-documented and streamlined processes.