With all the excitement about AI, it is easy to forget the hidden forces that can derail projects and programs. I am not talking about operations issues — those you can see and hopefully fix. In this instance, I am talking about trust. Trust from executives that data and analytics teams are "doing the right thing" and trust from business stakeholders who have to use predictions to make decisions. Executives who don't trust your program will cut back or pull the plug. If business stakeholders don't trust results, they won't use them, which might even be worse. In Dataiku 11, new capabilities continue to help build trust in AI projects across the organization.
Building Executive Trust With AI Governance
For executives, the critical elements are visibility, adherence to operational best practices, and legal and regulatory compliance. The word they need to hear is governance, not just data governance or IT governance, but governance for AI and analytics projects.
In Dataiku 11, three enhanced capabilities help drive project and model governance and trust to new levels. The first is the flow document generator, which provides a quick and easy way to create a detailed snapshot listing all the information about what's in a project flow at a given time. Along with the existing model document generator, this provides everything that companies need to document the current state of analytics projects and models. Additionally, integrated triggers and automation allow for easy snapshot generation with each project or model version to ensure compliance with internal policies and external regulations.
AI and analytics governance must also include best practice plans with appropriate stage gates, reviews, and final sign-off if needed. Governance plans should vary with different governance strategies for different types of data, analytics, and AI projects. Gartner research calls this “adaptive governance,” citing the need to ensure the desired business outcomes without killing innovation. In Dataiku 11, the governance module, originally introduced in Dataiku 10, expanded to include a project bundle registry and governance plans for data and analytics projects that don't have predictive models. An example of this practice is when an analytics team builds a data pipeline to support a business dashboard. With these new capabilities, executives and analytics teams can govern all types of data and analytics projects, apply appropriate plans, and view the status of projects from design through deployment.
Finally, increasing trust also means that models are proven resilient in real-world conditions. Validation and testing before models reach production are critical to creating trust. Model stress tests put models through a battery of stress tests simulating potential data quality issues to check behavior and performance under real-world conditions, such as missing input values or shifting feature/target distributions. By running the model through robustness checks before deployment, analytics teams can find and fix issues with models before they cause problems for business stakeholders in processes and applications.
Building Business Trust With What-If Analysis & Optimization
For business stakeholders, seeing is believing, and they know the data and the business problem. The results must be in context, and they must be able to play with model outputs visually to see that the results match their expectations. Who knows, they might even help improve model outcomes by recognizing inputs that might drive better results.
I mentioned this topic in a previous blog, but I repeat it here to help us understand how we can generate trust. Interactive what-if analysis is vital for business users to interrogate models by changing inputs and viewing outputs. Once users start changing values, they quickly begin looking for an optimal outcome based on their knowledge. Outcome optimization in Dataiku 11 shows the user the optimal and most plausible combinations of values that lead to better results. With this new information, business users can look for ways to change and influence processes to drive more desirable outcomes.
Building trust with executives and business stakeholders is not that hard with the right tools. For executives, the key is providing a framework to ensure that data, analytics, and IT teams follow operational and regulatory compliance best practices. The key for business stakeholders is providing ways to understand and interact with projects, like what-if analysis and outcome optimization. Both executives and business stakeholders need visibility into particular aspects of analytics projects to develop trust. With the right system in place, trust goes from becoming a hidden blocker to an asset that leads to successful projects and AI programs.