With generative AI's growing importance, enterprises must address various challenges to ensure security, scalability, and compliance when implementing GenAI. Organizations that are succeeding in bringing GenAI to enterprise scale leverage Dataiku’s LLM Guard Services as the cornerstone of their strategy. These Guard Services enhance the Dataiku LLM Mesh by providing guardrails that help control cost, maintain quality, and reduce operational risks.
As seen above, there are three Guard Services.
- Cost Guard: Monitor LLM usage and expenses across teams with real-time dashboards and detailed logs. Set quotas and alerts to prevent budget overruns and facilitate internal cost allocation.
- Safe Guard: Automatically detect and handle sensitive data, toxic content, and forbidden terms in LLM inputs and outputs. Actions include redaction, blocking, and administrator notifications to maintain data privacy and compliance.
- Quality Guard: Evaluate LLM performance using standardized metrics and side-by-side comparisons. Ensure your AI applications deliver accurate, consistent, and unbiased results from development to production.
In addition to these foundational Guard Services, Dataiku is excited to expand our ecosystem of LLM Mesh partners to address advanced topics like model scanning, red teaming, data protection and privacy, and private LLM deployments.
Enhance Trust With Model Scanning
When deploying an AI model to production, it's critical that you can trust it. Trust is cultivated by understanding a model's behavior and potential security vulnerabilities, whether developing your own model or leveraging a third-party model. The expanding use of LLM foundation models has opened up new security concerns like serialization attacks and architectural backdoors.
- Serialization attack – An exploit of serialization process vulnerabilities to gain unauthorized system access or data manipulation.
- Architectural backdoor – A hidden vulnerability in an AI model that allows an attacker to control how the model behaves.
Dataiku has partnered with Protect AI, allowing joint customers to leverage Guardian to enable comprehensive scanning and continuous monitoring of your first and third-party models. Using Guardian within your Dataiku flows empowers you to better understand your models with robust model security and compliance checks.
Our partnership with Dataiku empowers joint customers to build robust, trustworthy, and secure AI by leveraging Guardian's comprehensive model scanning and continuous monitoring directly within their Dataiku flows.
– Zoe Hillenmeyer, CMO, Protect AI
Guardian provides methods to scan for known vulnerabilities, malicious code, and unexpected behaviors before deploying a model into production. Guardian is also partnered with Hugging Face, providing security insight into the millions of available open-source models. This rigorous oversight ensures that the AI agents and GenAI use cases you build are robust, trustworthy, and secure.
Go Further: Scanning Models With Protect AI Guardian Using Python
Understand the Model Through Red Teaming
Even the most sophisticated AI models can have blind spots or vulnerabilities that aren't apparent during standard testing. "Red teaming," a concept borrowed from cybersecurity, involves adversarially testing AI models to uncover these weaknesses, improve their robustness, and ensure they behave as expected under various inputs, including malicious or unexpected ones.
Dataiku has partnered with PRISM Eval to assist you with understanding the behavioral boundaries of an LLM before using it in a production AI application. PRISM's Behavior Elicitation Tool (BET) is an AI system that actively explores your model's behaviors through dynamic adversarial optimization, providing precise metrics for behavioral control. BET disrupts traditional static evaluation methodologies by deploying a dynamic, adversarially optimized approach that can also be leveraged to generate tailored synthetic data for each GenAI system.
The BET Leaderboard showcases how well AI models resist attempts to elicit harmful behaviors from expert prompting, providing an estimated average number of steps required to elicit harmful behavior across various categories (e.g., crimes, hate, etc.). By proactively identifying and mitigating these potential failure points, organizations can build more reliable, secure, and ethical AI systems that users can trust.
Static benchmarks miss what BET finds: real behavioral vulnerabilities that emerge in context. Our tool gives Dataiku users precise adversarial coverage and actionable metrics and data for safer GenAI deployment.
– Nicolas Miailhe, Co-founder & CEO, PRISM Eval
Secure Fine-Tuning With Synthetic Data
Fine-tuning an LLM allows you to tailor a model to your unique business context, vocabulary, and organizational knowledge. In Dataiku, non-technical users and experienced data scientists can leverage LLM fine-tuning. There is both a visual approach and a code-based approach, ensuring the benefits of fine-tuned LLMs are accessible across the organization.
LLM fine-tuning is adjusting a pre-trained LLM to perform better on a specific task or within particular domains. Commonly performed using an organization’s proprietary data.
Synthetic data is the key to unlocking the full potential of LLM fine-tuning without compromising privacy. By combining MOSTLY AI’s privacy-preserving synthetic data with Dataiku’s flexible AI platform, organizations can confidently train models on sensitive domains.
– Tobias Hann, CEO, MOSTLY AI
However, fine-tuning does carry a risk of training data leakage. When used to generate content, the fine-tuned model may inadvertently reveal sensitive or private internal information embedded during model training. Dataiku can provide a security-first approach to fine-tuning by using masked or synthetic data to mitigate these risks. In partnership with MOSTLY AI, you can generate synthetic data based on your proprietary data from the Dataiku flow for model training and fine-tuning. Synthetic data allows you to develop specialized and accurate models while upholding data privacy and security standards.
Go Further: Leveraging MOSTLY AI for Synthetic Data Generation: Dataiku Developer Guide
Empowering Private LLM Deployments
Many enterprises choose not to use hosted LLMs due to regulatory or data sovereignty requirements. Others need tighter control over model behavior, cost optimization, or seamless integration with internal systems. The Dataiku LLM Mesh is built to support private deployments — giving you full control over data, models, and infrastructure, whether on-prem or in your virtual private cloud. To that end, we’ve partnered with the strongest players in the space to deliver deep integrations for these use cases:
- Hugging Face: Download, deploy, and use open-source models directly within Dataiku for flexible, transparent AI development.
- NVIDIA: Streamline high-performance LLM deployment with NVIDIA NIM, enabling low-latency, GPU-accelerated inference for enterprise workloads.
- Doubleword: Use the Doubleword (formerly TitanML) plugin to easily deploy custom, domain-specific, or open-source models with enterprise-ready serving.
With these capabilities, organizations can deploy a private LLM, keeping all sensitive operational data securely within their environment while benefiting from scalable and efficient model serving and the full capabilities of LLM Mesh.
Secure, Governed, and Open
The Dataiku LLM Mesh, emphasizing security, governance, and openness through strategic partnerships, provides The Universal AI Platform™ for orchestrating enterprise AI. It empowers organizations to move beyond proof-of-concepts and confidently build, deploy, and manage enterprise-grade AI agents and GenAI use cases at scale. By addressing critical aspects like model scanning, red teaming, secure fine-tuning, and private deployments, Dataiku ensures that your journey into the world of LLMs is not only innovative but also secure, compliant, and aligned with your core business values, truly enabling enterprise AI.