Mastering AI Agents in Dataiku

Businesses are looking for ways to leverage AI agents to optimize workflows, enhance productivity, and automate complex processes. In a recent webinar, Christian Capdeville, Director of Product Marketing at Dataiku, and Christiaan Burrett, Solutions Engineer at Dataiku, walked us through the step-by-step process of building an AI agent in Dataiku. They covered everything from AI agent fundamentals to live demonstrations of implementation.

In this blog post, we will break down the key takeaways, highlight some of the most insightful moments from the discussion, and provide you with everything you need to get started with AI agents in Dataiku.

What Is an AI Agent? How Does It Differ From a Chatbot?

Before diving into the technical aspects, Christian clarified the definition of an AI agent:

An agent is an LLM powered system designed to achieve objectives across multiple steps… that goes beyond just answering a question.

— Christian Capdeville, Director of Product Marketing at Dataiku

A chatbot, for example, might provide an answer about a company’s expense policy. An AI agent, on the other hand, could review receipts, check policies, flag issues, route approvals, update systems, and send notifications — all autonomously.

This ability to execute multi-step workflows makes AI agents invaluable for enterprises looking to go beyond simple Q&A interactions.

Building an AI Agent in Dataiku

Christiaan took on the challenge of live-building an AI agent in under 30 minutes. His demo showcased the full power of Dataiku, from knowledge base retrieval to ticket creation. Here’s a breakdown of the process:

1. Defining the Use Case

The demo was based on a real-world scenario where a company needed an AI system to help employees retrieve information from internal documentation. Employees frequently asked subject matter experts (SMEs) for answers, consuming 50%-70% of SME time. The AI agent was designed to:

• Pull answers from a knowledge base (a RAG pipeline).
• Escalate unanswered questions by creating support tickets.
• Ensure high-quality responses by drafting answers for support agents.

2. Setting Up the Knowledge Base

The first step was embedding documents into a vector database. This allowed the AI agent to retrieve the most relevant information when a user asked a question.

Embedding documents can take a ton of time. Figuring out how you’re going to process images, text, and different file formats can be a bit of a nightmare. With Dataiku, we did this in less than a minute.

— Christiaan Burrett, Solutions Engineer at Dataiku

The demo connected to a folder of PDFs, which were transformed into a searchable knowledge bank.

3. Building the AI Agent With Tools

AI agents become powerful when they interact with external systems. Dataiku allows users to add tools that extend agent capabilities. In the demo, two tools were created:

• Document Retrieval Tool: Used to fetch relevant documents from the knowledge base.
• Ticket Creation Tool: If the AI agent couldn’t answer a question, it created a ticket in Airtable (though this could be any support system like ServiceNow).

4. Creating the Agent and Testing Responses

After setting up the tools, Christiaan configured the agent’s prompting logic:

You are a support assistant. If you find relevant documents, provide an answer. If not, create a support ticket with a draft response.

A series of test questions confirmed that the agent successfully retrieved answers when possible and escalated issues when necessary.

5. Deploying a User-Friendly Interface

Instead of having to build a custom frontend and interact with the agent via API calls, Dataiku provides Dataiku Answers, a visual web app that allows users to chat with the AI agent seamlessly. The demo also highlighted Agent Connect, which routes queries to the right AI agent in organizations with multiple agents.

Monitoring & Improving Agent Performance

One of the most important aspects of deploying AI agents is monitoring their performance over time. Christiaan demonstrated how Dataiku’s LLM evaluation metrics help ensure reliability:

• Faithfulness: Measures if responses are factually accurate.
• Answer Relevancy: Evaluates how well responses align with queries.
• User Feedback Tracking: Allows teams to refine answers based on real interactions.

Additionally, Cost Guard was introduced as a tool that helps businesses monitor LLM usage costs and optimize for efficiency. With Dataiku, you can track performance metrics, ensure compliance, and make cost-effective LLM decisions — all from a single platform.

Here’s What You Need To Remember

1. AI Agents Go Beyond Chatbots: They can execute actions, make decisions, and automate workflows instead of just answering questions.
2. Dataiku Makes It Easy: Embedding documents, setting up tools, and building AI agents can be done with minimal coding.
3. Monitoring Is Essential: Tools like LLM evaluation, user feedback, and cost tracking help ensure AI agents perform well and stay cost-effective.
4. Scalability Matters: AI agents should be able to integrate with various systems (e.g., ServiceNow, SharePoint, APIs) and evolve over time.