Six months ago, we shared an essential update on Dataiku's journey into the world of Generative AI — a journey that has only gained momentum since. Back in February, we saw the early stages of this transformative technology being adopted by our customers, with 83% of AI leaders already experimenting with Generative AI. Today, those experiments have matured into sophisticated, production-level implementations that are reshaping industries and redefining business strategies. Our just-released survey with Databricks reveals that 65% of senior AI professionals with GenAI models in production are experiencing positive returns on those investments.
The last six months have been a whirlwind of innovation at Dataiku. We've been relentlessly focused on empowering our customers with the latest models, tools, and guardrails to strategically integrate Generative AI technologies into their operations, with the Dataiku LLM Mesh at the forefront of these advancements. In this blog, we’ll recap some of the most exciting platform additions and give you a sneak peek of roadmap features currently under development.
What's New?
1. More Connections and Models in the Dataiku LLM Mesh
In today’s highly-competitive and volatile AI ecosystem, the Dataiku LLM Mesh enables organizations to embrace a multi-LLM strategy and easily swap out models powering existing applications as new or better technologies become available. We’ve expanded the LLM Mesh to integrate with 15 leading cloud and AI services providers, including new dedicated connections to Snowflake Cortex, Mosaic AI, Mistral AI, and NVIDIA NIM. We’ve also been keeping up with the latest and greatest models as they emerge (and believe me, the waves of innovation just keep on coming!), adding support for cutting-edge LLMs such as Llama3, Gemma, Claude 3, Arctic, DBRX-Instruct, GPT-4o, and Mixtral 8*7B through these connections.
Dataiku provides direct connections to LLM providers via a secure API gateway with built-in security and usage controls.
Other LLM Mesh enhancements include token streaming for models that support it, the ability to add organization ID to OpenAI calls for internal chargeback or cost allocation purposes, and several additions to the LLM Mesh API to support advanced parameters, multimodality, and function calling.
2. Turbocharge Retrieval Augmented Generation (RAG) Workflows
With over 60 of our customers eager to build internal AI assistants to improve employee productivity and provide a better working experience, we’ve also focused a lot of attention on speeding up and removing friction from the end-to-end process of developing a RAG-powered chatbot. Because many foundational LLMs were not trained for a specialized domain or were not exposed to very recent or proprietary company data, one of the biggest barriers to Generative AI adoption at scale is fear of model hallucinations. The RAG approach of augmenting an LLM’s pre-existing knowledge “just in time” with additional context and approved source information is one of the most practical and powerful ways organizations can build user trust and mitigate risk of hallucinations (without needing to fine-tune models).
First, we updated the text extraction and Optical Character Recognition (OCR) recipe to extract text in more meaningful semantic chunks and preserve metadata like header, section, or page number. This “smart chunking” reduces the risk of splitting a section at an awkward breaking point, and the metadata is also useful for semantic search and manual document navigation when fact-checking cited sources.
Next, we wanted to improve the builder’s RAG experience by granting more visibility and control over the chunks of data that will be embedded. In the prepare recipe, a new “Split column into chunks” processor gives you many options for chunk size, overlap, and the text delimiters used. Automatic step preview allows you to review how the text chunks look before proceeding further in your workflow, and adjust the settings or add more cleaning steps as needed. For example, you may choose to filter out chunks that are too small to be useful or apply pseudonymization to mask sensitive information in the source data prior to embedding.
Once your documents or document chunks are vectorized (thanks to the embed recipe), the information is efficiently housed in a vector store and represented by an object called a “knowledge bank” in your Dataiku Flow. Knowledge banks are now shareable across projects to streamline reuse of source material for multiple use cases. For example, internal employee policy documents may be useful reference material for chat applications in both the HR and legal departments.
Last but certainly not least, we’ve made many updates and enhancements to Dataiku Answers, the packaged, scalable chat interface that democratizes LLM-powered Q&A to everyone in the enterprise. For instance, you can now ask questions of your enterprise data, radically cutting down time spent submitting requests to other colleagues or teams for data insights. A business analyst working for a bank might ask the chatbot,“What is the most used reward program?” and behind the scenes the application identifies the relevant approved dataset, constructs and executes a SQL query, and returns the answer in natural language (along with the generated SQL code used, if desired).
For more control over the data sources used to answer a question, you can configure Dataiku Answers to:
- Retrieve and provide relevant information from an assigned knowledge bank to the LLM as additional context.
- Use only the LLM (and not the knowledge bank).
- Let the LLM decide on the fly, based on the specific question asked, which of the two options above is the right choice.
3. Customize Models With Fine-Tuning
Need to refine an LLM to perform better on a specific task or in a highly specialized domain? With Dataiku, fine-tune models from Hugging Face or hosted models from service providers like OpenAI using either a visual or code-based approach. Both methods register the resulting fine-tuned models in the Dataiku LLM Mesh, so your organization can ensure the same level of control and governance for customized models as for foundational models.
What's Next?
1. Dataiku LLM Guard Services
With the initial shock wave of enthusiasm caused by mainstream awareness of ChatGPT now almost two years behind us, many companies’ initial Generative AI experiments have matured into advanced, large-scale implementations. For many data and IT leaders, the question has shifted from, “Can we leverage this technology to be more productive and create new experiences?” to “How can we efficiently scale to deliver even more use cases, without compromising our risk and safety principles?”
This year, Dataiku has been developing an array of Guard Services to provide oversight and assurance for LLM usage in the enterprise.
- LLM Cost Guard: Enables teams to monitor and control LLM costs and enforce budget-based usage rules.
- LLM Quality Guard: Provides tools for LLM evaluation and fine-tuning, so you can maximize response quality and relevance for your use cases.
- LLM Safe Guard: Secures LLM usage with tooling and additional customization to avoid abuse and leakage.
2. AI Regulations and Governance
With the EU AI Act published in the Official Journal and now having come into force on August 1, 2024, ensuring your organization is ready for compliance is more crucial than ever. Since non-compliance with the EU AI Act can result in severe financial penalties including maximum fines of up to €35 million or 7% of a company’s global annual turnover, organizations must thoroughly understand the requirements of the Act, assess their impacts, and implement both organizational and technical foundations.
Recognizing this imperative, Dataiku has introduced its EU AI Act Readiness Program — a comprehensive solution designed to help global organizations navigate the complexities of the new EU AI Act, drive Responsible AI innovation, and effectively govern AI at scale. As part of its robust suite of AI Governance capabilities and as a leader in the IDC MarketScape — AI Governance Platforms, Dataiku has recently added two new components to help minimize risk and ensure teams can systematically oversee the operational workflows of all AI projects:
First, the EU AI Act Readiness solution: A framework with pre-built blueprints containing clear steps to build, test, and deploy Al projects with optimized speed, value, and compliance responsibilities management. With Dataiku, risk and AI managers can easily customize approval workflows to collect sign offs from key stakeholders at each step, fostering accountability and ensuring audit readiness.
Next, the Dataiku LLM Registry supports the LLM Mesh with an extra layer of governance, allowing teams to qualify, document, and frame acceptable usage for both AI services connections and their included models. The registry serves as a central hub for details about contractual terms, connections permissions, and usage controls (e.g., PII or toxicity detection, forbidden terms, full audit trails) for individual LLM providers.
The Dataiku LLM Registry is a critical step towards regulatory readiness and managing the use of LLM technologies across the organization, helping CIOs and their teams keep model documentation up to date as well as rationalize which models should (or should not) be used for what use cases. For example, a company might permit usage of third-party LLMs such as OpenAI GPT4 for creating marketing copy or analyzing public news content, but require use cases involving sensitive internal data to use self-hosted LLMs.
3. A Generative AI Technical Guide From O'Reilly and Dataiku
Finally, in this fast-moving tech landscape, we recognize that it can be overwhelming to keep up with each successive wave of Generative AI innovation and its implications on your business. Kurt Muehmel, Head of AI Strategy at Dataiku, is partnering with O’Reilly Media to produce a technical guide called "The LLM Mesh: A Practical Guide to Using Generative AI in the Enterprise." This book will provide a range of helpful information on adopting LLMs at scale and navigating concerns around safety, security, performance, and cost.
In fact, you can check out the first chapter now to learn about LLM Mesh fundamentals, model licenses and hosting options, and how to choose the correct models for your use cases. We’ll continue to release more chapters for early preview throughout the remainder of the year, with the full guidebook available for download in early 2025.
The Road Ahead
As we look back on the past six months, it’s clear that what started as a promising technology has now become a game-changer for industries across the board, driving innovation and efficiency in ways we could only imagine a short while ago. At Dataiku, we’ve been at the forefront of this transformation, not just by keeping pace with the rapid advancements in AI, but by leading the charge with powerful tools like the Dataiku LLM Mesh, easy-to-setup RAG workflows, and robust AI Governance capabilities.
But we’re not stopping here; the future holds even more exciting developments. Whether it’s through enhanced connectivity with top-tier providers, the introduction of new guard services and quality metrics, or our partnership with O’Reilly to demystify the complexities of Generative AI in the enterprise, remember that Dataiku is here to support you every step of the way. With the right tools and strategies, you can stay ahead of the curve! Here’s to what’s new, what’s next, and the incredible journey that lies ahead.