Here we are on the cusp of 2024. So many exciting product updates have happened at Dataiku this year over the course of six releases. In the first article of this series, we’ll take a look back through the year, but stay tuned for a look ahead to the future in another post, as well as an article on new Generative AI features. As it’s the holiday season, let’s have some fun and review some of the highlights of 2023 as though we’re walking through a winter wonderland!
The Bakery: Generative AI Application Development
Let's begin our tour with some exciting treats to fuel our journey. The best place to go is the bakery with its made-to-order goodies! Dataiku’s Generative AI tools are like the appliances in a commercial bakery. Each device has a specific use and works together to make delicious treats.
Dataiku continues to improve tools and features to increase efficiency for companies seeking to use or develop Generative AI applications:
- LLM Mesh - The recently introduced LLM Mesh is a set of specialized features in Dataiku that serve as a technical backbone containing common functionality that enables teams to build enterprise-grade LLM applications efficiently.
- Prompt Studios - Design, explore, test, manage, & compare prompts with different LLMs.
- RAG - Embed Recipe & Knowledge Bank - Transform your specified documents into a knowledge bank, applying semantic search to include the most relevant information in the request to the LLM to inform a more reliable and accurate response.
- Text Classification & Text Summarization Recipes - Use new native recipes to classify or summarize lengthy text. The visual recipes contain options to use generic LLMs or pre-trained models for predefined tasks such as sentiment and emotion analysis.
These tools work together to make Generative AI application development more efficient and sustainable.
Elves' Workshop: AI Assistants
Step inside for a visit to the elves' workshop! Everyone knows elves add a special touch, while being incredibly productive and unbelievably fast workers. Have you ever wished for your own "elves" to help you out? Meet our AI Assistants that not only improve efficiency, but also may improve the quality of the overall finished product.
Dataiku released three new AI-enabled assistants in private preview:
- AI Prepare - Simply describe the transformation you want to apply in natural language, and the AI assistant automatically generates the necessary data preparation steps.
- AI Code Assistant - Submit a code-related question or magic command through AI Code Assistant and receive context-enriched answers. This feature helps you write, explain, or debug code, comment and document your work, create unit tests, and more.
- AI Explain - In Dataiku, the Flow is where your entire project structure and data pipeline are visually represented. Use AI Explain to automatically generate descriptions that explain Dataiku Flows or individual Flow zones, and save precious time versus manually writing documentation or investigating individual elements to grasp the purpose of the pipeline.
Example of a Dataiku Flow
Ski Lift: MLOps
Let’s take the ski lift to better view this winter wonderland. At Dataiku, our ski lift is MLOps. As a ski lift has a continuous route, many moving parts, and a powerful control center, so does MLOps. Ski lifts also get you where you want to go while keeping you on the right path. We’ve introduced some exciting new MLOps features this year to help teams optimize and have more flexibility:
- Model Overrides - Add a human layer over the model’s predictions to help ensure models will not predict outlandish values, enforce ethical boundaries, and comply with regulations. Confidently develop & deploy models under safe boundaries.
- External Models - You can now utilize existing AWS Sagemaker, Microsoft AzureML, or Google Vertex AI models in Dataiku. By integrating external models, you can leverage the benefits of traditional Dataiku models for models deployed externally.
- Deploy Anywhere - Deploy models to other production environments like AWS SageMaker, Azure ML, and Google Vertex. This gives teams the flexibility to develop a model in one place but deploy in another, all while leveraging Dataiku as a central location to monitor, govern, and democratize access to all models.
Show Schedule: Data Intelligence & Discoverability
As we journey through this winter wonderland, we may want to catch one of the many performances. Carolers will be performing in the morning, a school group is putting on a play, and a local quartet will be playing holiday tunes too. The show schedule organizes each performance by time and group, as well as gives an overview of what the audience can expect. Likewise, Dataiku has introduced some features this year that allow you to get quick overviews and help discover and organize your company’s datasets:
- Data Collections - Centralize and organize key datasets by use case, team, or other theme.
- Recipe Summaries and Visual Previews - Select a recipe or dataset to review a summary of the transformations and inspect a sample of the data, all without leaving the Flow.
Holiday Recipes: New Visual Recipes
Dataiku's new visual recipes are like your family's tried and true recipes. They always give you a reliable, effective outcome and improve the final baked product. Dataiku is no stranger to visual recipes, but we are committed to providing new and improved recipes to help users with common tasks.
- Auto Feature Generation - With just a few clicks, you can generate new features from existing datasets. A visual recipe guides users through assigning relationships and join keys between primary and enrichment datasets to generate these features.
- Statistics Cards to Recipe - Export Principal Component Analysis (PCA) statistics card as a PCA recipe.
- Right and Left Anti-Joins - Run a join recipe that only outputs the unmatched data.
Waxing the Sled: Enhancement to Visual Time Series Forecasting
Waxing the sled is essential to having fun on the snow, enabling you to go further and faster. Dataiku’s enhancements to visual time series forecasting provide similar benefits. Go further with your forecast horizons and faster with series decomposition, auto-documentation, and Prophet support.
- Evaluate Beyond Forecast Horizon - If you have a project where you don’t plan to retrain the model as frequently as the forecast horizon, you now can specify an evaluation period longer than the horizon.
- Visual Time Series Decomposition - Explore your data with new statistical analyses for time series decomposition, then publish the analyses to dashboards.
- Automatic Model Document Generation - Automated, comprehensive, schedulable model documentation for time series forecasting models.
- Prophet Support - You can now easily access the well-known Prophet forecasting procedure as a built-in algorithm. You’ll also be able to export the predicted dataset directly from your model experiments.
Hot Cocoa Stand: Uplift Modeling With Causal Predictions
The next stop on the winter wonderland tour is the hot cocoa stand, where you can select marshmallows, a peppermint stick, whipped cream, or other add-ons. Free marshmallows might sell more hot cocoa than free cinnamon, but how do we model and quantify the cause and effect nature of these relationships?
With Dataiku’s Causal Predictions AutoML task, model cause-and-effect relationships to know the influence of a specific treatment on the desired outcome.
By calculating the incremental impact of a treatment, such as a direct marketing action, you can prioritize the most “influenceable” cases for the outcome or behavior you want. Use uplift modeling not just for retail and marketing use cases, but also for fundraising, medical treatment and clinical trials, human resources programming, or even political campaigns.
Reading by the Fire: Upskilling & Reference Documentation
It’s fun to walk through a winter wonderland, but it’s also great to warm up by the fire while reading! If you’re looking for something to read by the fire, check out Dataiku’s latest guides and documentation so you can do more, faster with Dataiku:
- Help Center - The dynamic help center in Dataiku centralizes a wide variety of useful resources for both technical support and educational purposes and provides contextual, personalized content recommendations. Consume reference documentation without leaving the product.
- Developer Guide & New Tutorials - Find new task-based quickstart tutorials for getting started with Dataiku, a Responsible AI course, and a wide variety of tutorials in the Developers Guide on topics ranging from webapp development to Generative AI to MLflow integrations.
Holiday Lights: Data Visualization & Consumption Improvements
You see some beautiful holiday lights this time of year in all sorts of fun patterns. You can also spot patterns easily with data visualization. Equally as beautiful as holiday lights, are the latest visualization updates in Dataiku, enabling you to quickly explore your data or build a specific visualization.
Designers can easily build data products that are interactive, intuitive, and flexible. These new features expand on Dataiku’s goal to make it easy to create and share data products with business stakeholders:
- New chart types (radar charts, sankey diagrams, multi-pair scatter plots, etc.)
- Dynamic filtering & relative date filters
- Share and export filtered views
- Custom aggregations and reference lines
- Copy/paste charts
What's Next?
Thanks for walking through this winter wonderland with us! We had so many treats in 2023, but let me be the first to tell you even more are coming in 2024! I can’t pull back the curtain fully yet, but stay tuned for more exciting Dataiku features. (Let me give you a hint … Think Generative AI and MLOps features.) Happy Holidays!