The arrival and adoption of Generative AI shows the importance of being ready for whatever comes next. Maintaining tech optionality is a matter of survival when the tech landscape is transforming before our eyes. But an overly complicated tech stack makes the integration of new tech even more difficult.
Given that there are multiple pieces of the puzzle across different areas of the data science, machine learning, and AI (including Generative AI) lifecycle, gluing even just a few of these together can become complex quickly. Plus, although point solutions for Generative AI might be all the rage, they are not scalable, as they augment one process but do not provide any benefit to adjacent processes.
While the tooling inconsistencies outlined above are at the crux of the issue, it doesn’t stop there. These problems can compound into much more serious concerns with detrimental impacts. For example, getting the myriad of analytics tools to talk to each other is cumbersome and lacks scalability because:
- Data lineage is difficult to track across tools.
- Stitching together tools can complexify the handoff between teams, slowing down the entire data-to-insights process.
- Managing the approval chains between tools becomes tremendously risky.
- Missed opportunities for automation between steps in the lifecycle arise.
- Audit and versioning across tools is nearly impossible.
Our survey with Cognizant of 200 senior analytics and IT leaders also revealed that teams still rely heavily on spreadsheets, even in the age of Generative AI. Further, while respondents alluded to smooth sailing from the offset with LLMs, it’s likely because they aren’t using them at scale yet and lack a long-term management plan (88% of respondents have faced infrastructure barriers to using LLMs in the way that they would like). All of these issues — that all come down to tooling — will only intensify with Generative AI.
Let’s break down some of the data.
How Modern Are Organizations' Modern Data Stacks?
The respondents who did not describe their current data analytics stack as “modern” were asked what modernizing a data stack would mean in their environment. Over three-quarters (79%) mentioned adding AI capabilities to their data stack, closely followed by tool consolidation (65%). Those who answered “Other” cited easier integration, tool migration, and automation as elements to modernizing a data stack in their environment.
Q26: What would modernizing a data stack mean in your environment?
To combat all of this tool proliferation chaos (not to mention infrastructures and LLMs!), analytics and IT leaders should be sure to consolidate to the capabilities that best serve the business. It’s important to invest in end-to-end technology that is open and extensible, allowing teams to leverage existing data architecture as well as invest in best-of-breed technologies in terms of storage, compute, algorithms, languages, frameworks, etc.
62% of Organizations Have Faced Serious Issues Due to Spreadsheet Errors
We’ve said it before and we’ll say it again: spreadsheets ≠ scale. A staggering 24% of respondents from our survey of 200 senior analytics and IT leaders use spreadsheets as their primary tool for working with data.
By using spreadsheets for data analytics, teams essentially stifle their growth. Spreadsheets are detrimental for business for reasons such as:
- Manual errors
- Single points of failure (i.e., they are not productionized, so if a laptop crashes or someone leaves the company, that work is missing)
- Data redundancy (due to it residing in multiple desktops and several formats)
- Limitations associated with a lot of data
- Lack of recovery and audit capabilities
- Overall regulatory and compliance challenges
The respondents who use spreadsheets as their main tool for working with data were asked why they do so, and the top response was that they have other tools but don’t know how to use them (or aren’t comfortable using them).
Q21a: Why do you use spreadsheets as your primary tool for working with data?
Interestingly, in a survey we ran last year with 375 line of business leaders around the world, 52% relied on spreadsheets as their primary data tool, so it’s not an isolated issue.
💡Dataiku’s AI Prepare allows people with the widest possible range of skills to build production-ready data transformations, simply by typing what they want done to their data. This breaks down the last barriers between knowing what needs to be done and making it happen in enterprise databases and cloud environments.
Conclusion
As we've seen, the challenges of tool proliferation, spreadsheet dependence, and integration difficulties are significant hurdles for many organizations looking to modernize their analytics and AI capabilities. These issues not only hinder efficiency and scalability but also pose risks in terms of data governance, compliance, and the ability to adapt to rapidly evolving technologies like Generative AI.
For IT leaders, the message is clear: maintaining technological optionality while streamlining your tech stack is crucial in today's fast-paced data landscape. The key lies in adopting flexible, end-to-end solutions that can integrate with existing infrastructure and best-of-breed tools, while also providing a foundation for future innovations. As we move forward, consider these action items:
- Assess your current tech stack and identify areas of redundancy or inefficiency.
- Invest in solutions that offer both comprehensive capabilities and openness to integration.
- Prioritize tools that can scale with your organization's growing data needs and emerging AI technologies.
- Focus on upskilling your team to reduce reliance on outdated tools like spreadsheets.
- Stay informed about emerging trends and technologies to maintain your competitive edge.
By addressing these tooling challenges head-on, IT leaders can position their organizations to not only survive but thrive in the ever-evolving world of analytics and AI. Remember, the goal is not just to solve today's problems, but to build a flexible foundation that can adapt to whatever technological advancements the future may bring.