The Ultimate Test of ChatGPT

ChatGPT was released almost two years ago now, so we thought this would be a good time to analyze its performance. After two years of reinforcement learning, how knowledgeable has ChatGPT become? To attempt to answer this question, this blog post will compare responses from ChatGPT and surveyed individuals.  

In May 2024, Dataiku and Databricks surveyed 400 senior AI professionals from leading global companies to explore the deployment of GenAI across industries. We have decided to put ChatGPT to the test and ask five questions that were presented to these 400 AI leaders. This will be ChatGPT’s annual exam. Let's score the app one point per correct answer, up to five points total. It’s not time to dive into the responses and see how they differ (or not!). 

Note: We have rephrased the questions to ask GPT-4-turbo to characterize the AI adoption and impact trends in the market today. We also made sure to include the same filters that were put in place in the survey to ensure the responses are indeed comparable: 

• We are considering companies of more than three billion in revenue
• Companies must be U.S., Europe, or APAC-based 

Question 1: AI Adoption

What best describes the level of adoption of data science and/or AI in organizations today?

ChatGPT 

For organizations with over three billion in revenue in the U.S., Europe, or APAC, the adoption level of data science and AI typically falls between Establishing and Expanding. Many of these large enterprises have moved beyond initial experimentation and have demonstrated value in multiple analytics and AI use cases. They are now working on building the necessary processes, infrastructure, and governance to scale these capabilities more broadly.

A significant subset is at the Expanding level, where they are actively working to deploy data science and AI across various departments outside a central team, integrating these capabilities into multiple business functions. Only the most advanced organizations with mature data-driven cultures have reached the Embedding stage, where analytics and AI are a deeply ingrained part of the organizational DNA across all functions.

Survey Respondents

Yes, and … yes! Indeed, according to the 2024 “AI, Today” survey report, a majority of organizations are either establishing or expanding their use of data science and/or AI, with a larger part of them being in the expanding category. We also find a smaller percentage of organizations are embedding AI into the core functions of their organizations — which ChatGPT also called out. 

Maybe this question was too easy. Let’s dig a little deeper. 

Question 2: AI Organizational Structure

What best describes organizations’ structure around data science, analytics, and AI initiatives?

ChatGPT 

For large organizations with over three billion in revenue in the U.S., Europe, or APAC, the Hub & Spoke model is currently the most common structure for managing data science, analytics, and AI initiatives. In this setup, a central "hub" team typically oversees essential infrastructure, establishes standards, and monitors advancements in data and AI technology. Simultaneously, individual business units or "spokes" hold increasing responsibility for developing and managing AI applications specific to their needs.

Some of these organizations also use a Centralized Center of Excellence (CoE) model, particularly when they are earlier in their AI maturity. However, as they look to scale, many transition to the Hub & Spoke model to balance the benefits of central governance with the flexibility and customization needs of different departments.

A few advanced organizations are approaching an Embedded structure, but this is less common and typically seen only where data and AI have become core components of daily operations across functions.

Survey Respondents

So close! According to our survey respondents, a majority of organizations actually use the Centralized CoE model. While the Hub & Spoke model does follow closely behind, ChatGPT aimed a little too high on this one. 

The percentage of organizations with a CoE has decreased from 39% to 38% from 2023 to 2024, which might suggest organizations are moving very slowly towards a more mature Hub & Spoke or Embedded operating model. Indeed, the percentage of organizations operating with an Embedded model increased from 10% to 14% in just one year — maybe this explains ChatGPT’s optimism? 

Question 3: Accounting for Value

To what extent do organizations account for the value delivered with data, analytics, and AI initiatives?

ChatGPT

In large organizations with over three billion in revenue across the U.S., Europe, and APAC, the value delivered by data, analytics, and AI initiatives is typically tracked using a framework that is inconsistently applied. While many of these organizations recognize the importance of measuring ROI, applying frameworks consistently remains a challenge. Often, different business units or departments may use varied metrics, leading to inconsistencies in how value is reported and evaluated across the organization.

A smaller but growing subset of these large organizations has managed to develop a framework that is consistently applied across teams, allowing them to track AI and analytics initiatives more rigorously. However, it is rare for organizations — even at this scale and level of maturity — to fully account for these initiatives on their balance sheets.

Survey Respondents

Touché! Indeed, 41% of survey respondents stated their organizations have a framework that is inconsistently applied. We also find that 28% of organizations have a consistently applied framework, but only 7% have a fully accounted balance sheet — both of these findings are reflected in ChatGPT’s answer.

Question 4: Return on AI

Approximately what return do organizations deliver for each $1 spent on data, analytics, and AI initiatives?

ChatGPT

For large organizations with over three billion in revenue in the U.S., Europe, or APAC, the typical return on each $1 spent on data, analytics, and AI initiatives generally falls in the $2-5 range. This ROI reflects the level of maturity and investment that many large enterprises have achieved, often focusing on initiatives that can directly improve efficiencies, enhance decision-making, or drive revenue growth.

While returns of $5-10 per dollar invested are occasionally reported, these are usually in highly data-driven sectors or in particularly high-impact projects where AI enables substantial revenue gains or operational savings. However, achieving and sustaining returns beyond $2-5 generally requires mature infrastructure, a data-literate workforce, and well-integrated AI capabilities across the organization.

Survey Respondents

Another one hits the mark! The majority of respondents claimed a $2-5 estimated return on AI initiatives. 

While ChatGPT mentions the organizations who estimate larger returns, it did not mention the ones who estimated lower returns. However, according to survey findings, there are actually more organizations that estimate a $1 return on investment (20% of respondents) than there are organizations that estimate $5-10 returns (12% of respondents). So, let’s give ChatGPT 50% on this one. 

Question 5: Barriers From AI Value

Please rank all barriers preventing organizations from delivering more value from data, analytics, and AI, where 1 is the biggest/most present barrier.

ChatGPT

1. Lack of quality data or ability to easily access the right data: Despite having the resources, many large organizations struggle to access and integrate high-quality data from across different business units, departments, or geographies. This is often the most significant hurdle.
2. Inability to quickly operationalize and iterate on data, analytics, and AI products: Many organizations have proof-of-concept projects but struggle to deploy AI solutions at scale or iterate quickly enough to see sustained value. The gap between development and operationalization can be a key barrier.
3. Lack of visibility and control of data, analytics, and AI products across the organization: Without centralized visibility and governance, organizations face inefficiencies, duplication of efforts, and challenges scaling their AI initiatives across multiple teams or departments.
4. Lack of data talent or data-literate workforce: Even in large organizations, the shortage of skilled data scientists, engineers, and data-literate employees can significantly hamper the execution and scaling of data-driven projects.
5. Infrastructure that is too complex: Legacy systems, siloed data, and overly complicated infrastructure slow down data and AI initiatives. Managing and maintaining these systems becomes a barrier to growth and scalability, even for large enterprises.
6. Cost: While cost can be a concern, it is less of a barrier for large organizations with significant budgets. However, the need for continuous investment in advanced AI infrastructure, talent, and tools can still present challenges.
7. Lack of a clear business case: Large organizations may have multiple departments or stakeholders with varying objectives. A lack of alignment or clear business cases for AI and data initiatives can delay or prevent critical investments, even if there’s awareness of the technology's potential.

Survey Respondents

Wow, that was impressively close. According to our survey findings, the top four obstacles to extracting more value from analytics and AI are indeed:

1. Lack of quality data or ability to easily access the right data
2. Inability to quickly operationalize and iterate on data, analytics, and AI products
3. Lack of data talent or data-literate workforce 
4. Lack of visibility and control of data, analytics, and AI products across the organization 

However, the majority of respondents found the lack of data talent more hindering than the lack of visibility and control. Besides this order switch, ChatGPT got all the other barriers correctly reflected. We’ll allow 75% of the points. 

The Final Countdown

ChatGPT comes out of this exam with an impressive 3.15 out 5 points. Its assessment of large organizations’ AI maturity levels is quite similar to findings from our 2024 survey of 400 AI professionals. 

This exercise underscores ChatGPT's potential as a valuable tool for understanding and analyzing complex industry trends. While there are nuances that AI alone may overlook, ChatGPT demonstrates it can deliver credible insights, aligning with the views of leading AI practitioners. As this technology evolves, it will be fascinating to see how its responses further improve, potentially providing even more precise guidance to professionals and leaders navigating the future of AI.