Every single day, I see companies attempting to figure out how to maximize the value of their data. The cultural change required is often the biggest challenge. Finding the right data and getting the right data assets to the right people in a way that embraces collaboration is absolutely necessary to ensure success in any project that involves data but is still oftentimes very difficult for organizations to achieve.
This blog outlines the top three biggest challenges I see most for data and analytics leaders and provides examples of tactical steps required when driving a cultural change within an organization.
1. Siloed Data and Teams
Every company is different: Each has different business needs, different initiatives, and different infrastructures. Data comes from disparate sources and is not easily or fully accessible by other groups in the same organization. In addition to this, teams change through time, and diverse skill sets are expected. Different experiences or educational backgrounds bring varying skill sets to the equation and that usually translates into an infrastructure that supports these people in isolated ways.
For example, you will need to start by connecting to the data silos that contain the data that will represent attributes in the data science pipeline. This data will have quality issues, mixed category names, unparsed dates, concatenated text, etc. After fixing quality issues, you will have users that create dashboards or ask questions about this data. Data exploratory techniques, as well as data visualization, can be done in many ways, using open and non-open-source tools. At this stage, usually, analysts or data scientists enrich the dataset, calculating new columns, extracting even more insights from your data. These users could be programmers or not, but they understand how to read your data in a way that everyone in the company can benefit from.
What I have seen through my career is that the deeper you go into the data science pipeline, the deeper the gap between the skills across the people that are asking questions about this data.
The lack of integration in this ecosystem isolates people from processes, contributing to the gap between internal teams. Data will continue to come in your pipeline and, the more visibility you have, the better the questions you can ask and the more you can use the ML algorithms to predict your KPIs — ultimately allowing you to be one step ahead and make better, more educated decisions.
2. Proving Value
With siloed data and siloed teams comes poor transparency, making it not so easy for groups in the same organization to prove the value of a data science project. In many cases, evaluation teams dig too deep on the technical side instead of setting up the use case for it to become a win, connecting the project goal to a measurable KPI to start with. KPIs are tied to variables and variables are represented by data. The lack of integration in the ecosystem adds to the challenge as IT does not have years to do this.
Creating an AI culture must involve a mixture of managing the expectations from the business side and delivering measurable wins. Measurable wins require experts, and the challenge comes when you attempt to connect the dots for all these to work together, maintaining your experts in the loop. I still see the industry struggle with this, regardless of the size of the company or the vertical.
3. Scaling
Siloed teams usually work with siloed tools. For example, open source programming languages and IDEs are great resources to embrace. The challenge comes when script owners move on or new players, perhaps from extended teams with no coding backgrounds, also need insight into this data. Empowering people to accumulate reusable artifacts that go on to produce helpful insights accelerates the learning curve of new users. They can then stand on the shoulders of giants, produce their own assets, and repeat the cycle. This will organically create an accelerated flywheel of analytics projects.
Don't Boil the Ocean! Implement Small Steps Towards a Cultural Change
Simple concepts like sharing the “assets” the teams are building and being able to communicate about it are great starting points.
Sharing code and data assets not only makes things more efficient but also empowers other data practitioners. There is no need to reinvent the wheel — reuse as much of what is already working in a way that is inclusive for all the diverse groups that ultimately are consuming these data assets one way or another.
Getting the teams to work together as ONE extended team that collaborates in a responsible way is the goal. It requires a common vision across departments, and one cannot simply rely on the selection of a technology application or platform to solve the problem. In fact, I haven’t seen one successful data science project without an SME as part of it (and I have 15+ years of experience in the industry). You need to ensure the expertise in your team is part of the data science pipeline, meaning they understand what the data is telling and conceptually validate the results of the ML.
Ultimately, every organization is going to have to figure out how to bring together teams of people with many different skill sets (data scientists, analysts, data engineers, executive business users, etc.) and learn how to build a culture to respect those differences if the company is to truly extract all the value out of its data. Some of the key things you must consider when developing a data strategy include:
- Responsible AI (i.e., If your ML algorithm produces unexpected outcomes, who is responsible for it?)
- AI Governance, so you can detect problems early and oversee stages of models, and overall increase efficiency.
- Technology resilience, because today you have a set of databases that hold your data but that, most likely, will continue to change. You may be working on a digital strategy that may or may not involve a cloud deployment or a hybrid infrastructure.
These are all key to ensuring a good strategy.
So, What Does "Cultural Change" Mean Tactically?
These are three simple yet tactical steps that I see successful companies adopt to implement cultural change:
- First, simplify the layers of complexity related to connecting to data and configuring compute resources. Democratize the data in a systematic way, so it can support diverse use cases.
- Second, keep working with your favorite programming languages, packages, and IDEs without sacrificing collaboration with other team members, who may or may not be using the same set of tools. Embrace open source with governance.
- Reuse and share code, datasets, and other assets to help teams reduce inefficiencies and inconsistent data handling, while simultaneously empowering less technical users to go farther on their own. Then, repeat at scale, systemizing the use of data and AI.
In summary, keep the human as the key component of data science projects, capture the expertise from your teams, question what can go wrong when deploying ML into production. Embrace collaboration, enable your teams to work together, regardless of the skills they bring to the table. If you empower your teams with the option of sharing and reusing data assets and artifacts and building up your AI portfolio in a systematic way, then innovation will naturally follow.