Thinking of Analytics as a Product

14 years ago I (Doug) met with the director of business intelligence at NCR in his old headquarters in Dayton, Ohio. He’d recently finished a BusinessObjects implementation and proudly told me that he had 400 reports in production. I really didn’t have an understanding of that so I asked, “Is that too many or not enough?” It was obvious that he’d never considered the question, as a look of shock came over his face. A few months later, he disabled all the reports and received only two user complaints.  

More than a decade later, just last month, a CDO told me that she recently joined her company and found 1,200 production reports. She had no idea of their value so she tagged them with Google Analytics to measure monthly unique users and time on page. Only 20 reports, less than 2%, were actually being used.

How can we prevent this from happening to the analytics that we’re developing today?

Switching from a project mindset to a product mindset will help, but there are some big differences between them:

The first big difference between projects and products is that products have customers. Identify them, talk to them, understand their personas, and capture their use cases. A working backward or outside-in approach like this is uncomfortable for many data scientists and worthy of its own follow-up article.

Products have owners who are responsible for generating business value while simultaneously meeting service-level agreements (SLAs) with the customer, and therein lies the rub. There’s often a tension between business value and SLAs, so that’s one of the reasons they’re often separated. The product owner brings them together. When an owner cannot generate value while meeting SLAs, then they might relax the SLAs or simply discontinue the product.

Uber had a good article this year about treating data as a product where they identified many key quality issues. Analytics products have similar issues and, together, they form an implicit or explicit (but hopefully explicit) agreement between the product owner and customers. Issues include:

Model Accuracy:

This may include statistical metrics such as RMSE or AUC, but is more valuable to customers as a business metric such as incremental sales or conversion rate.

Frequency of Update:

Many analytics applications are time sensitive due to environmental changes (like cross-selling patio furniture in Minneapolis in the winter), business conditions (such as lumber prices doubling from May 2020 to May 2021), or regulatory changes (like tax codes and COVID-19 lockdowns). Customers of analytics products need to know the product’s update frequency.  

Time Since Last Update:

Things don’t always go as planned. Many retailers plan to update their customer segmentation models quarterly, but every two years is common. Transparency on updates allows customers to make informed decisions on whether to use an analytics product.  

API Call Volume and Latency:

Analytics are increasingly delivered via API. The volume and latency that APIs can handle has a big impact on both costs and the applications it can be used on.

Frequency of SLA Violations:

This is an SLA about SLAs but, like we said, things don’t always go as planned. Letting customers know the history and frequency of violations helps them to decide both whether to use the product and whether to press owners for better performance.

Projects generally have a fixed time period while products are continuous. Microsoft Excel is 35 years old and still going strong. Analytics products, like most software products today, begin with a minimal viable product (MVP) and go through many iterations if business value is captured. Consider an example of recommending books on an e-commerce website. It might start with a minimal implementation that always returns best-sellers and evolve, as value is validated, to an ensemble of hundreds of machine learning models:

1. Always return “The Hobbit,” “Harry Potter,” and “The Little Prince,” three all-time best sellers 
2. Best sellers of the previous 30 days
3. Best sellers by week and by genre (self-help, business, fantasy, sci-fi, and so on) 
4. Personalized using a simple algorithm such as conditional probability, Jaccard similarity, or cosine similarity 
5. Personalized using an ensemble of hundreds of sophisticated models as was used in the Netflix Prize competition

This same approach works for other “next best action” products such as in pharmaceutical sales and private wealth management.  

MVPs are key to value capture and apply to every industry. A food processing company might plan an AI roadmap that begins with preventative maintenance and evolves to a digital twin simulator as value is captured:

1. Predict the failure of each chicken deboning machine and schedule maintenance prior to failure.
2. Optimize the maintenance schedule to maximize uptime across 50 machines in five processing plants.
3. Predict which spare parts are needed in which plant by day and work shift.
4. Predict how many maintenance engineers are needed in each plant each work shift. 
5. Develop a digital twin simulator that can answer questions such as, “Should we buy spare parts in bulk once a year and stockpile them in each plant?” 

The continuous nature of products changes how budgets are allocated. Rather than estimating which projects might generate value and investing in them, product management can place many small bets and reinvest in what’s proven to work. That’s how we find the best analytics products as AI matures and finally upgrade from Microsoft Excel. 

For further reading on this topic, feel free to check out:

1. “How to Build Great Data Products,” Emily Glassberg Sands, Harvard Business Review, Oct. 30, 2018 
2. “Data Is Code,” Doug Bryan, LinkedIn, Sept. 13, 2021
3. "Why Design Thinking Works," Jeanne Liedtka, Harvard Business Review, September–October 2018, pp.72–79