Let's elevate the role of analytics
How data-forward companies generate more impact with analytics.
The modern data stack is now proclaimed dead. The shift from the "pre-modern data stack" has happened. SQL is cool again; new roles have been created, cloud data warehouses have been deployed, and new categories of tools have been established.
The massive technological progress over the past few years is undeniable. The sheer power of cloud data platforms and maturing tools to move, transform, and present data empowers data teams to start building data ecosystems with datasets of almost any size.
Yet, something still doesn’t feel right.
Despite this phenomenal technological progress and the growth of teams, I often hear a somewhat bitter sentiment:
Analytics still needs to be taken seriously.
Data as a second-class citizen
Last week, I had an opportunity to spend a morning with a group of data practitioners. Not too far into the first hour of our conversation, we ended up in a very familiar discussion, one that repeats frequently:
"They don’t get it. They don’t care about data."
I could sense the frustration. One person described how engineering teams often break analytics data, not realizing the consequences it might have. Others chipped in with similar stories, topped with the sentiment that stakeholders and leadership still see data teams as a service center.
Data teams are caught in the middle between stakeholders demanding new dashboards, drill-downs, reports, and "insights," and data-producing teams that exhaust data into analytics infrastructure with no time to invest in analytics, busy with their roadmaps.
This can make working in data feel like being a second-class citizen.
The diagnosis
Many companies and data teams are stuck in a self-reinforcing negative spiral:
Teams across the company don’t intentionally invest in producing high-quality data. Data teams have difficulty getting wider teams to act when data goes wrong. It's seen as less important than their own goals. The new feature has to be shipped. The deal has to be closed. Producing data is deemed less critical.
The stakeholders demand dashboards and insights. The most common use case of data is still reporting or analysis to support strategic decision-making. Analysts take requests from stakeholders to create new dashboards, and reports, or to provide insights (often unclear what insight even is). They act as a service function, crunching tickets and never-ending streams of ad-hoc requests.
Analytics teams are away from the source, so they have no option but to patch together data they can get into complex DAGs of transformations to power an increasing number of various dashboards. They often have to act fast without a mandate to build the foundations right.
With the above combined, the analytics platform is often under-architected due to many incohesive increments of ad-hoc requests and data that feed the platform from across the company that was barely designed for such a purpose.
As a result, the analytics platform is not trusted. Data is never really 100% right. When severe use cases for analytics data emerge, the product and engineering teams often build "their own" side platform instead.
And since data is mainly used for decision-making support and can always be replaced or "excused to be wrong," the loop starts again. More ad-hoc requests. More messy data. More patches. More issues. Less trust.
—
And so, is the "they don’t get it" sentiment the root cause?
I don’t think there is any bad intent here at all. Teams that interact with data analytics teams, either as producers of the data or stakeholders that consume it, are busy building the rest of the business. They simply don’t think about data analytics nearly as much.
I thought this was an incentive problem, especially on the data producer side. I’ve heard many data practitioners say that the "push for data" has to come from the top. Maybe, to an extent. But I don’t think that is it.
Instead, I think we should go deeper. This is an incentive problem, but less about setting data as a priority in the next OKR and rather defined by what role analytics plays in the business.
Role of analytics in the business
Data analytics teams across companies often start with a goal to use data to help teams understand how the business works. Analytics is used to generate insights as an ingredient of decision-making. Over time, the goal is to ensure that many important decisions always use data.
But as much as it's a noble idea, it only affects critical business processes and customers indirectly. In between the data and the impact is a decision-maker.
One way you can put this to the test is to quantify its ROI. How do you quantify the value of the data that leads to a “better decision”? If you try enough, you could—at least somewhat—compare possible outcomes of contradicting choices, but this is still relatively weak. This lack of clarity on impact makes the whole data use optional. Decision-makers should use data, but in the end, they don’t have to.
Making data critical
Yet, there are companies that “made it.” Their analytics and data teams have made one significant shift: They deployed analytics data into business-critical processes.
I recently spoke with a couple of data practitioners who described how that looks like:
data practitioner who explained how data from their battery factories goes through Redshift and dbt aggregations to identify faulty machines that must be stopped and fixed.
Or a logistics company modeling their supply and demand in Snowflake to drive their operations.
Or a grocery business that crunches stock levels, predicts demand and automatically puts purchase orders to suppliers entirely based on data from BigQuery.
Or a company managing a large fleet of batteries to balance energy in the grid, forecasting demand, and bidding energy capacity to the grid powered by ClickHouse.
And the extreme, Synq is built as an operational data platform. Our analytics is the business.
These use cases are exciting.
They put analytical data at the company's heart and deploy analytics to business-critical use cases. This approach shifts incentives.
Analytics can no longer be a second-class citizen. In the end, it runs the business. The deeper data goes, the more impact and leverage it creates for the data analytics teams. As a consequence, analytics gets more budget, more senior roles, and more attention. When things go wrong, everyone understands data analytics has to be fixed.
But perhaps you’re now—as much as I was—wondering: how did they get there?
—
Unfortunately, I don’t know the shortcut. But what I can see is that they have a couple of things in common:
The first ingredient is a champion—or even better, champions. It is invaluable to find believers in the company leadership or executive sponsors who will fight for data with you, but that is likely not enough. The magnitude of cultural change needed is too big for a top-down approach. Instead, think about yourself or the people on your team. Who are the true champions who want to make the shift, elevate data, go business critical, daring to go deeper into the business? They need support and encouragement to act and drive the day-to-day work to make this shift happen.
The second is technology and quality. Warehouses, (reverse) ETL, CICD, and observability are all mature enough to support the business-critical use of data but you need to prioritize quality proactively. Without a reliable platform, you can’t support business-critical systems. Invest in a solid foundation, care about reliability, and ensure that the analytics infrastructure is trusted.
Most importantly, you will have to figure out where to start.
Think in increments.
Get to know your business and obsess about how data analytics can play a more critical role. What would be the next immediate step?
Keep escalating the criticality of use cases and build a flywheel.
More critical use cases, push on quality, more evident value, and more investment in data. Repeat.
You can get a better snapshot of the customer in front of support reps, automate part of customer conversion based on data triggers, and help CX prioritize customers to upsell or flag churn risk. These small use cases take you away from dashboards and the ad-hoc world into data being used directly in actionable workflows.
If you do well, you’ll start pushing data directly in front of customers. In-app analytics, automated customer workflows, and model training on top of analytics data will all be critical to the success of your company and the value you deliver to customers.
If you want more details, I recommend following Robert Sahlin, who has introduced me to the concept of data flywheels and has implemented them firsthand.
Conclusion
I hope to see meaningful progress on this front and hear more and more stories of critical data use.
We just need to be bold and push. We need bold data analytics leaders who are fearless in putting their data into business-critical use cases that generate more pressure on the quality of their teams and systems. We need company leaders who demand data teams go beyond ad-hoc analysis and give them a mandate to build things right. We need practitioners who dare to fight more for the “doing it right” mentality.
Perhaps the shifted attention to AI—which heavily relies on data quality—is a trend that teams can use to elevate their work. As analytics technology and teams evolve, we could generate even more momentum for data and incentivize companies to take it seriously and make data teams first-class citizens.
Image credits—https://unsplash.com/photos/twom-white-flying-rockets-during-daytime-MEW1f-yu2KI