Businesses have been championing “digital transformation” for over two decades now. In 2022 alone, Statista projected spending on digitalization to reach $1.6 trillion. Despite the high hopes, time, and money these organizations have invested, many still aren’t reaping the results they expected.
Take business intelligence, for example. With all the hype around big data, machine learning, data lakes, and cloud, you’d expect most businesses to feel confident using data to make decisions. In reality, according to Deloitte, only 10% of today’s executives believe their company is analytics mature.
So, what gives?
In most cases, it comes down to a lack of a solid data infrastructure. True digital transformation isn’t about throwing tools at the problem and hoping for the best. It’s about creating and refining the systems, models, tech stack, and company culture to enable fundamental organizational change.
For companies hoping to dominate the decade of data, it’s past time to take action.
What is data infrastructure?
Before Sully McConnell was Head of Insurance with Snowflake, he was Chief Data Officer at The Hartford, a purpose-driven insurance firm with a 200-year-old legacy. When he joined The Hartford in 2018 he knew he’d got his work cut out for him: “The bad news is we are a 200-year-old company, but of course the good news is that our systems are only a hundred years old!”
Transforming the venerable insurance firm into a modern data-driven business would take a complete overhaul of their systems, processes and culture. In other words, McConnell needed to build a modern data infrastructure.
To quote the Open Data Charter, “A data infrastructure consists of data assets, the organizations that operate and maintain them, and the processes, policies and guides describing how to use and manage the data.” The modern data infrastructure consists of all the components you need in your data pipeline:
The physical infrastructure (storage hardware, processing hardware, data center)
The information infrastructure (databases, data warehouses, virtualization systems, cloud services)
The business infrastructure (business intelligence tools, analytics tools, AI and machine learning software)
The people infrastructure (the processes, guidelines, accessibility, and governance that control how people interact with the data)
Why is a modern data architecture important in 2023?
Despite the incredible developments in data and technology over the past decade, the majority of companies are still failing to take full advantage of data. According to a survey from Harvard Business Review, 86% of frontline workers have difficulty accessing data and insights. Meanwhile, adoption rates for analytics solutions have stagnated at under 30%, research from Eckerson Group shows. Business users are still waiting an average of 4-5 days for dashboard reports.
On the flip side, leaders in this area are embedding engaging analytics experiences into apps, using data to create entirely new revenue streams, and optimizing business decisions with modern self-service analytics.
In other words, data is the new battleground for business. And your data infrastructure plays a big role in your company’s ability to succeed.
McConnell agrees.
"The pace of change in our field is crazy…driven in large part by data volumes growing and the need for data freshness to grow,” he said. “As well as the need to embed analytics into business processes and the importance of cloud."
If you are struggling with siloed data, static dashboards, and customized reporting, you won’t be able to keep up in 2023.
5 tips for building a modern data infrastructure
So, what does an effective data infrastructure look like? Here are 5 tips to create the infrastructure that will take you from where you are to where you want to be:
1. Modernize and unify your data tech stack
Fragmented data begets fragmented insights. Your data team should not have to spend hours on remedial data prep tasks and maintaining fragile data-to-dashboard pipelines. Yet according to Deloitte, 92% of data workers report spending time on menial operational tasks outside their roles, resulting in less time to spend on the sophisticated analysis you hired them for.
A fragmented data stack also leaves room for inconsistent data. Say for example a business is curious about the average deal size in Q4. They might receive three completely different answers from three analysts working on data from disparate sources.
Fragmented insights often look like this:
How to modernize and unify your data stack
Modernizing your tech stack doesn’t mean moving over to a one-size-fits-all solution. Rather, it’s about selecting a centralized cloud data platform for all your data, and then adding in best-of-breed, cloud-native tech solutions to create a modern data stack.
The result is a cleaner, more collaborative way to look at your data. In fact, the Deloitte study found that 80% of companies that have unified their tools and methods for working with data report that they have exceeded their business goals.
Some tips for what to look for when considering adding a new technology to your stack:
Is it easy to try and deploy? Will it help us move quickly to take our place in the defining decade of data?
Is it cloud native? Can it take advantage of live connectivity from cloud data platforms at scale?
Can it handle modern data volumes–while also busting data silos?
Is it open and straightforward to integrate into our tech stack? Will it be easy to adapt in the future as our data strategy evolves?
Does it have a user experience layer built in–and was it designed with the business user in mind?
Will it help us put the value of our stack into the hands of every employee?
One example of a modern data stack might look like this:
|
Huel: Perfecting the recipe for the modern data stack For an example of building a fit-for-purpose data infrastructure, look no further than Huel, a healthy food company. Back in 2019, Huel knew they needed to be able to make data-driven decisions, faster. Their existing data stack relied too heavily on siloed data and spreadsheets, meaning that they had no single source of truth. Huel’s solution was to build a new data infrastructure with best-in-class tools, which would allow their growing team to self-serve the insights they needed to make better decisions at pace, freeing up their data team to add strategic value. The outcome? Business users at Huel can find the answers they need in minutes, not hours. Meanwhile, their data team can focus on answering the big business questions that drive bottom-line results. The Huel team also tells us that their company’s data literacy is on the up, as more and more users engage with their easy-to-use data stack. |
2. Build a scalable foundation
The right data infrastructure should balance flexibility and data governance to keep your data clean and allow room for iteration as you grow. When choosing a data platform, look for one that has these three dimensions:
Performance under pressure: Can this solution still give you a delightful experience even when handling hundreds of billions or even trillions of records?
Ability to serve millions of users: Does the tool work when your user base is in tens of thousands? Or, if you’re using it externally with customers, possibly millions of users?
Adaptability: As your use cases grow in number and complexity, can the platform keep up?
Once you’ve selected a cloud data platform, you need data modeling and transformation tools that:
Connect directly with your data sources
Allow you to model in SQL and dbt
Let you build flexible, iterative data models that will let users ask new questions you didn’t anticipate when you built the model
3. Swap out static dashboards for modern self-service analytics
Dashboards are dead. The world is moving faster than ever before and that means your frontline workers need to know what’s happening in your business as it’s happening. However, many businesses still rely on legacy BI dashboards that create a dead zone between data and their analytics layer.
Extracting data onto servers or desktops means that reports are being built on yesterday’s news. Worse, given how long many of these reports take to prepare and customize, more than 40% of business insights in these companies are based on data that is 2 months old–or even older.
Instead of creating dead zones, your analytics solution needs to sit directly on top of your cloud data platform, so that everyone in your organization can engage directly with the live data.
4. Empower users with Search and AI
A truly effective data infrastructure doesn’t just support the needs of the data experts—it’s also user-friendly for frontline workers. It’s crucial to address the fact that 86% of frontline workers have difficulty accessing data and insight, and adoption rates are under 30%. To do this, look for a platform that can offer these things:
Delightful UX: Your data platform should be as easy to use as your favorite app. When looking for a modern data analytics solution, test it out with your technical users and your non-technical users—the UX should be user-friendly to both.
Accessible data: Your infrastructure needs an experience layer that makes the data incredibly accessible, even to non-expert data users. Otherwise, you’ll soon notice your business users are back to exporting and manipulating the data they want in Excel sheets.
Search functionality: Your new data infrastructure needs to incorporate search functionality, so that users can find the information they want in the way that they’re used to searching. They also need to be able to search via live query, so that no aggregations or data movements are required.
Powered by AI: To compete at pace with the data leaders, your infrastructure needs to be augmented with AI. You’ll need automated tools to help users by surfacing data insights they would otherwise have missed.
|
Accern: Empowering customers with no-code AI and modern data analytics As a platform that provides fast and accurate insights for financial services, Accern is in the business of actionable data. Their challenge was customization—they were limited by the single dashboard provided to them, and weren’t able to provide self-service access to data visualizations. “There was an undeniable need for customization,” Christian Felix, VP of Product at Accern, said. “We were providing customers with value on the analytics side, but needed something that would visualize insights better.” By choosing a modern analytics platform, they were able to provide seamless implementation with their existing data and tech stack, and boost adoption of analytics within their product. Merging their no-code AI capabilities with their new data analytics platform capabilities—such as scriptable deployments and reusable worksheets—helped Accern’s team embed in hours what would normally take other solutions several days. “As we continue to improve Accern’s data customization, it naturally improves the quality of our data output and the visualizations customers have access to,” Felix said. Read more. |
5. Drive action with embedded analytics
If you want employees to actually use your data infrastructure to support decision-making, you need to give them the information they need when and where they need it. In other words, engaging analytics experiences must be embedded.
“This is where the real revolution starts,” Matt Lukowski, Director of Analytics at Harri, said. “This is the first time in the history of this industry where we put our users in the driving seat. They decide where they want to go. They decide what they want to search for. They decide which areas they want to explore.”
This is also crucial for reaching and engaging the modern day customer. According to Harvard Business Review, eight out of ten customers attempt to take care of matters themselves before reaching out to a live representative. With embedded analytics, you’re giving customers valuable insights they can use to make informed decisions, while increasing customer satisfaction and loyalty.
Moving from data-aware to data-driven starts here
Building the right data infrastructure is the first step in going from data-aware to data-driven. With data infrastructure that is cloud-based, user-friendly for the whole team, and driven by self-service and embedded analytics, your journey from data to insight to action becomes exponentially faster.
Digital transformation is a marathon not a sprint. Taking the time and investment to build the right data infrastructure is the starting point—and once your modern infrastructure is in place, you’re off to the races.
Looking to take the first step towards building a new-and-improved data infrastructure? Request a 10-minute 1:1 demo with ThoughtSpot.
Tony Hammond is the VP of Solutions Engineering at ThoughtSpot where he provides guidance to customers on how to leverage the power of data to make meaningful business decisions. He has over 25 years of experience in the enterprise software space, specifically in performance management, business intelligence, and analytics.
