Data is only useful with context. For example, a date is just one day in the time continuum. But, if that data happens to be the day you were born, it has much more meaning. Comparing your birthday to today's date, we can interpret your age and gain more insights about you. Thus, knowing what data represents allows us to put it in context—making it much more valuable.
Data context empowers us to better understand how data sets relate to each other and glean richer insights into why events occur in the real world. Consequently, data context is a crucial component of your data management strategy.
Table of contents:
Data context refers to the background information and relevant details that surround and describe a dataset. In databases, this data context is stored in the form of metadata. The metadata provides users with a deeper understanding—citing the meaning and implications of the dataset.
Gartner defines metadata as information that describes various facets of an information asset to improve its usability throughout its life cycle. Essentially, that means metadata provides answers to the following questions about a particular dataset:
Who collected the data
What the data is about
When the data was collected
Where the data was collected
Why the data was collected
How the was collected
Answering these questions has always been an essential part of effective data analysis. But the deluge of data and data sources created through new technology streams is driving the need for more robust context strategies. Modern data modeling and data-driven decision making require unfettered access to data from across data silos. Advancements in AI and ML are also driving the rapid transformation across the data ecosystem—from management to consumption.
Metadata is the key to organizing this increasingly complex and dynamic ecosystem. The more context your data has, the more it can be tracked, organized, and trusted—providing the most value for your business or organization.
With the data ecosystem becoming so complex, tracking data quality and integrity is increasingly important and much more challenging. If decision-makers don’t trust the data they are consuming, the investment in collecting, storing, and cleansing it is wasted. On the flip side, a solid data context strategy offers many benefits. Below, I’ll cover a few of the most notable.
1. Data collection and lineage
Metadata can provide context around how data was collected. This includes sample sizes, collection date and time, and the identity of the collector. All of this context helps analysts understand the validity of different datasets. For example, if a specific set of data is collected by a reputable entity with a large sample size and solid methodology, you will put more trust in it.
The rapidly growing complexity of data pipelines makes data lineage vital for data integrity, and your metadata is at the core of data lineage. Systems that track data sources and how datasets are constructed ensure analysts and engineers understand the process. That understanding helps your data team have confidence that pipelines are operating effectively and producing quality data.
2. Data quality and governance
Metadata is a crucial component of data quality and governance. Without information on datasets, you can’t apply data governance rules. For example, if data access rules are defined by the country where it was collected, you need to have the metadata that tracks this information. Metadata can also help you define data ownership and stewardship, monitor and enforce data access policies, and comply with data protection regulations.
Similarly, metadata is essential for tracking and implementing data quality metrics and rules. For example, if you measure and monitor the quality of a data set by its variability or number of null values, you need to track and manage this metadata. The more granular this data is, the more robust, flexible, and targeted rules and policies can be. More targeted rules and policies allow you to be more effective in managing compliance without limiting legitimate data access.
3. Data standardization and integration
Metadata can also be a valuable component in integrating and standardizing data—providing vital context on data structure, types, and formats. With more information about your data, mapping is much easier. Identifying matching fields, handling null values, and resolving data conflicts is also simplified.
Metadata-driven integration is a growing strategy to support richer data sets with greater business context. In many cases, data is integrated using hard code and scripts. A metadata-driven approach is much more flexible and automated, using rules that evaluate metadata to generate mapping code that can be used for integration.
As the data world gets more complex and dynamic, managing this metadata becomes much more important. Demand for metadata from modern data stack tools and technologies is only increasing.
Everyone gets more value from more information and context. Greater clarity at one level also supports greater context at subsequent layers of data analysis. Consequently, the entire data ecosystem should be thinking about data and its context.
Successful data context management strategies incorporate data producers, data consumers, and metadata managers. Each of these three groups must have a well-defined role in executing a successful metadata management strategy.
Metadata producers
Metadata producers are the architects and engineers who are building systems and work most regularly with metadata. The pipelines they build not only produce metadata as a byproduct, but they create metadata to organize and label data sets. Better data context strategies make their jobs easier and enable them to deliver better data products and pipelines. Metadata producers need to be involved in data management strategy to ensure it meets their needs.
Metadata consumers
Data consumers are the analysts and data scientists that need to understand the data and its context. Data analysts must be able to interpret the metadata and leverage it to tell more insightful, rich, and valuable stories about data. When data is enriched with context, analysts can interpret causation, trends, forecasts, and use data for decision intelligence.
They can also add multiple layers of context to their analysis. By incorporating additional datasets that contain different description attributes, you gain a clearer, more holistic understanding of how real-world events are impacting your business. This approach leads to better decisions. That’s why data management strategies should aim to make it easier for data consumers to find, understand, and integrate data for more sophisticated analysis.
Metadata stewards
In enterprises and more sophisticated organizations, metadata stewards or managers are focused exclusively on metadata management. They are tasked with designing, implementing, and maintaining metadata systems. These metadata stewards may also be responsible for data catalogs that use metadata to make datasets more discoverable by data consumers.
One of the biggest challenges of data producers and consumers is integrating data that resides in different databases. Metadata stewards can simplify this by creating strategies to share metadata. Creating processes to incorporate foreign metadata entering their system and formatting metadata headed to other systems to make it easier for partners to ingest can facilitate greater access to data across systems.
A wide variety of strategies are being developed across the ecosystem to provide greater access to data, including dynamic strategies such as data mesh and data fabrics. These approaches require new metadata management frameworks that allow data to more easily and effectively move across silos. Metadata management strategies are migrating from passive approaches to more active ones to support these changing architectures.
Active metadata management
Traditionally, metadata has been managed passively by aggregating and storing it in a static data catalog. A new approach, known as active metadata management, allows metadata to flow bi-directionally across your entire data stack.
This bi-directional flow enables context to move across systems where it can be cross-checked and combined with other data sources. Active metadata also embeds metadata in all your platforms in your data stack, where it can be enriched. With a richer understanding of the data around us and its context, you can do much more with it.
For example, relevant metadata can be embedded in BI solutions so analysts don't have to switch from a BI tool to a data catalog to understand the context of the data that they are analyzing. In turn, this process empowers non-technical business users to access and find quality data insights through self-service analytics.
Active metadata management is always on—actively collecting and processing metadata from across your stack. It is action-oriented and can generate recommendations or alerts to help analysts and engineers make better decisions. For example, you may set up notifications based on certain data quality metrics and thresholds. That notification might recommend that you deactivate or fix a pipeline where the metadata around the output signals data quality issues.
Data enrichment
With more dynamic metadata management, data context can be taken to the business level through data enrichment. More metadata means you’re able to produce richer datasets with valuable real-world context to understand what is happening and why.
For example, knowing that a consumer searched for a particular product on their phone at a specific time and from specific GPS coordinates is valuable. But combining that data with location information can tell you that this data point was collected outside a Walmart. Adding data on the operating hours of the particular Walmart will tell you if the data was collected outside a Walmart after hours. This level of context provides much more valuable information to help drive business decisions, such as adjusting operating hours.
Metadata strategies are being reimagined in the face of the modern data stack. They are going to get very complex very quickly. Having a sound plan will be key. Adopting metadata standards and building a team to manage the evolution of your metadata strategy will improve your chances of success.
Ready to start seeing value in your modern data stack investments?
Providing your front-line decision makers with the ability to search and surface business insights with AI-Powered Analytics is the best way to see direct ROI from your data team’s hard work.
That’s why businesses like Comcast, CVS, and Afterpay turn to ThoughtSpot to find the insights hiding in their data. Start your free trial to see the value for yourself.
