Data needs a blueprint.
Just as a building needs more than walls and a roof, organizations need more than individual systems to manage their data. They need a blueprint that determines how everything connects and works together.
In the previous installment of The Data Journey, we explored where data lives. From flat files and databases to data warehouses, data lakes, and modern lakehouses, we learned how organizations store and organize the information they collect.
But having a place to store data is only one part of the picture. Organizations also need to consider how data moves between systems and reaches the people who need it.
This naturally leads us to another question:
How do organizations design systems that support their evolving data needs?
The answer lies in data architecture. Let's explore what data architecture is, what makes an architecture effective, and some of the different approaches organizations use to manage their data.
What Is Data Architecture?
Data architecture provides the blueprint for how an organization collects, stores, manages, and uses data to support its changing business needs.
Think about designing a building.
Before construction begins, an architect needs to understand what the building will be used for. A hospital, an office building, and a residential home all serve different purposes, requiring different designs.
The architect must consider the layout, infrastructure, accessibility, safety, and potential for future expansion.
Data architecture follows a similar principle.
Before choosing technologies or building data pipelines, organizations need to understand what they are trying to accomplish.
For example, a retail company may need to collect customer transactions, monitor inventory, analyze historical sales, and provide dashboards to business leaders. Each requirement influences how its data systems should be designed.
A data architecture brings together the different systems and processes needed to move information from its original source to the people and applications that use it.
The diagram above illustrates the major components of a modern data architecture and how they work together:
- Data Sources: Where information originates, including operational systems, files, cloud applications, and external sources.
- Data Ingestion: How information enters the environment through pipelines, APIs, and other integration methods.
- Data Storage: Where information is maintained, including databases, data warehouses, data lakes, and lakehouses.
- Data Processing: How information is prepared and transformed for different needs, often through the same pipelines used for ingestion.
- Data Consumption: How information is accessed and used for dashboards, analytics, reporting, and artificial intelligence.
Although there are common architectural designs and best practices, the right architecture ultimately depends on the organization's requirements, available resources, and intended use cases.
What Makes Good Data Architecture?
Connecting systems is only part of designing a successful data architecture.
A good architecture must provide a dependable foundation that supports an organization's current needs while allowing it to adapt as those needs change.
There are several important characteristics to consider.
Supporting Business Requirements
The most important consideration is understanding what the business actually needs.
An organization analyzing historical sales may have very different requirements from one monitoring financial transactions in real time.
The goal is not to build the most complicated system or use the newest technologies. It is to design an environment that supports the questions the organization needs to answer.
Good data architecture starts with the business problem, not the technology.
Scalability and Elasticity
As organizations grow, the amount of data they collect and process often increases.
Scalability refers to a system's ability to handle growing workloads by expanding its capacity, while elasticity refers to its ability to adjust computing resources as demand changes.
For example, an online retailer may experience significantly more traffic during the holiday shopping season. An elastic system can increase its resources during periods of high demand and reduce them when demand returns to normal.
These characteristics help organizations support growth while using resources efficiently.
Availability and Reliability
Data is only useful when people and systems can access it.
Availability measures how often a system is operational and accessible, while reliability describes its ability to function consistently and correctly over time.
For example, a system with 99.99% availability would have an annual downtime allowance of approximately 52.6 minutes, assuming a 365-day year.
Even small amounts of downtime can significantly impact organizations that depend on continuous access to their data.
This is why data architectures often incorporate monitoring, redundancy, and recovery mechanisms to reduce disruptions.
Security and Governance
As organizations collect more information, protecting that information becomes increasingly important.
A well-designed architecture should prioritize security through measures such as access controls, encryption, and monitoring.
Data governance establishes rules around data quality, ownership, and consistency.
Together, security and governance help ensure that information is accessible to the right people while remaining protected and trustworthy.
Ultimately, good data architecture balances these characteristics based on the organization's needs. Not every system requires the same performance, availability, or level of complexity.
Different Approaches to Data Architecture
Now that we understand what makes an effective data architecture, let's explore some of the different approaches organizations use.
One important consideration is how frequently data needs to be processed.
Some organizations can rely on scheduled updates, while others need information almost immediately after it is generated.
From a processing perspective, we can explore three common approaches:
1. Batch-focused
2. Streaming-focused
3. Hybrid architectures
Batch-Focused Architectures
Batch processing involves collecting data and processing it in groups, often at scheduled intervals.
Rather than continuously updating information after every event, a system might process data every hour, every night, or once a week.
Imagine a retailer that wants to analyze its daily sales performance. Instead of updating its reports after every transaction, the company could process its sales data overnight and provide updated reports the following morning.
Batch processing is commonly used in data warehouse pipelines, data lake processing, and other analytical environments where immediate updates are unnecessary.
Streaming-Focused Architectures
But what happens when waiting until tomorrow is not an option?
Stream processing allows systems to continuously receive and process information as events occur.
For example, a financial institution monitoring suspicious transactions may need to identify unusual activity almost immediately rather than waiting for a scheduled batch process.
One streaming-focused approach is Kappa Architecture, which uses a stream-processing system for incoming events and supports reprocessing historical events by replaying retained data through that system.
This approach can be useful when organizations rely heavily on continuously generated information.
Hybrid Architectures
Not every organization needs to choose between batch and streaming.
Many businesses require both historical analysis and near-real-time information.
One example is Lambda Architecture, which combines separate batch and real-time processing layers. This allows organizations to maintain comprehensive historical results while also providing faster updates from incoming data.
Modern data lakehouse platforms can also support both batch and streaming workloads, although a lakehouse describes a broader data platform architecture rather than a specific processing pattern.
Architectural approaches are not limited to processing frequency, either.
For example, the Modern Data Stack describes an ecosystem of technologies for ingestion, storage, transformation, and analytics, while Data Mesh emphasizes decentralized data ownership across business domains.
These concepts address different architectural challenges and can coexist with batch, streaming, or hybrid processing systems.
There is no single architecture that fits every organization.
The most appropriate approach depends on how quickly information is needed, the complexity of the environment, and the business requirements.
Looking Ahead
In this installment of The Data Journey, we explored how data architecture provides the blueprint for connecting the different systems organizations use to manage their information.
We learned what data architecture is, what makes an architecture effective, and how batch, streaming, and hybrid processing approaches support different business needs.
Whether an organization is analyzing historical transactions or responding to real-time events, the goal remains the same: to design systems that deliver reliable, secure, and accessible data to the people who need it.
But understanding where data lives and how it moves through these systems introduces another question:
What does the data inside these systems actually look like?
In the next installment of The Data Journey, we'll examine structured, semi-structured, and unstructured data, introduce the concept of a schema, and explain how the way data is organized influences where it can be stored and how it can be used.