Data needs a home.
Whether you are shopping online, scrolling through social media, or streaming your favorite show, every interaction generates data. But what happens once that data is created? Before it can be shared, analyzed, or used to support business decisions, it needs a place to live.
In the previous installment of The Data Journey, we explored what data is, where it comes from, and why organizations collect it. We also introduced the high-level lifecycle of Source → Analysis → Use, showing how observations become insights and how those insights drive better business decisions.
This naturally leads us to another question:
Where does all of this data actually live?
The simple answer is that it has changed over time. As people created more data and organizations collected more of it, they needed better ways to store it. What started as simple files evolved into databases, data warehouses, data lakes, and modern data lakehouses.
Each innovation solved a new challenge as businesses grew and their data became more complex.
It Starts with a File
For many people, their first experience working with data is through a spreadsheet.
Whether it is an Excel workbook, a CSV file, or a text file, these are all examples of flat files. They provide a simple way to organize and store information in a single dataset.
Flat Files
Flat files organize information into rows and columns.
- Each row represents a single record.
- Each column represents a field, or one characteristic about that record.
For example, if a spreadsheet tracks a company's product prices over time, each row might represent a single product price record, while the columns store the date, category, scent, and price.
Common file formats include:
- CSV (.csv)
- Text (.txt)
- Excel Workbook (.xlsx)
For individuals and small organizations, flat files work extremely well. They are easy to create, simple to understand, and useful for tracking smaller amounts of information.
Think about a personal budget, a customer contact list, or inventory for a small business. In situations like these, a spreadsheet may be all you need.
But as organizations grow, so does the amount of data they collect. Instead of one spreadsheet, there may be hundreds. Different departments begin maintaining their own files, duplicate information appears, and keeping everything synchronized becomes increasingly difficult.
Eventually, businesses need something more organized.
Businesses Need Organization
Databases
Rather than storing information across hundreds of separate files, organizations often move their data into databases.
Unlike a spreadsheet, which typically stores everything in a single worksheet, databases organize information across multiple related tables. One table might contain customers, another orders, and another products. These tables are connected with relationships, allowing information to be stored once and referenced whenever it is needed.
This organization reduces duplication, improves consistency, and allows multiple people and applications to access the same information at the same time.
Today, databases power many of the systems we rely on every day, including online shopping, banking, healthcare, and streaming services.
But as organizations accumulated more historical data, running large analytical queries against operational databases became increasingly difficult. This created the need for systems designed specifically for reporting and analytics.
Businesses needed more than organized data. They needed answers.
Businesses Need Understanding
Data Warehouses
As organizations accumulated years of operational data, they began asking questions that looked beyond today's transactions.
Rather than asking:
"How many orders were placed today?"
They wanted answers to questions like:
- How have sales changed over the last five years?
- Which products consistently perform the best?
- Which regions generate the highest revenue?
- How has customer behavior evolved over time?
Although operational databases can answer many of these questions, as organizations grow and data volumes increase, running large queries can become inefficient.
Rather than replacing operational databases, organizations began creating separate environments dedicated to reporting and analytics.
This led to the development of data warehouses.
A data warehouse brings together structured data from across an organization into a centralized environment designed specifically for reporting, dashboards, and historical analysis. Data warehouses help analysts identify trends, measure performance, and support strategic decision-making without affecting the systems that keep the business running.
Up to this point, every storage system we've discussed has something in common. They're all designed around structured data that fits neatly into tables.
But not all data fits neatly into rows and columns.
Businesses Need Flexibility
Data Lakes
Today's organizations generate emails, PDFs, images, videos, application logs, JSON files, IoT sensor readings, and countless other forms of information that don't naturally fit into traditional tables.
Preparing all of this information for storage in traditional tables is not always practical, which led to the development of data lakes.
A data lake stores large amounts of raw data in a variety of formats. Instead of organizing everything before it is stored, organizations can keep the original data and decide how it will be used later.
This flexibility makes data lakes especially valuable for artificial intelligence, machine learning, and advanced analytics, where future use cases are not always known in advance.
However, flexibility comes with tradeoffs. Without proper governance, large collections of raw data can become difficult to manage and trust.
Organizations didn't want to choose between structure and flexibility.
The Best of Both Worlds
Data Lakehouses
For many years, organizations often maintained separate environments for reporting and advanced analytics.
Data warehouses provided structure and governance.
Data lakes provided flexibility and scale.
A data lakehouse combines the strengths of both.
It supports structured, semi-structured, and unstructured data while also providing the performance and governance needed for reporting and analytics. Instead of choosing between two different storage platforms, organizations can work from a single, unified environment.
Today, lakehouses are becoming the foundation of many modern analytics platforms.
Modern data platforms are built to support both.
Looking Ahead
In the second installment of The Data Journey, we explored where data lives and how storage has evolved over time. From flat files and databases to data warehouses, data lakes, and lakehouses, we examined how organizations store and organize data as their needs become more complex.
Whether data is stored in a spreadsheet, an operational database, or a modern data platform, it must first travel from its original source to the system where it will be used.
Understanding where data lives naturally leads to another question:
What does the data inside these systems actually look like?
In the next installment of The Data Journey, we’ll explore the different forms data can take. We’ll examine structured, semi-structured, and unstructured data, introduce the concept of a schema, and explain how the way data is organized influences where it can be stored and how it can be used.