The Data Engineering Lifecycle has multiple steps, and at first glance it can be hard to know what each of these means. Put simply, it represents the journey that data takes from raw data to structured data for specific use cases. Each part of the diagram above is a unique and essential part of this journey.
Below I will go into this step by step to help make sense of which each part is doing.
Generation:
Where is data coming from and in what format? This can be anything from sales on a till in a cafe, to clicks on a website, or it can be pre-existing data thats been used somewhere else.
As engineers we need to consider when its generated, its volume, and how long it stays in the database. We also need to think about whether it is really the start of the pipeline or if there is somewhere earlier.
Potential risks: bad data quality, schema changes, system failures, access issues
Ingestion:
This is the Extract and Load steps. Ingestion can be done either as Batch (in bulk, usually based on time intervals or data size) or Stream (live flow of data). Data Engineers will primarily use Batch Ingestion, because Stream is required more rarely.
Transformation:
Here we define the purpose of transforming data and the end product. The process of working can be done in three ways: ETL, ELT or EtLT. Alongside this, we also need to think about other things such as whether data should be normalised (splitting into multiple tables) or denormalised.
Data Transformation often follows a Medallion Architecture Pattern:
- bronze = raw data,
- silver = data modelling decisions,
- gold = use case structure.
Serving:
This is how the data is going to give business value - the 'why'. The three outputs include Analytics (data structured for charts and dashboards), Machine Learning (data quality can be a big issues here! also need to consider data discoverability and data bias), and Reverse ETL (this is when end point goes back into a source system)
Storage:
Data storage has evolved a lot, growing from from data warehouses to data lakes, to the newer data lakehouses. It is important to consider where the best place to store data is.
There are also key undercurrents that play into these steps, such as Security, DataOps and Architecture. The Data Engineering Lifecycle involves connecting these steps and undercurrents to help data flow efficiently and effectively, so that it can be used in the best way possible.