Setting Up a Data Engineering Project

We’re now in the second week of data engineering training, and even in just over a week we’ve covered a lot, so it’s been a lot to take in but in a good way, and it’s been exciting learning something so new. One of the things that stood out to me the most was setting up a data engineering project, because it’s the first thing you do whenever you start a new one, so it felt like a good place to start for my first data engineering blog.

Before you write any code in a data engineering project, there are a few steps worth doing first. In this blog I’ll walk through the setup we used in training, which was in Python and VS Code on Windows, so the commands might look a bit different depending on your tools, but the ideas behind each step are the same.

Step 1: Create a repo on GitHub

A repository (repo) is where your project lives on GitHub. When you create it, turn on Add README and choose a .gitignore template to add two files:

  • README.md: explains what your project does, why it’s useful and how someone can get started with it. Even if you’re the only one using the project, future you will thank you for it.
  • .gitignore: tells Git which files not to track, so they never end up on GitHub. GitHub has templates for different languages, and we used the Python one.

If your repo is public, it’s worth adding a license, and MIT is a common choice. It means anyone can use your code, but you’re not responsible for how it’s used or what might break.

Creating the repo with a README, the Python .gitignore template and an MIT license

Step 2: Clone the repo and open it in your code editor

Cloning makes a copy of the repo on your laptop, so you can work on it locally. Clone it from GitHub or GitHub Desktop, then open the folder in your code editor. We used VS Code.

Cloning the repo with GitHub Desktop from the Code button

Step 3: Create a virtual environment

What is it? In Python, a virtual environment is a self contained area of your laptop with its own Python and packages, kept separate from everything else.

Why do you need it? To avoid conflicts. If Project A needs pandas 1.3 and Project B needs pandas 2.0, installing both on your system causes problems, and fixing that can take hours.

How do you set it up? In the VS Code terminal on Windows, we ran:

Plain text
python -m venv .venv
.venv\Scripts\activate

On a Mac, the second line is source .venv/bin/activate. You’ll see (.venv) at the start of the terminal line, which means you’re now working inside it.

(.venv) at the start of the line shows the virtual environment is active

Step 4: Install packages and create requirements.txt

What is it? A file listing the packages your project needs.

Why do you need it? So your code can run on someone else’s laptop with the same package versions.

How do you set it up? Install the packages you need, then save them to the file. For our extract scripts, we installed requests and python-dotenv:

Plain text
pip install requests python-dotenv
pip freeze > requirements.txt

If you’re cloning someone else’s project instead, run pip install -r requirements.txt to install everything in one go.

requirements.txt from our BikePoint project, which also includes the packages for the load script

Step 5: Create a .env file

What is it? A file that stores your secrets, like API keys, usernames and passwords.

Why do you need it? So you never type secrets straight into your script, where anyone could see them once it’s on GitHub.

How do you set it up? Create a file called .env and add your key:

Plain text
API_KEY=your_key_here

Then read it into your script. In Python, we used the dotenv package:

python

Python
import os
from dotenv import load_dotenv

load_dotenv()
api_key = os.getenv('API_KEY')

Step 6: Update your .gitignore and check it

The .env only keeps your secrets safe if Git ignores it. For our projects, we added these to the .gitignore:

Plain text
.env
.venv/
data/
logs/

Then run git status. If .env isn’t listed, it’s being ignored and you’re safe to commit and push.

The .env file is greyed out and doesn’t appear in git status, so Git is ignoring it

Reflection

What I found most interesting is that every project started with these same steps, whatever the data was. We used this setup for both the BikePoint and Amplitude projects, and it’s now a checklist I can reuse. It’s tempting to jump straight into the script, but getting the setup right first keeps your secrets safe, lets your code run anywhere and means future you can understand what you built.

Author:
Aisha Senkubuge
Powered by The Information Lab
1st Floor, 25 Watling Street, London, EC4M 9BR
Subscribe
to our Newsletter
Get the lastest news about The Data School and application tips
Subscribe now
© 2026 The Information Lab