Orchestration in Kestra

Kestra is a modern, open-source orchestration and workflow automation platform. It helps you schedule, monitor, and manage complex data workflows—like running scripts, moving files, or calling APIs—using a simple YAML configuration.

Docker is a tool that packages your apps and services into containers—small, isolated environments that run anywhere.

Here’s a step-by-step guide based on your instructions to set up Kestra on your laptop using Docker Compose and GitHub-stored Python scripts:

Prerequisites

• Ensure Docker and Docker Compose are installed.
• Have PowerShell (Windows Terminal is fine) ready.
• Make sure you have a GitHub repo with a Python script to use in your flow.

Step 1: Get Docker Compose File for Kestra

• Open PowerShell.
• Create a folder named docker (or any directory of your choice):

mkdir docker
cd docker

Download the Docker Compose file for Kestra:

Invoke-WebRequest -Uri https://raw.githubusercontent.com/kestra-io/kestra/main/docker/docker-compose.yml -OutFile docker-compose.yml

Step 2: Launch Kestra

docker-compose up -d

Go to http://localhost:8080 in your browser.

Step 3. Set Up Secrets (Environment Keys)

1. Inside the docker folder, create a file .env_encoded.
2. Convert your secrets to Base64 in PowerShell:

[Convert]::ToBase64String([Text.Encoding]::UTF8.GetBytes('YOUR_SECRET_KEY'))

3. Copy the output and paste it in .env_encoded, like:

MY_SECRET_KEY=BASE64ENCODEDVALUE

****All keys must be in ALL CAPS and base64 format****

Step 4. Restart Docker with Updated Env

1. Stop containers:

docker-compose down

2. Make sure you're still in the correct directory.
3. Launch again:

docker-compose up -d

Step 5. Upload Secrets in Kestra

• Open http://localhost:8080
• Navigate to Secrets
• Add the base64-encoded secrets from .env_encoded.

Step 6. Create a Flow That Runs a GitHub Python Script

1. Click Flows > Create
2. Use the plugins such as io.kestra.plugin.scripts.python.Python to run the Python script.
3. Your flow YAML might look like this:

id: run-python-script
namespace: my.namespace

tasks:
  - id: run_python
    type: io.kestra.plugin.scripts.python.Python
    inputFiles:
      script.py: https://raw.githubusercontent.com/YOUR_USERNAME/YOUR_REPO/main/script.py
    script: |
      python script.py

Step 7. Commit and Merge

• After verifying everything works:
  • Push your changes to GitHub.
  • Merge to the appropriate branch.

There you have it, orchestration in Kestra!