Data Engineering Bootcamp: Python, dbt, Airflow & SQL

Want to Become a Data Engineer? Start Building Real Data Pipelines!

Have you ever wondered how companies collect data from multiple sources, transform it into useful information, and deliver reliable datasets for analytics and business decisions?

What happens when an API stops responding, a database schema unexpectedly changes, or a scheduled data pipeline fails in the middle of the night?

And how do professional data engineers build systems that can recover from these problems without losing important information?

Welcome to Data Engineering Bootcamp: Python, dbt, Airflow & SQL!

This practical, project-based course introduces you to modern data engineering through a complete learning journey covering data ingestion, transformation, modelling, orchestration, testing, and production-style operations.

Rather than learning disconnected tools, you’ll explore how different technologies work together to build reliable data pipelines.

At the centre of the course is the FreshCart Data Platform, a practical data engineering project that demonstrates how data moves from multiple sources through processing and transformation to analytical reporting.

You’ll work with technologies and concepts used in modern data engineering, including:

• Python and SQL for working with data and building pipelines.
• Docker for setting up a consistent development environment.
• REST APIs for retrieving data from external services.
• Amazon S3 event files for exploring data lake ingestion.
• dbt for data transformation, modelling, and testing.
• Apache Airflow for scheduling and orchestrating workflows.
• GitHub Actions for exploring automated testing and CI/CD.

By the end of the course, you’ll understand how to build, test, orchestrate, troubleshoot, and present an end-to-end data engineering pipeline.

Why Learn Modern Data Engineering?

Data is essential to modern organisations.

Online retailers, financial institutions, healthcare providers, technology companies, and many other businesses rely on data pipelines to collect, process, and organise information.

But raw data rarely arrives in a perfect format.

It may come from databases, REST APIs, event files, or other systems. It can contain duplicates, missing values, inconsistent structures, and unexpected changes.

Data engineers help solve these challenges by designing pipelines that move data reliably from source systems into useful analytical datasets.

Modern data engineering involves much more than writing SQL queries or moving files.

It also requires understanding:

• Incremental data ingestion
• Data transformation and modelling
• Data quality and validation
• Workflow orchestration
• Pipeline performance
• Automated testing and deployment
• Monitoring, troubleshooting, and recovery

This course brings these concepts together through practical demonstrations and a connected data engineering project.

What You’ll Learn

Throughout this bootcamp, you’ll learn how to:

• Understand the architecture of an end-to-end data engineering platform.
• Set up a hands-on data engineering environment using Docker Desktop.
• Explore the FreshCart Data Platform and its components.
• Implement incremental database ingestion using watermarks and upserts.
• Understand data contracts and their importance in reliable pipelines.
• Ingest data from REST APIs using authentication tokens and pagination.
• Handle API rate limits, retries, and ingestion failures.
• Work with S3 event files and understand schema drift.
• Create and organise dbt sources, models, and dependencies.
• Understand data lineage and how transformations connect.
• Design star schemas for analytical reporting.
• Explore Slowly Changing Dimensions Type 2 (SCD2).
• Apply dbt tests and source freshness checks.
• Orchestrate data pipelines using Apache Airflow.
• Understand pipeline retries, alerts, and recovery.
• Work with incremental models and performance optimisation.
• Explore CI/CD using linting, unit tests, and GitHub Actions.
• Investigate a simulated production schema-change incident.
• Complete a final dashboard and portfolio capstone project.

The emphasis is on understanding how real data pipelines are designed, operated, tested, and maintained.

Build the FreshCart Data Platform

One of the strongest features of this course is its connected, end-to-end project.

Instead of completing unrelated exercises, you’ll explore the development of the FreshCart Data Platform, a practical environment that brings together multiple data engineering concepts.

The project provides a realistic context for understanding how data engineers work with source systems, data ingestion, transformations, analytical models, and production workflows.

You’ll follow the data engineering lifecycle from initial setup through to the final dashboard and portfolio deliverable.

Part 1: Ingest Data from Multiple Sources

Every data pipeline begins with data ingestion.

In this part of the course, you’ll explore three important ingestion patterns.

Incremental Database Ingestion

Learn how data engineers avoid repeatedly processing entire datasets.

You’ll explore:

• Watermarks for tracking ingestion progress.
• Upserts for inserting and updating records.
• Data contracts for defining expectations between systems.

REST API Ingestion

Discover how applications and services expose data through APIs.

You’ll work with concepts including:

• Authentication tokens
• Pagination
• Rate limits
• Retry strategies

These techniques are important when building pipelines that depend on external data services.

Data Lake Ingestion

Explore ingestion from S3 event files and understand how data lakes support modern data engineering.

You’ll also investigate schema drift, a common challenge that occurs when incoming data structures change unexpectedly.

Part 2: Transform Data with dbt and SQL

Collecting data is only the beginning.

Before data can support reliable analytics, it often needs to be transformed, organised, and validated.

You’ll explore dbt (data build tool) and learn how it supports structured data transformation workflows.

Topics include:

dbt Fundamentals: Understand sources, models, dependencies, and data lineage.

Dimensional Modelling: Explore star schemas and how they organise data for analytical reporting.

Slowly Changing Dimensions: Learn the purpose of SCD Type 2 and how historical changes can be represented.

Data Quality: Explore dbt tests and source freshness checks to help identify data problems.

These skills provide a foundation for developing trustworthy analytical datasets.

Part 3: Orchestrate Pipelines with Apache Airflow

A data pipeline becomes more useful when its tasks can run in a coordinated and repeatable way.

That’s where Apache Airflow comes in.

You’ll explore how Airflow supports workflow orchestration and learn important concepts related to:

• Scheduling and coordinating pipeline tasks.
• Managing retries when tasks fail.
• Understanding alerts and failure handling.
• Recovering from interrupted workflows.

You’ll also examine incremental models and performance considerations that help make data processing more efficient.

Part 4: Explore CI/CD for Data Engineering

Modern data engineering teams need ways to validate changes before introducing them into important workflows.

You’ll explore how Continuous Integration and Continuous Delivery (CI/CD) practices apply to data pipelines.

The course introduces:

• Code linting
• Unit testing
• Automated validation
• GitHub Actions

You’ll see why testing and automation are valuable for improving the reliability and maintainability of data engineering projects.

Part 5: Investigate a Production-Style Pipeline Failure

What happens when a data pipeline breaks at 2 a.m.?

This is one of the most valuable practical scenarios in the course.

In the lesson “Incident: The 2 a.m. Schema Change,” you’ll investigate a simulated production problem involving an unexpected schema change.

This scenario introduces an important aspect of data engineering that many introductory courses overlook:

Building a pipeline is only part of the job. Understanding how to investigate and recover from failures is equally important.

You’ll apply your knowledge of pipeline operations, data quality, and recovery to understand how such incidents affect data workflows.

Final Capstone: Dashboard and Data Engineering Portfolio

At the end of the course, you’ll bring together the concepts you’ve explored throughout the FreshCart project.

The final capstone focuses on the dashboard and your data engineering portfolio.

This provides an opportunity to demonstrate your understanding of the end-to-end data pipeline and communicate the work you’ve completed.

Rather than finishing with theoretical knowledge alone, you’ll have a practical project that can support your continued learning and portfolio development.

Who Is This Course For?

This course is ideal for:

Aspiring Data Engineers: Learners who want to understand how modern data pipelines are built and operated.

Python and SQL Developers: Programmers interested in expanding their skills into data ingestion, transformation, and orchestration.

Data Analysts: Professionals who want to move beyond querying datasets and understand how analytical data is prepared.

Aspiring Analytics Engineers: Learners interested in dbt, dimensional modelling, data quality, and reliable transformation workflows.

Software Developers: Individuals who want to explore data engineering tools and production-style pipeline development.

IT Professionals and Career Changers: Motivated learners exploring practical skills for data engineering and related technical careers.

Computer Science and IT Students: Students who want hands-on exposure to modern data engineering technologies.

Project-Based Learners: Anyone who prefers learning through a connected practical project rather than isolated theoretical lessons.

What Do You Need to Get Started?

You don’t need previous professional data engineering experience to begin.

However, some basic familiarity with Python, SQL, and general programming concepts will make the practical lessons easier to follow.

You’ll need:

• A computer capable of running Docker Desktop.
• An internet connection for downloading the required tools and resources.
• Basic computer and file-management skills.
• A willingness to practise and troubleshoot technical problems.

The course uses accessible development tools and a local Docker-based lab environment.

Why Choose This Data Engineering Bootcamp?

End-to-End Project: Explore the FreshCart Data Platform from ingestion to analytical reporting.

Modern Technology Stack: Work with Python, SQL, dbt, Apache Airflow, Docker, REST APIs, S3 concepts, and GitHub Actions.

Practical Ingestion Patterns: Learn about incremental database ingestion, API pagination, retries, and schema drift.

Data Modelling and Quality: Explore star schemas, SCD2, dbt tests, and source freshness.

Production-Style Engineering: Understand orchestration, performance, CI/CD, and pipeline recovery.

Realistic Incident Simulation: Investigate an unexpected schema change and its impact on a data pipeline.

Portfolio-Focused Capstone: Bring your work together through a final dashboard and project deliverable.

Structured Learning Journey: Progress logically from data ingestion through modelling, orchestration, testing, and production operations.

Start Building Real-World Data Pipelines Today!

Data engineering is not simply about moving data from one place to another.

It’s about creating reliable systems that collect, transform, validate, and deliver useful information.

In this course, you’ll explore how modern data engineering technologies work together through a practical, end-to-end project.

You’ll learn why incremental ingestion matters, how dbt supports analytical modelling, how Apache Airflow coordinates workflows, and how testing and recovery contribute to dependable pipelines.

Whether you’re preparing to explore a data engineering career, expanding your Python and SQL skills, or developing your understanding of modern data platforms, this course provides a practical learning experience.

Don’t just learn individual data engineering tools. Learn how they work together to build reliable data pipelines.

Enrol today and start building your data engineering skills with Python, SQL, dbt, and Apache Airflow!