Apache Airflow Course Content
Module 1: Course Introduction
- Important Prerequisites
- Course Objectives
- Who am I?
- Development Environment
Module 2: Getting Started with Airflow
- Why Airflow?
- What is Airflow?
- How Airflow works?
- Installing Airflow 2.0
- The CLI in Airflow 2.0
- What you need to know about the UI
Module 3: Coding Your First Data Pipeline with Airflow
- What is DAG?
- Time to code your first DAG
- DAG Skeleton
- What is an Operator?
- Creating Table
- The secret weapon!
- The Providers
- Is API available?
- Extracting users
- Processing users
- Storing users
- Order matters!
- You data pipeline in action!
- Dag scheduling
- Backfilling and catchup
Module 4: Databases and Executors
- The default configuration
- Start scaling with the Local Executor
- Scale to the infinity with the Celery Executor
- Scaling Airflow with the Celery Executor in action!
- Changing the executor
- Concurrency, the parameters you must know!
- Concurrency in practice
Module 5: Implementing Advanced Concepts in Airflow
- Adios repetitive patterns
- Minimising DAGs with SubDAGs
- Adios SubDAGs, Welcome TaskGroups!
- Sharing data between tasks with XComs
- XComs in action!
- Choosing a specific path in your DAG
- Executing a task according to a condition
- Trigger rules or how tasks get triggered
- Changing the way your tasks are triggered
Module 6: Creating Airflow Plugins with Elasticsearch and PostgreSQL
- Installing Elasticsearch
- How the plugin system works?
- Creating a hook interacting with Elasticsearch
- Creating the PostgresToElasticOperator
Module 7: Using Apache Airflow with Docker
- Quick Reminder About Docker
- Running Airflow on Docker with the Celery Executor
- Running Airflow on Docker with the Local Executor
Module 8: Airflow 2.0
- What to expect from Airflow 2.0?