TroveGrid
Back to all tools
Apache Airflow logo

Apache Airflow

Author, schedule, and monitor data workflows as Python DAGs.

0Open Source
Workflow Orchestration
Official Website View on GitHub
Share:XLinkedInWhatsApp

Overview

Apache Airflow is an open-source platform to author, schedule, and monitor workflows as directed acyclic graphs (DAGs) written in Python, widely adopted for orchestrating data pipelines across ETL, ML, and analytics workloads.

Key Features

  • DAGs defined as Python code with rich scheduling expressions (cron, timetables)
  • Rich UI for monitoring task execution, logs, and Gantt charts
  • 250+ provider packages for AWS, GCP, Azure, Spark, dbt, and more
  • Dynamic task mapping for parallel fan-out on variable-size inputs
  • Managed Airflow via Amazon MWAA, Google Cloud Composer, and Astronomer
  • XCom for passing data between tasks with flexible backend storage

Real-World Workflows

Orchestrate a dbt + Spark ETL pipeline

  1. 1Define a DAG with SparkSubmitOperator to run a transformation job
  2. 2Chain downstream dbt tasks using BashOperator or dbt Cloud hook
  3. 3Set inter-task dependencies with >> and << operators
  4. 4Schedule the DAG on a cron, monitor runs in the Airflow UI

ML training pipeline orchestration

  1. 1Create tasks for data ingestion, feature engineering, model training
  2. 2Use SageMakerTrainingOperator or VertexAICreateCustomJobOperator
  3. 3Gate downstream evaluation tasks on training completion
  4. 4Trigger retraining DAGs on drift alerts from monitoring systems

Getting Started

# Install and init
pip install apache-airflow
export AIRFLOW_HOME=~/airflow
airflow db init

# Start services
airflow webserver -p 8080 &
airflow scheduler &

# Open UI at http://localhost:8080

Compare Alternatives

See how Apache Airflow stacks up against similar tools.

VSdagsterVSprefectVSmageVSkestra