TroveGrid
Back to all tools
Dagster logo

Dagster

Software-defined assets — observable, testable data pipelines with full lineage.

0Open Source
Workflow Orchestration
Official Website View on GitHub
Share:XLinkedInWhatsApp

Overview

Dagster is a cloud-native data orchestrator for building, testing, and monitoring data pipelines as software-defined assets, giving engineers and data teams full visibility into data lineage, freshness, and partitioning.

Key Features

  • Software-defined assets (SDA) for asset-centric pipeline modeling
  • Asset lineage graph with freshness tracking and metadata
  • Partitioned assets for incremental processing by date or key range
  • Built-in sensor and schedule framework for event-driven pipelines
  • Dagster Cloud for fully managed orchestration with branching deployments
  • Integrations with dbt, Spark, Airbyte, Fivetran, Snowflake, and more

Real-World Workflows

Model data pipeline as software-defined assets

  1. 1Define raw, staging, and transformed tables as @asset decorated functions
  2. 2Dagster auto-infers dependencies from function arguments
  3. 3View the full asset lineage in the Dagster UI
  4. 4Materialize upstream assets on demand or on schedule

Integrate dbt models as Dagster assets

  1. 1Install dagster-dbt and load the dbt project into Dagster
  2. 2dbt models appear as Dagster assets with lineage from source tables
  3. 3Run dbt tests as asset checks in the Dagster pipeline
  4. 4Chain Spark ingestion or Airbyte syncs upstream of dbt models

Getting Started

# Install Dagster
pip install dagster dagster-webserver

# Create a new project
dagster project scaffold --name my_project
cd my_project

# Start the Dagster UI
dagster dev

Compare Alternatives

See how Dagster stacks up against similar tools.

VSairflowVSprefectVSmageVSkestra