Airflow vs Dagster — Tasks vs Assets

Two philosophies of 'what is a pipeline?'

Airflow (and most older schedulers) models a pipeline as a DAG of tasks. A task either succeeded or failed. The orchestrator does not know what data the task produced.

Dagster (and newer systems like Prefect 2, Mage, Kestra) models the pipeline as a graph of assets — the tables, files, ML models the business cares about. The orchestrator knows the lineage between them and can answer 'which downstream assets are stale because the upstream was rerun?'.

Neither is universally better. Airflow wins on ecosystem and ops familiarity. Dagster wins when data lineage and selective rebuilds matter — which they almost always do in a mature platform.

Airflow thinks like a to-do list: 'wash car, mow lawn, take out bins' — it ticks each chore done or failed, but has no idea the car is now clean. Dagster thinks like a smart home inventory: it tracks the things — 'clean car', 'mown lawn', 'empty bin' — so when it rains it knows the 'clean car' asset is now stale and which chores to redo. Task-thinking asks 'did the step run?'; asset-thinking asks 'is the data fresh, and what downstream went stale?' — the question you actually wake up at 3am needing answered.

Same pipeline, two mental models

Airflow tracks the verbs (extract, transform, load). Dagster tracks the nouns (the assets each step produces). The asset view answers 'what is stale?' for free.