Apache Airflow 2 to Airflow 3
Cohash upgrades self-managed Airflow 2 deployments to Airflow 3. We bring the cluster to the latest 2.x release, lint DAGs with Ruff's AIR rules, and fix what Airflow 3 removed: SubDAGs, direct metadata database access, old context keys and the v1 REST API. DAGs then move in batches to a parallel Airflow 3 environment, with scheduling behavior checked so no job runs twice or gets skipped.
- 01Get to the latest 2.x
- 02Lint and fix DAGs
- 03Remove direct database access
- 04Config, auth and plugins
- 05Parallel environment
- 06Verify scheduling and SSO
The story
Told without the client's name or anything that could identify them.
A retailer's data engineering team was still running self-managed Airflow 2 after its end of life on 22 April 2026. Custom operators opened SQLAlchemy sessions against the metadata database. Old DAGs still used SubDAGs and execution_date, and CI pipelines triggered runs through /api/v1. We got the cluster onto the latest 2.x release and ran Ruff's AIR rules, which turned the work into a list. Custom code moved onto the Airflow Python client with API tokens. DAGs then moved in batches to a parallel Airflow 3 environment. For each batch we paused the schedules on the old side and checked the catchup and cron timetable changes. Once the last batch checked out, the Airflow 2 environment was retired.
why it was urgentAirflow 2 is out of support. It went into limited maintenance on 22 October 2025 and reached end of life on 22 April 2026, so it gets no fixes now. Airflow 3 has been out since 22 April 2025, and the current patch is 3.3.2. If you're still below 2.7, plan an extra hop, because the official upgrade path starts there.
- openYour scheduler still runs Airflow 2.x on infrastructure you manage yourself.
- recurringSomebody's custom operators open SQLAlchemy sessions straight against the Airflow metadata database.
- blockedOld DAGs still use SubDAGs or read execution_date from the task context.
- openOther systems trigger DAGs through the /api/v1 REST endpoints.
The estate, before and after
Before: self-managed Airflow 2 past end of life
The Airflow 2 webserver and scheduler shared one metadata database. Custom operators wrote to it directly too. External tools used the v1 REST API to trigger DAGs, and that API is gone in Airflow 3.
Read the diagram as text
- CI pipelines → Airflow 2 webserver: /api/v1 trigger
- Monitoring tools → Airflow 2 webserver: /api/v1 status
- Airflow 2 webserver → Metadata database: reads
- Airflow 2 scheduler → Metadata database: scheduling
- Custom operators → Metadata database: direct writes
- Airflow 2 scheduler → Custom operators: run tasks
- Custom operators → Data platforms: load jobs
After: Airflow 3 with API-only access to metadata
Task code and external callers both go through the Airflow 3 api-server these days, and only Airflow's own services touch the metadata database. The dag-processor runs as its own service.
Read the diagram as text
- CI pipelines → Airflow 3 api-server: /api/v2 trigger
- Monitoring tools → Airflow 3 api-server: /api/v2 status
- Custom operators → Airflow 3 api-server: python client
- Airflow 3 api-server → Metadata database: reads · writes
- Scheduler → Metadata database: scheduling
- dag-processor → Metadata database: serialized DAGs
- Scheduler → Custom operators: run tasks
- Custom operators → Data platforms: load jobs
How we run it, phase by phase.
6 phases take you from Apache Airflow 2.x to Apache Airflow 3. Each one ends with something you can check before the next starts. As you scroll, the blocks show the estate moving across.
- phase 01
Get to the latest 2.x
Get onto the newest 2.x release before anything else. Install apache-airflow-providers-standard and run airflow db clean so the history tables are smaller when you migrate. When airflow dags reserialize finishes without errors, the code work can start.
- phase 02
Lint and fix DAGs
Run Ruff with the AIR301 and AIR302 rules and take the safe fixes it offers. The rest is manual. SubDAGs turn into TaskGroups or Assets, execution_date and its relatives become logical_date, SLAs become Deadline Alerts, and imports move to the airflow.sdk namespace.
- phase 03
Remove direct database access
Airflow 3 won't let task code open metadata database sessions. We move custom operators and callbacks onto the Airflow Python client, authenticated with API tokens, and point anything still calling /api/v1 at /api/v2.
- phase 04
Config, auth and plugins
Start with airflow config update. Next, choose an auth manager (Simple Auth or FAB) and update OAuth redirect URLs for the new /auth prefix. Flask-AppBuilder plugins either get ported to FastAPI or kept alive through the FAB compatibility layer.
- phase 05
Parallel environment
Back up the metadata database and run airflow db migrate against a copy, never the live one. The api-server and the dag-processor now run as separate services. DAGs move over in batches, and each batch's schedules are paused on the Airflow 2 side as soon as it goes live on 3.
- phase 06
Verify scheduling and SSO
After each batch we look for DAGs hit by the new catchup default or the cron timetable change, and we check that SSO still signs people in. When the last batch checks out, the Airflow 2 environment gets switched off.
What the client got
What changed for the client once the work was done. Described, not scored: we don't publish client figures.
- result 01
The scheduler runs a supported Airflow 3 release
We got production DAGs onto Airflow 3. The api-server and dag-processor run as separate services, and they're defined as code with the auth and plugin configuration.
- result 02
Task code no longer touches the metadata database
We moved custom operators and callbacks onto the Airflow Python client with API tokens, as Airflow 3 requires. None of them opens a SQLAlchemy session any more.
- result 03
No job ran twice or got skipped at cut-over
We checked every batch against the catchup_by_default change and the new CronTriggerTimetable default. Once a batch was live on 3, its Airflow 2 schedules got paused.
- result 04
External callers use /api/v2
We tracked down the CI pipelines, other schedulers and monitoring tools that called /api/v1 and repointed each one. There's a list of every caller that moved.
- result 05
DAG code is current and linted
SubDAGs became TaskGroups or Assets, context keys moved to logical_date, SLAs became Deadline Alerts and imports moved to airflow.sdk, all in version control.
Risks we plan for
-
Silent schedule changes
catchup_by_default is now False, and bare cron strings now use CronTriggerTimetable unless create_cron_data_intervals is set. DAGs that depend on data_interval values can run with different dates.
-
Integrations on the old API
The /api/v1 REST API is gone. CI pipelines, other schedulers and monitoring tools that call it break on cut-over unless found first.
-
xcom_pull behavior
xcom_pull without task_ids now reads only from the current task. DAGs that relied on the old behavior pass the wrong values without failing.
-
Managed service timelines
MWAA, Cloud Composer and Astronomer publish their own version support dates. If part of the estate runs on one of them, its dates set the plan for that part.
Where AI helps, and where it needs guardrails
Airflow has no vendor AI features, so any AI help is around the code. Coding agents are good at the mechanical rewrites Ruff flags, like context keys, imports and SubDAG conversions, provided every change is a reviewed pull request with a test run behind it. Once you're upgraded, an agent can read run status through /api/v2 with a read-only token. Triggering or clearing a DAG run still needs a person to approve it.
forward deployed engineers →What you're left with
We write these for your setup, and they're working documents. Your team should be able to keep things running from them long after we've gone.
- inventory01
DAG inventory with owners, run history and Ruff findings
yours to keep - evidence02
Fixed DAGs and custom operators in version control
yours to keep - code03
Airflow 3 environment as code, with auth and plugin configuration
yours to keep - runbook04
Batch cut-over plan with schedule checks per DAG
yours to keep - inventory05
List of external callers moved to /api/v2
yours to keep
Platforms and services on this move
Services involved
Certified engineers, wherever you need them.
Some clients want a project team, some want a couple of engineers on theirs. Either works. Onsite, nearshore or offshore, with handovers across time zones that keep things moving.
Onsite
Engineers at your sites in the US
Nearshore
Overlapping hours with US teams
Offshore
Overnight progress on your backlog
All time zones
Coverage handed over across time zones
01Is staying on Airflow 2 a real risk?
For a self-managed install, yes. Airflow 2 reached end of life on 22 April 2026, so security fixes have stopped. Managed services set their own dates, which we check for your provider.
02Can we upgrade in place?
You can, once the database is backed up. We'd still rather stand up a parallel environment. DAGs move over in batches that way, and if one batch misbehaves you still have a working Airflow 2 to fall back on.
03How much DAG code has to change?
It depends on how many DAGs use removed features and how much custom code touches the metadata database. Ruff's AIR rules give a count early, which turns the estimate into a list.
04Should we move to a managed Airflow service at the same time?
It's possible, but two changes at once make problems harder to trace. We usually upgrade first, then move hosting if the business case holds.
Often on the same roadmap
AI pilot to production with forward deployed engineers
Taking an AI pilot into production with forward deployed engineers who work with the business team, build evals, connect real systems and hand over cleanly.
read the case studyNasdaq Calypso upgrade to V19
Upgrading Nasdaq Calypso to V19, with server consolidation, properties moved to the database, the WebUI and deployment changes tested on your own trades.
read the case studyHand-built cloud to Terraform or OpenTofu
Bringing console-built AWS, Azure and Google Cloud resources under Terraform or OpenTofu with import blocks, clean state and pipelines, without rebuilding.
read the case study109df43e - 01Apache Airflow: Supported versionsairflow.apache.org ↗
- 02Apache Airflow: Upgrading to Airflow 3airflow.apache.org ↗
Planning this move?
A 30-minute working session with a senior engineer. No slides.
- The platform and the version you're on
- Where you need to get to, and by when
- What's custom: rules, plugins, integrations
- Any audit finding or deadline driving it