AIP-66: Add support for parsing DAG bundles #45371

jedcunningham · 2025-01-03T05:50:40Z

Let's start parsing DAG bundles! This moves us away from parsing a
single local directory to being able to parse many different bundles,
including optional support for versioning.

This is just the basics - it keeps the parsing loop largely untouched.
We still have a single list of "dag files" to parse, and queue of them.
However, instead of just a path, this list and queue now contain
DagFilePaths, which hold both a local path and the bundle its from.

There are a number of things that are not fully functional at this
stage, like versioned callbacks. These will be refactored later. There
is enough churn with the basics (particularly with the number of test
changes).

airflow/dag_processing/manager.py

airflow/models/dagbundle.py

airflow/dag_processing/manager.py

airflow/dag_processing/collection.py

airflow/models/dag.py

airflow/dag_processing/manager.py

dstandish

Some suggestions

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes).

dstandish

go for it

This reverts commit 72ab1d5.

* Revert "Move `list_py_file_paths` test to the right file (#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <[email protected]>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <[email protected]>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

Let's start parsing DAG bundles! This moves us away from parsing a single local directory to being able to parse many different bundles, including optional support for versioning. This is just the basics - it keeps the parsing loop largely untouched. We still have a single list of "dag files" to parse, and queue of them. However, instead of just a path, this list and queue now contain `DagFilePath`s, which hold both a local path and the bundle its from. There are a number of things that are not fully functional at this stage, like versioned callbacks. These will be refactored later. There is enough churn with the basics (particularly with the number of test changes). Co-authored-by: Daniel Standish <[email protected]>

* Revert "Move `list_py_file_paths` test to the right file (apache#45521)" This reverts commit c29b6e8. * Revert "AIP-66: Add support for parsing DAG bundles (apache#45371)" This reverts commit 72ab1d5.

boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:Scheduler including HA (high availability) scheduler labels Jan 3, 2025

uranusjr reviewed Jan 3, 2025

View reviewed changes

airflow/dag_processing/manager.py Outdated Show resolved Hide resolved

uranusjr reviewed Jan 3, 2025

View reviewed changes

airflow/models/dagbundle.py Outdated Show resolved Hide resolved

jedcunningham force-pushed the bundle-parsing branch 2 times, most recently from 5fa6022 to 5f5b30c Compare January 3, 2025 16:04

jedcunningham mentioned this pull request Jan 4, 2025

Remove some unnecessary FAB test setup code #45400

Merged

ephraimbuddy reviewed Jan 6, 2025

View reviewed changes

airflow/dag_processing/manager.py Outdated Show resolved Hide resolved