Replace external_trigger check with DagRunType
#45961
Conversation
boring-cyborg
bot
added
area:API
jason810496
force-pushed
the
replace-external_trigger-with-dag-run-type
branch
3 times, most recently
from
45cf835 to
9eef4f1
Compare
jason810496
changed the title
external_trigger check with DagRunTypeexternal_trigger check with DagRunType
jason810496
requested review from
ephraimbuddy,
potiuk,
ashb,
jedcunningham,
mobuchowski,
bbovenzi,
pierrejeambrun,
ryanahamilton,
jscheffl and
XD-DENG
as code owners
|
|
||
| print(f"Dag information:{dag_id} Run id: {run_id} external trigger: {external_trigger}") | ||
| print(f"Dag information:{dag_id} Run id: {run_id}") |
There was a problem hiding this comment.
Not that it matters, but lets include the run_type here in its place.
|
|
||
| def upgrade(): | ||
| """Apply remove external_trigger field.""" | ||
| # ### commands auto generated by Alembic - please adjust! ### |
There was a problem hiding this comment.
| # ### commands auto generated by Alembic - please adjust! ### |
|
|
||
| def downgrade(): | ||
| """Unapply remove external_trigger field.""" | ||
| # ### commands auto generated by Alembic - please adjust! ### |
There was a problem hiding this comment.
| # ### commands auto generated by Alembic - please adjust! ### |
|
|
||
| # ### end Alembic commands ### |
There was a problem hiding this comment.
| # ### end Alembic commands ### |
airflow/models/dagrun.py
Outdated
| @@ -1227,11 +1217,11 @@ def _emit_true_scheduling_delay_stats_for_finished_state(self, finished_tis: lis | |||
| rid of the outliers on the stats side through dashboards tooling. | |||
|
|
|||
| Note that the stat will only be emitted for scheduler-triggered DAG runs | |||
| (i.e. when ``external_trigger`` is *False* and ``clear_number`` is equal to 0). | |||
| (i.e. when ``run_type`` is *MANUAL* and ``clear_number`` is equal to 0). | |||
There was a problem hiding this comment.
| (i.e. when ``run_type`` is *MANUAL* and ``clear_number`` is equal to 0). | |
| (i.e. when ``run_type`` is not *MANUAL* and ``clear_number`` is equal to 0). |
Though this brings up another question: How should this function deal with run_type==BACKFILL?
There was a problem hiding this comment.
How about adding an externally_triggered_type in airflow.utils.types? Instead of checking run_type == DagRunType.MANUAL, we could use run_type in externally_triggered_type. WDYT?
externally_triggered_type: frozenset[DagRunType] = frozenset([DagRunType.BACKFILL_JOB, DagRunType.MANUAL, DagRunType.ASSET_TRIGGERED])There was a problem hiding this comment.
I think we need to look at what the intent here was, and how that maps on to the new dag run types. My guess is that when "external_trigger" was added, there was either scheduled, or external.
So Manual is clearly not scheduled, but what abouty Backfill runs? How should we treat those. And for that I delegate to @uranusjr and @dstandish boing flip 😄
providers/src/airflow/providers/standard/operators/latest_only.py
Outdated
Show resolved
Hide resolved
| @@ -53,7 +54,7 @@ def choose_branch(self, context: Context) -> str | Iterable[str]: | |||
| # If the DAG Run is externally triggered, then return without | |||
| # skipping downstream tasks | |||
| dag_run: DagRun = context["dag_run"] # type: ignore[assignment] | |||
| if dag_run.external_trigger: | |||
| if dag_run.run_type == DagRunType.MANUAL: | |||
There was a problem hiding this comment.
Ditto here about BACKFILL. Need to decide how to handle this
ashb
added
the
airflow3.0:breaking
jason810496
force-pushed
the
replace-external_trigger-with-dag-run-type
branch
from
a65904d to
2fee34c
Compare
closes: #45932
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.