HA Scheduler does not respect max_active_runs in edge case

[wolfier]

Apache Airflow version

2.10.4

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Two queued dagruns of a DAG with max_active_runs of 1 started within 0.2 seconds of each other.

The deployment has two schedulers, A and B. I suspect scheduler A started one dagrun and scheduler B started the other dagrun. Because a scheduler queries the active dagrun information every scheduling loop via _start_queued_dagruns, it is possible for the limit to be exceeded as the information is not shared between schedulers. Both schedulers thought there were no active dagruns and started their respective queued dagrun.

The question is how does one dagrun end up in one query and not the other. One explanation could be scheduler A goes out and locks only 1 dagrun row because the other dagrun row is out of the max_dagruns_per_loop_to_schedule range. The other scheduler then picked up the dagrun that did not get queried. Even though it is very unlikely, I suspect both scheduling loops ran the query very closely in time.

What you think should happen instead?

No response

How to reproduce

This scenario requires extreme luck (or lack thereof) so I have not been able to reproduce this behaviour.

Perhaps the key to reproduce this is with max_dagruns_per_loop_to_schedule set to 1.

Operating System

n/a

Versions of Apache Airflow Providers

No response

Deployment

Astronomer

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

Yes. That is quite possible edge case though rather infrequent. I am not sure if there is a way we could protect it - we would likely have to add lock on dag not only on DagRun, but it would likely heavily decrease some parallelism scenarios for scheduling. @ashb wdyt ?

[jedcunningham]

As I was chatting with Alan about this yesterday, another thought I had was to do optimistic locking. e.g. set the state to running only if the count of running for that dag matches what we expect (and used to determine it's okay to start another!).

[potiuk]

Can we do it ? How?

I think this is the "logical" problem to solve.

Whatever optimisitc query you can run is "read" and in order to make it really protected, you need to make it part of a transaction - you need to query and update the state in the same transaction essentially.

Or you have to read a "state" of the system and use it in the "write" statement to detect that the state has changed. You cannot really separate "read" and "update" action - they should be connected by either transaction or state retrieved from the "read" action has to be used as a "potentially failing" check in the update state.

[potiuk]

Generally:

a) pessimistic locking

• lock update
• run query and update state
• save the state change and unlocki

b) optimistic locking

• read a state
• run query and only update if the state has not changed in a single transaction and commit in the same operation (and fail otherwise)

I am not sure what "state" we can read here and how we can verify the state has not changed since we read it.