Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix deserializing callbacks to proper types #45305

Merged
merged 1 commit into from
Jan 7, 2025

Conversation

mobuchowski
Copy link
Contributor

Without this fix, the callback requests are deserialized as a base class CallbackRequest and are not executed - they are silently failing through this check.

def _execute_callbacks(
    dagbag: DagBag, callback_requests: list[CallbackRequest], log: FilteringBoundLogger
) -> None:
    for request in callback_requests:
        log.debug("Processing Callback Request", request=request.to_json())
        if isinstance(request, TaskCallbackRequest):
            raise NotImplementedError(
                "Haven't coded Task callback yet - https://github.com/apache/airflow/issues/44354!"
            )
            # _execute_task_callbacks(dagbag, request)
        elif isinstance(request, DagCallbackRequest):
            _execute_dag_callbacks(dagbag, request, log)

@boring-cyborg boring-cyborg bot added the area:Scheduler including HA (high availability) scheduler label Dec 30, 2024
@mobuchowski mobuchowski marked this pull request as ready for review December 30, 2024 21:31
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Callbacks haven't yet been ported to TASK SDK @mobuchowski
Is this an attempt to do that?

@mobuchowski
Copy link
Contributor Author

@amoghrajesh dag ones (DagCallbackRequest) should work, and are mostly implemented in some form. This PR makes basic ones work. There are missing things - like the context is empty dict - but neverthless is improving the situation.

My main motivation is unblocking ticket #44547 that has been really hard to get to get merged.

@mobuchowski mobuchowski force-pushed the tasksdk-deserialize-callbacks branch 2 times, most recently from 879a9aa to 7b9eb4a Compare December 31, 2024 09:18
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

few nits

@mobuchowski mobuchowski force-pushed the tasksdk-deserialize-callbacks branch from 7b9eb4a to 0cbdab7 Compare January 7, 2025 14:04
@mobuchowski mobuchowski force-pushed the tasksdk-deserialize-callbacks branch from 0cbdab7 to 7734b3e Compare January 7, 2025 15:14
@mobuchowski mobuchowski merged commit 2cf9eb4 into apache:main Jan 7, 2025
45 checks passed
@jscheffl
Copy link
Contributor

jscheffl commented Jan 7, 2025

I assume this PR broke main, failed tests are with https://github.com/apache/airflow/actions/runs/12657999447

Someone able to fix this or shall we revert to fix canary again?

@kaxil
Copy link
Member

kaxil commented Jan 8, 2025

I assume this PR broke main, failed tests are with https://github.com/apache/airflow/actions/runs/12657999447

Someone able to fix this or shall we revert to fix canary again?

Fixed in #45482

agupta01 pushed a commit to agupta01/airflow that referenced this pull request Jan 13, 2025
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:Scheduler including HA (high availability) scheduler
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants