Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set up / fix alerting for CI pipelines #9305

Closed
2 of 3 tasks
ScottTodd opened this issue Jun 2, 2022 · 7 comments
Closed
2 of 3 tasks

Set up / fix alerting for CI pipelines #9305

ScottTodd opened this issue Jun 2, 2022 · 7 comments
Assignees
@ScottTodd
Labels
bug 🐞 Something isn't working infrastructure Relating to build systems, CI, or testing
Milestone
Infrastructure Fixit

Comments

@ScottTodd
Copy link
Member

ScottTodd commented Jun 2, 2022 •
edited
Loading

https://github.com/google/iree/blob/main/.github/workflows/build_package.yml or https://github.com/google/iree/blob/main/.github/workflows/validate_and_publish_release.yml should alert on failure

this does not appear to be working:
https://github.com/google/iree/blob/427a94a09be70631c5d9f89d12616bb7f1954257/build_tools/buildkite/cmake/build_configurations.yml#L69-L72

Set up / fix alerting:

  • Set up / fix alerting for Buildkite based pipelines
  • Set up / fix alerting for GitHub actions based pipelines
    • iree-samples (baseline notification for failure)
@ScottTodd ScottTodd added bug 🐞 Something isn't working infrastructure Relating to build systems, CI, or testing labels Jun 2, 2022
@ScottTodd ScottTodd added this to the Infrastructure Fixit milestone Jun 2, 2022
@jpienaar
Copy link
Member

jpienaar commented Jun 3, 2022

Should we split this in two for buildkite and GitHub actions bases CI?

@ScottTodd
Copy link
Member Author

Should we split this in two for buildkite and GitHub actions bases CI?

We could. I'd prefer just using a task list on one issue to keep it more centralized.

@GMNGeoffrey GMNGeoffrey added this to IREE Jun 28, 2022
@github-project-automation github-project-automation bot moved this to Not Started in (Deprecated) IREE Feb 21, 2023
@allieculp allieculp moved this from Not Started to Inbox in (Deprecated) IREE Apr 5, 2023
@allieculp allieculp moved this from Inbox to Not Started in (Deprecated) IREE Apr 10, 2023
@allieculp allieculp moved this from Not Started to Backlog in (Deprecated) IREE May 16, 2023
@ScottTodd
Copy link
Member Author

Discord alerts have been working reasonably well for ci.yml.

Could copy this code into build_package.yml: https://github.com/openxla/iree/blob/6f2c98f4fa482f6172cb5129fd45135592772b9e/.github/workflows/ci.yml#L1208-L1214

@ScottTodd
Copy link
Member Author

Eh, could do more but I'll call this obsolete for now. We get alerts for ci.yml and benchmark.yml failing, and that's usually enough. Could also add similar alerting for build_package and validate_and_publish_release though.

@github-project-automation github-project-automation bot moved this from Backlog to Done in (Deprecated) IREE Jul 11, 2024
@ScottTodd ScottTodd reopened this Dec 10, 2024
@ScottTodd
Copy link
Member Author

Alerts were enough when ci.yml was the main workflow, but we've switched jobs to a mix of ci.yml, pkgci.yml, and other workflows.

@ScottTodd ScottTodd self-assigned this Dec 10, 2024
This was referenced Dec 10, 2024
Closed
Merged
Merged
Merged
@ScottTodd
Copy link
Member Author

Posted three PRs to add more alerting:

ScottTodd added a commit that referenced this issue Dec 11, 2024
@ScottTodd
Share workflow failure alerting between ci.yml and pkgci.yml. (#19444)
e6266f7 
Progress on #9305.

Changes included:

1. Extracts the code for parsing the results of multiple jobs and
optionally posting alerts to Discord from `ci.yml` into a new [reusable
workflow](https://docs.github.com/en/actions/sharing-automations/reusing-workflows)
in the `workflow_summary.yml` file.
2. Uses the new reusable workflow in `pkgci.yml`.
3. Renames to `summary` step to `ci_summary` and `pkgci_summary` to
disambiguate "required checks". You'd think GitHub would use keys that
aren't ambiguous for required checks but nooooope:

![image](https://github.com/user-attachments/assets/5c3665bc-7933-41d0-8d5a-c6ecf966a3b4)
ScottTodd added a commit that referenced this issue Dec 11, 2024
@ScottTodd
Post to Discord on failures in more workflows. (#19446)
274977c 
Progress on #9305.

See also #19445, which triggers the
presubmit workflows (`linux_x64_bazel`, `linux_x64_clang`,
`linux_x64_clang_asan`) from `ci.yml`, removing the need to add alerting
code to those files. If that PR lands, the changes in those workflows
can be merged with these other changes.
@ScottTodd
Copy link
Member Author

Added alerting to most workflows. Calling this fixed [again].

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ScottTodd ScottTodd

Labels
bug 🐞 Something isn't working infrastructure Relating to build systems, CI, or testing
Projects
No open projects
IREE
Status: No status
Milestone
Infrastructure Fixit
Development

No branches or pull requests
3 participants
@jpienaar @ScottTodd @allieculp