Add some flaky tests #56
Conversation
These tests failed in https://automation-dashboard.vercel.app/cycle/12422671923_1-44d9567-pr-onprem-ent, but they passed when I ran them locally
The followup CI run was https://automation-dashboard.vercel.app/cycle/12641178131_1-44d9567-pr-onprem-ent which also had a different test fail
|
@hmhealey is attempting to deploy a commit to the Saturnino Abril's projects Team on Vercel. A member of the Team first needs to authorize it. |
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
Thanks @hmhealey for taking the time for this PR. I'll put this on hold for now since "known issue" submission is no longer in use in favor of auto-detecting flaky tests based on past results (from nightly builds). Likely, the "known issue" will be reserved to those that were product/system/infra bug with corresponding Jira ticket. I know there's an instruction to submit a PR here and it's something I'm going to update too. I'll take a look this week. |
|
@saturninoabril Is there still a way with that process to manually add flaky tests that aren't caught on nightly builds? My worry with just detecting them from nightly builds is that they don't run as often as PR builds which means they're less likely to encounter flaky behaviour |
These tests aren't related to mattermost/mattermost#29224, but they all were flaky when running the tests for it.
The first set of tests failed on CI in https://automation-dashboard.vercel.app/cycle/12422671923_1-44d9567-pr-onprem-ent, but they passed when I ran them locally. I ran the tests again to verify that in https://automation-dashboard.vercel.app/cycle/12641178131_1-44d9567-pr-onprem-ent, and then a different test failed, so I added that as well