-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[red-knot] Property test improvements (#15358)
## Summary - Add a workflow to run property tests on a daily basis (based on `daily_fuzz.yaml`) - Mark `assignable_to_is_reflexive` as flaky (related to #14899) - Add new (failing) `intersection_assignable_to_both` test (also related to #14899) ## Test Plan Ran: ```bash export QUICKCHECK_TESTS=100000 while cargo test --release -p red_knot_python_semantic -- \ --ignored types::property_tests::stable; do :; done ``` Observed successful property_tests CI run
- Loading branch information
Showing
2 changed files
with
97 additions
and
6 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,71 @@ | ||
name: Daily property test run | ||
|
||
on: | ||
workflow_dispatch: | ||
schedule: | ||
- cron: "0 12 * * *" | ||
pull_request: | ||
paths: | ||
- ".github/workflows/daily_property_tests.yaml" | ||
|
||
permissions: | ||
contents: read | ||
|
||
concurrency: | ||
group: ${{ github.workflow }}-${{ github.head_ref || github.run_id }} | ||
cancel-in-progress: true | ||
|
||
env: | ||
CARGO_INCREMENTAL: 0 | ||
CARGO_NET_RETRY: 10 | ||
CARGO_TERM_COLOR: always | ||
RUSTUP_MAX_RETRIES: 10 | ||
FORCE_COLOR: 1 | ||
|
||
jobs: | ||
property_tests: | ||
name: Property tests | ||
runs-on: ubuntu-latest | ||
timeout-minutes: 20 | ||
# Don't run the cron job on forks: | ||
if: ${{ github.repository == 'astral-sh/ruff' || github.event_name != 'schedule' }} | ||
steps: | ||
- uses: actions/checkout@v4 | ||
with: | ||
persist-credentials: false | ||
- name: "Install Rust toolchain" | ||
run: rustup show | ||
- name: "Install mold" | ||
uses: rui314/setup-mold@v1 | ||
- uses: Swatinem/rust-cache@v2 | ||
- name: Build Red Knot | ||
# A release build takes longer (2 min vs 1 min), but the property tests run much faster in release | ||
# mode (1.5 min vs 14 min), so the overall time is shorter with a release build. | ||
run: cargo build --locked --release --package red_knot_python_semantic --tests | ||
- name: Run property tests | ||
shell: bash | ||
run: | | ||
export QUICKCHECK_TESTS=100000 | ||
for _ in {1..5}; do | ||
cargo test --locked --release --package red_knot_python_semantic -- --ignored types::property_tests::stable | ||
done | ||
create-issue-on-failure: | ||
name: Create an issue if the daily property test run surfaced any bugs | ||
runs-on: ubuntu-latest | ||
needs: property_tests | ||
if: ${{ github.repository == 'astral-sh/ruff' && always() && github.event_name == 'schedule' && needs.property_tests.result == 'failure' }} | ||
permissions: | ||
issues: write | ||
steps: | ||
- uses: actions/github-script@v7 | ||
with: | ||
github-token: ${{ secrets.GITHUB_TOKEN }} | ||
script: | | ||
await github.rest.issues.create({ | ||
owner: "astral-sh", | ||
repo: "ruff", | ||
title: `Daily property test run failed on ${new Date().toDateString()}`, | ||
body: "Runs listed here: https://github.com/astral-sh/ruff/actions/workflows/daily_property_tests.yaml", | ||
labels: ["bug", "red_knot", "testing"], | ||
}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters