Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Outcome analysis: detect obsolete exceptions #111

Open
gilles-peskine-arm opened this issue Jan 2, 2025 · 0 comments
Open

Outcome analysis: detect obsolete exceptions #111

gilles-peskine-arm opened this issue Jan 2, 2025 · 0 comments
Labels
enhancement New feature or request size-s Estimated task size: small (~2d)

Comments

@gilles-peskine-arm
Copy link
Contributor

In outcome_analysis.py (possibly with an impact on analyze_outcomes.py in consuming repositories), add a way to detect obsolete exceptions to coverage checks.

Note that this is different from the existing mechanism that causes a failure if a test case is expected to be skipped, but actually passes. The goal here is to detect exceptions that do not match any test case, presumably because the corresponding test case has disappeared.

This may be costly, and is of low importance, so it can be a special command line option that we run manually, rather than something done at every invocation.

Example: Mbed-TLS/mbedtls@b62279d removed some exceptions that were no longer necessary because we removed (stopped generating) some test cases that were never executed. The goal of this issue is to have a way to detect those exceptions.

@gilles-peskine-arm gilles-peskine-arm added enhancement New feature or request size-s Estimated task size: small (~2d) labels Jan 2, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request size-s Estimated task size: small (~2d)
Projects
Status: No status
Development

No branches or pull requests

1 participant