-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Adding python fuzz script that runs rust tests in a loop and logs to …
…rollbar if it fails (#9) # Resolved Issues Using python, we run individual fuzz tests in a loop, and log any failed tests to rollbar. # Review Checklists Please check each item **before approving** the pull request. While going through the checklist, it is recommended to leave comments on items that are referenced in the checklist to make sure that they are reviewed. - [ ] **Testing** - [ ] Are there new or updated unit or integration tests? - [ ] Do the tests cover the happy paths? - [ ] Do the tests cover the unhappy paths? - [ ] Are there an adequate number of fuzz tests to ensure that we are covering the full input space? - [ ] If matching Solidity behavior, are there differential fuzz tests that ensure that Rust matches Solidity?
- Loading branch information
Sheng Lundquist
authored
Apr 30, 2024
1 parent
41db032
commit 9fc5529
Showing
3 changed files
with
48 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,10 @@ | ||
This directory sets up running rust fuzz tests. We use python as the orchestration language | ||
to (1) gather fuzz testing for rust, (2) run individual tests, and (3) log any errors from tests to | ||
rollbar. | ||
|
||
From the `hyperdrive-rs` root directory: | ||
|
||
``` | ||
pip install -r fuzz/requirements-fuzz.txt | ||
ROLLBAR_API_KEY=<rollbar_api_key> python fuzz/run-rust-fuzz.py | ||
``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
rollbar |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,37 @@ | ||
import os | ||
import subprocess | ||
|
||
import rollbar | ||
|
||
# Get rollbar api token from env variables | ||
token = os.environ['ROLLBAR_API_KEY'] | ||
|
||
# Set up rollbar | ||
rollbar.init( | ||
access_token=token, | ||
environment='rust-fuzz', | ||
) | ||
|
||
# Gather all tests using cargo test | ||
tests = subprocess.run(["cargo", "test", "fuzz_", "--", "--list"], stdout=subprocess.PIPE, check=True) | ||
tests = tests.stdout.decode("utf-8").split("\n") | ||
|
||
# Remove the final : test from tests | ||
# The cargo test list also has some printouts that are not tests, so we remove those as well | ||
tests = [o.split(": test")[0] for o in tests if "::" in o] | ||
|
||
# Run forever | ||
while True: | ||
# Loop through tests and run, while capturing failure output | ||
for test in tests: | ||
print(test) | ||
# We don't throw exception if underlying test fails | ||
output = subprocess.run(["cargo", "test", test], stdout=subprocess.PIPE, check=False) | ||
# If the test failed | ||
if output.returncode != 0: | ||
print(f"{test} failed") | ||
str_output = output.stdout.decode("utf-8") | ||
# Log to rollbar | ||
rollbar.report_message(f'Rust test {test} failed', 'error', extra_data={'output': str_output}) | ||
|
||
|