Skip to content

Commit

Permalink
Adding python fuzz script that runs rust tests in a loop and logs to …
Browse files Browse the repository at this point in the history
…rollbar if it fails (#9)

# Resolved Issues
Using python, we run individual fuzz tests in a loop, and log any failed
tests to rollbar.

# Review Checklists

Please check each item **before approving** the pull request. While
going
through the checklist, it is recommended to leave comments on items that
are
referenced in the checklist to make sure that they are reviewed.

- [ ] **Testing**
    - [ ] Are there new or updated unit or integration tests?
    - [ ] Do the tests cover the happy paths?
    - [ ] Do the tests cover the unhappy paths?
- [ ] Are there an adequate number of fuzz tests to ensure that we are
          covering the full input space?
- [ ] If matching Solidity behavior, are there differential fuzz tests
that
          ensure that Rust matches Solidity?
  • Loading branch information
Sheng Lundquist authored Apr 30, 2024
1 parent 41db032 commit 9fc5529
Show file tree
Hide file tree
Showing 3 changed files with 48 additions and 0 deletions.
10 changes: 10 additions & 0 deletions fuzz/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
This directory sets up running rust fuzz tests. We use python as the orchestration language
to (1) gather fuzz testing for rust, (2) run individual tests, and (3) log any errors from tests to
rollbar.

From the `hyperdrive-rs` root directory:

```
pip install -r fuzz/requirements-fuzz.txt
ROLLBAR_API_KEY=<rollbar_api_key> python fuzz/run-rust-fuzz.py
```
1 change: 1 addition & 0 deletions fuzz/requirements-fuzz.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
rollbar
37 changes: 37 additions & 0 deletions fuzz/run-rust-fuzz.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
import os
import subprocess

import rollbar

# Get rollbar api token from env variables
token = os.environ['ROLLBAR_API_KEY']

# Set up rollbar
rollbar.init(
access_token=token,
environment='rust-fuzz',
)

# Gather all tests using cargo test
tests = subprocess.run(["cargo", "test", "fuzz_", "--", "--list"], stdout=subprocess.PIPE, check=True)
tests = tests.stdout.decode("utf-8").split("\n")

# Remove the final : test from tests
# The cargo test list also has some printouts that are not tests, so we remove those as well
tests = [o.split(": test")[0] for o in tests if "::" in o]

# Run forever
while True:
# Loop through tests and run, while capturing failure output
for test in tests:
print(test)
# We don't throw exception if underlying test fails
output = subprocess.run(["cargo", "test", test], stdout=subprocess.PIPE, check=False)
# If the test failed
if output.returncode != 0:
print(f"{test} failed")
str_output = output.stdout.decode("utf-8")
# Log to rollbar
rollbar.report_message(f'Rust test {test} failed', 'error', extra_data={'output': str_output})


0 comments on commit 9fc5529

Please sign in to comment.