Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add observations to 8.7.2 performance report #308

Merged
merged 1 commit into from
Dec 18, 2023
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
42 changes: 38 additions & 4 deletions reports/2023-12-performance-8.7.2.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,17 +17,51 @@ For each version, we're gathering various metrics under 2 different workloads:
1. _value-only_: Each transaction consumes 2 inputs and creates 2 outputs, changing the UTxO set. This workload produces full blocks (> 80kB) exclusively.
2. _Plutus_: Each transaction contains a Plutus script exhausting the per-tx execution budget. This workload produces small blocks (< 3kB) exclusively.

Benchmarking is performed on a cluster of 52 nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs
Benchmarking is performed on a cluster of 52 block producing nodes spread across 3 different AWS regions, interconnected using a static, restricted topology. All runs
were performed in the Babbage era.


## Observations

*to be published calendar week 50*
These benchmarks are about evaluating specific corner cases in a constrained environment that allows for reliable reproduction of results; they're not trying to directly recreate the operational conditions on Mainnet.

## Conclusion
The observations stated refer to the direct comparison between the `8.1.2` and `8.7.2` versions.

*to be published calendar week 50*
### Resource Usage

1. Plutus workload, having a lower overall absolute CPU load, exhibits an average increase of 27% in Process CPU usage. Value workload, having a higher overall absolute CPU load, exhibits a near-jitter increase of 1%.
2. Allocation rates increase by ~8.9MB/s (value workload) and ~12.6MB/s (Plutus workload).
3. Heap sizes increase by 47% - 54%.
4. CPU 85% span duration shrinks by ~9.7 slots under value workload, and ~5.8 slots under Pluts workload.

Caveat: Individual metrics can't be evaluated in isolate; the resource usage profile as a whole provides insight into the system's performance and responsiveness.


### Forging Loop

1. Block Context Acquisition in the forging loop increases by ~10ms.
2. Mempool snapshotting shows an increase by 16ms under value workload; under Plutus workload, it increases by 3ms.
3. Ledger ticking improves slightly by 1-2ms.

The metric _'Slot start to announced'_ (see in attachements) is cumulative, and demonstrates how far into a slot the block producing node first announces the new header.

### Peer propagation

1. Block fetch time increases for full blocks by 9%. For small blocks, it improves by 7%.
2. Time to resend a block after fetching increases by 8% for full blocks, whereas it improves by 2% for small blocks.
3. Block adoption by a peer takes 12% more time for a full block, but happens faster by 4% for a small block.

### End-to-end propagation

This metric encompasses block diffusion and adoption across specific percentages of the benchmarking cluster, with 0.80 adoption meaning adoption on 80% of all cluster nodes.

The metric exhibits an increase by ~10% across all centiles for full blocks, whereas it improves by 5-6% for small blocks in the higher (80th and above) centiles.

## Contact

This is the first time we're publishing, to a wider audience, such benchmarking results. We are aware that more context and detail may be needed with regard to specfic metrics or benchmarking methodology.

We are looking to gather questions, both general and specific, so that we can provide a suitable FAQ and possibly improve presentation in the future.

## Attachments

Expand Down
Loading