Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimize uploading ballots for recreating prior elections #776

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ArendPeter
Copy link
Member

@ArendPeter ArendPeter commented Feb 6, 2025

Description

This significantly reduces the time it takes to upload ballots in bulk. All benchmarks are done with the Burling 2009 mayor election which has 8980 votes. And unless otherwise stated each test was using a batch size of 100. Increasing the batch size above 100 would exceed the request size limit

Prior optimizations
I already pushed a changing bypassing the ballot queue system. That design is important when processing real time votes and ensuring each vote is counted exactly once, but it's not as important when analyzing prior elections. This effectively reduce the time from several minutes to 77 seconds

Optimizations in Pull Request
Baseline - 77s
Backend no-op - 6.7s - This gives an estimate for how much of
w/ reduced logging - 69.791s - Previous logging was very verbose and it slowed down the response time.
w/ bulk database update - 34.935s - Executes the bulk upload in a single database query, rather than looping with one row per query
w/ batch size of 50 - 58.495s - Decreasing the batch size make it significantly slower (and justified optimizing batch size in the other direction)
w/ batch size of 700 - 25 s - Refactor the request format to allow for larger batch sizes

Screenshots / Videos (frontend only)

the video was even faster than my benchmarks. I probably should have averaged over more iterations to get a better sense

2025-02-05.22-12-22.mp4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant