Enforce quota and hard cap for monthly execution minutes #1284
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tw4l
force-pushed
the
issue-1261-execution-time-exceeded
branch
6 times, most recently
from
a2a94de to
d5cca67
Compare
tw4l
changed the title
tw4l
force-pushed
the
issue-1261-execution-time-exceeded
branch
3 times, most recently
from
764a867 to
9949676
Compare
|
@Shrinks99 Assigned you for a copy and docs check :) |
Shrinks99
reviewed
Oct 17, 2023
Shrinks99
reviewed
Oct 17, 2023
Shrinks99
reviewed
Oct 17, 2023
Shrinks99
reviewed
Oct 17, 2023
Shrinks99
requested changes
Oct 17, 2023
|
@Shrinks99 ready for re-review :) |
|
Change tab title from "Billing" to "Org Limits" for now until we sort out / add more billing features to the app. |
Handled similarly to storage quotas in backend and frontend, except execution minutes quotas are tracked and enforced monthly. Almost surely is buggy at this stage, and there is no soft/hard cap distinction yet.
|
Thanks for this! The keeping track of current running exec time is neat but has two potential issues:
I've attempted to refactor this a bit to instead force increments in short intervals to avoid tracking uncommitted exec time altogether. |
This adds an slight refactor to #1284 to ensure execution minutes are updated incrementally while a crawl is running. The execution minutes are updated on the org and on the crawl if: - 60 seconds elapsed from last update - any pod has newly exited - crawl has been canceled (compute to current time) - if month has changed This should ensure that the execution time quota is accurate within 1 minute, and also that crawls running at the end of the month properly count towards old and new month quotas (a bit of an edge case yes, but bound to happen!) other cleanup: - add background tasks to a set to avoid premature garbage collection (see: https://stackoverflow.com/a/74059981) - use dt_now() consistently instead of datetime.utcnow() to store second-rounded dates
|
With #1314 merged in, I think we might be finally go to go on this one! |
tw4l
force-pushed
the
issue-1261-execution-time-exceeded
branch
from
489d8b2 to
d268922
Compare
- Remove user-set overage - Just have a 'single execution minutes' quota, rename hard cap to be the quota (for now) - have backend compute storage and exec minutes quotas, report to frontend (will eventually involve additional variables)
tw4l
force-pushed
the
issue-1261-execution-time-exceeded
branch
from
081ee64 to
cee0c85
Compare
Co-authored-by: Tessa Walsh <[email protected]>
|
@ikreymer @Shrinks99 Pulled in the org dashboard execution time meter into this PR, should all be ready for a last review together now |
- Cap display of execution time usage at 100% - Remove extraneous tooltip
Shrinks99
approved these changes
Oct 26, 2023
ikreymer
approved these changes
Oct 26, 2023
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #1261 Closes #1092
The quota for monthly execution minutes is treated as a hard cap. Once it is exceeded, an alert indicating that an org has exceeded its monthly execution minutes will display and the user will be unable to start new crawls. Any running crawls will be stopped once the quota is exceeded.
An execution minutes meter bar is also added in the Org Dashboard and displayed if a quota is set. More detail in #1305, which was merged into this branch.
Changes
execution_minutes_quota_reacheddetail in backend from crawl config/runendpoint, and don't run new workflows on creation (similar to storage quota)To test
Manually
Org Monthly Execution Minutes Quotato a low number of minuteshttps://webrecorder.netwith extra hops enabled and no page limit)Nightly tests
To run the nightly tests locally (requires pytest):
pytest backend/test_nightly/test_execution_minutes_quota.pyThese tests may take 5 or so minutes to complete.