Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: deal time monitor aggregate threshold verification #90

Merged
merged 1 commit into from
Feb 14, 2024

Conversation

vasco-santos
Copy link
Contributor

There was a bug in monitoring code. Basically condition was preventing everything from getting fired. Details follow

We have 3 main variables:

  • AGGREGATE_MONITOR_THRESHOLD_MS - number of ms an aggregate can be in the pipeline without checking state
  • MIN_PIECE_CRITICAL_THRESHOLD_MS - number if ms a piece can be pending a deal until critical alert
  • MIN_PIECE_WARN_THRESHOLD_MS - number if ms a piece can be pending a deal until warn alert

With these variables we do:

  1. get all the aggregates pending deals from DB
  2. filter the aggregates that are already more than AGGREGATE_MONITOR_THRESHOLD_MS pending
  3. grab filtered aggregates oldest piece in the pipeline information
  4. check if their age in the pipeline should trigger warn or critical alerts

we were failing on step 2, and doing the comparison the other way around. So, we would filter out the aggregates that we should actually inspect. The aggregates that we would analyse were the ones that actually were recent, and therefore no alerts

Copy link

seed-deploy bot commented Feb 12, 2024

View stack outputs

@seed-deploy seed-deploy bot temporarily deployed to pr90 February 12, 2024 16:37 Inactive
@vasco-santos vasco-santos merged commit 4bc7511 into main Feb 14, 2024
3 checks passed
@vasco-santos vasco-santos deleted the fix/deal-time-monitor branch February 14, 2024 08:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants