Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add AMSMonitor interface and unify both RMQ API (#32) #62

Merged
merged 2 commits into from
Apr 4, 2024

Conversation

lpottier
Copy link
Collaborator

@lpottier lpottier commented Mar 23, 2024

AMSMonitor helps us monitor each stage of the python pipeline with simple decorator approach. Each call to AMSMonitor decorator or context manager populate a shared dictionary that track each call.

In addition of recording (by default, can be restricted with @AMSMonitor(record=["attr1", "attr2"])) all attributes of a class, AMSMonitor will also inject its own metrics, for example the duration of each block amsmonitor_duration. For example, the following code:

class ExampleTask1(Task):
  def __init__(self):
    self.total_bytes = 0
  @AMSMonitor()
  def __call__(self):
    i = 0
    with AMSMonitor(obj=self, tag="while_loop"):
      while (i<3):
        self.total_bytes += 10
        time.sleep(1)
        i += 1
    self.total_bytes += 5

will produce the following dictionary (dumped as JSON here):

{
  "ExampleTask1": {
    "while_loop": {
      "02/29/2024-19:27:53": {
        "total_bytes": 30,
        "amsmonitor_duration": 3.004607439041138
      }
    },
    "__call__": {
      "02/29/2024-19:29:24": {
        "total_bytes": 35,
        "amsmonitor_duration": 3.10461138
      }
    }
  }
}

This PR also addresses #32 and unify both RMQ API.

@lpottier lpottier requested a review from koparasy March 23, 2024 00:40
Copy link
Member

@koparasy koparasy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I have some minor comments.

Signed-off-by: Loic Pottier <[email protected]>
@koparasy koparasy merged commit 41687c2 into develop Apr 4, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants