OpenTelemetry integration #699

psardana · 2024-11-11T15:07:53Z

Fixes Issue

closes #701

Changes proposed

Check List (Check all the applicable boxes)

I sign off on contributing this submission to open-source
My code follows the code style of this project.
My change requires changes to the documentation.
I have updated the documentation accordingly.
All new and existing tests passed.
This PR does not contain plagiarized content.
The title of my pull request is a short description of the requested changes.

Screenshots

Note to reviewers

netlify · 2024-11-11T15:08:07Z

✅ Deploy Preview for opal-docs ready!

Name	Link
🔨 Latest commit	`4b24dc0`
🔍 Latest deploy log	https://app.netlify.com/sites/opal-docs/deploys/67598d96cb796400087c0a65
😎 Deploy Preview	https://deploy-preview-699--opal-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

…ck latency of data update events feat(prometheus_metrics.py): create data_update_latency histogram to monitor latency of data update events

…l into prometheus_integration

…etrics to use opal_server.metrics.prometheus_metrics for better organization chore(requirements.txt): add prometheus_client to dependencies for metrics tracking functionality

…c to track updates per topic feat(prometheus_metrics.py): introduce data_update_count_per_topic counter for monitoring data updates by topic

… to enhance observability fix(api.py): increment policy bundle request count and measure latency for bundle generation fix(callbacks.py): observe size of changed directories in policy update notifications fix(task.py): track policy update count and latency when triggering policy watcher

danyi1212 · 2024-11-18T08:56:48Z

Hey @psardana, thank you for this contribution! 💎

Can you please add documentation about the metrics and explain how to set it up?
Additionally, please add a docker-compose example with a Prometheus container collecting the metrics from OPAL, which will be good for showcasing it in docs 🙂

Notice that there are conflicts against the main branch, please make sure to rebase from master.
Also note that the tests are failing, I think due to invalid label names in the metrics. Please take a look at it 😁

Looking forward for this! 🙏

psardana · 2024-11-18T13:21:25Z

Hey @psardana, thank you for this contribution! 💎

Can you please add documentation about the metrics and explain how to set it up? Additionally, please add a docker-compose example with a Prometheus container collecting the metrics from OPAL, which will be good for showcasing it in docs 🙂

Notice that there are conflicts against the main branch, please make sure to rebase from master. Also note that the tests are failing, I think due to invalid label names in the metrics. Please take a look at it 😁

Looking forward for this! 🙏

Thank you for the review! I have added commits for documentation, docker compose and fixed the label names.

danyi1212

Looks very good! 🌟
I've left some comments about specific areas and improvements.

Upon review, some instrumented parts appear to align more with tracing rather than pure metrics. This led to some unnatural metrics and duplications, having separate latency - count and error metrics.

To address this, we suggest exploring OpenTelemetry, which offers native Prometheus integration alongside robust tracing capabilities.

Here's a proposed mapping of the current metrics to OpenTelemetry:

opal_server_data_update -> Trace
opal_server_policy_update -> Trace
opal_server_policy_bundle_request -> Trace
opal_server_policy_bundle_size -> Metric
opal_server_active_clients -> Metric
opal_client_data_subscriptions -> Metric
opal_client_data_update_trigger -> Trace
opal_client_data_update_apply -> Trace (new)
opal_client_policy_update_apply -> Trace (new)
opal_client_policy_store_status -> Metric

We believe this approach will provide a more comprehensive observability solution.

Please let us know your thoughts, and let's work together to enhance OPAL's observability 💎

documentation/docs/tutorials/monitoring_opal.mdx

docker/prometheus/prometheus/docker-compose-with-prometheus-metrics.yml

documentation/docs/tutorials/monitoring_opal.mdx

packages/opal-client/opal_client/client.py

packages/opal-common/opal_common/monitoring/prometheus_metrics.py

packages/opal-server/opal_server/server.py

packages/opal-server/opal_server/policy/bundles/api.py

danyi1212

Look very good! Kudus on the work. I've left a few small details to fix and improve, let me know if you have question and when they are ready

packages/opal-client/opal_client/client.py

packages/opal-server/opal_server/data/data_update_publisher.py

packages/opal-server/opal_server/policy/bundles/api.py

packages/opal-server/opal_server/policy/watcher/task.py

packages/opal-server/opal_server/pubsub.py

danyi1212

Great job! 👏💎🎉

added initial config for prometheus integration in opal server

63b1fca

psardana and others added 3 commits November 11, 2024 16:11

Merge branch 'master' into prometheus_integration

cff9251

feat(data_update_publisher.py): add data_update_latency metric to tra…

d1ac117

…ck latency of data update events feat(prometheus_metrics.py): create data_update_latency histogram to monitor latency of data update events

Merge branch 'prometheus_integration' of github-personal:psardana/opa…

dcdbbf7

…l into prometheus_integration

psardana marked this pull request as draft November 11, 2024 15:30

psardana added 12 commits November 12, 2024 06:14

refactor(api.py, data_update_publisher.py): update import paths for m…

faf0cd8

…etrics to use opal_server.metrics.prometheus_metrics for better organization chore(requirements.txt): add prometheus_client to dependencies for metrics tracking functionality

feat(data_update_publisher.py): add data_update_count_per_topic metri…

bc06d6b

…c to track updates per topic feat(prometheus_metrics.py): introduce data_update_count_per_topic counter for monitoring data updates by topic

moved prometheus metrics to opal common

af89d39

scopes and security prometheus metrics added

6e50055

added client metrics endpoint and total active clients metric

f44e2bb

data topic subscribed by client

00830eb

added token type in prometheus metric

192a255

added labels to the metrics for data and policy updates

bd32ddb

added labels in token requests generations and errors

a6f39b9

added more labels for prometheus metrics for scope

699e9da

added metrics for opal client

140bac3

psardana changed the title ~~added initial config for prometheus integration in opal server~~ prometheus integration Nov 15, 2024

Merge branch 'master' into prometheus_integration

fa0937e

psardana marked this pull request as ready for review November 15, 2024 07:29

danyi1212 self-requested a review November 17, 2024 12:27

danyi1212 assigned psardana Nov 18, 2024

psardana and others added 6 commits November 18, 2024 10:47

Merge branch 'master' into prometheus_integration

e6a1096

added docker compose example with prometheus

1cb9ccf

Merge branch 'master' into prometheus_integration

5a9bfbb

fixed metric labels

e1fec7e

Merge branch 'master' into prometheus_integration

035755d

added documentation

78ef777

Merge branch 'master' into prometheus_integration

8a57392

danyi1212 requested changes Nov 19, 2024

View reviewed changes

added open telemetry traces and metrics

5bde819

psardana requested a review from danyi1212 November 20, 2024 11:20

psardana added 6 commits November 20, 2024 12:32

added metrics and traces in documentation

81e989a

added scope id as an attribute

d197fe8

renamed docker compose

f63beb6

fixed how span is being used

52e3114

added documentation

2db0782

fixed descriptions

837780e

psardana changed the title ~~prometheus integration~~ OpenTelemetry integration Nov 25, 2024

danyi1212 requested changes Dec 2, 2024

View reviewed changes

psardana added 3 commits December 3, 2024 12:21

removed top level code and protected metrics end point

61fd24c

fixes for tracing spans

7f2fab0

fix metrics end point

191cebf

psardana requested a review from danyi1212 December 4, 2024 12:44

psardana and others added 2 commits December 4, 2024 15:34

fixed docker compose and removed logging exporter from otel

ea0c770

Merge branch 'master' into prometheus_integration

6f07718

danyi1212 approved these changes Dec 4, 2024

View reviewed changes

danyi1212 added 2 commits December 11, 2024 14:49

Merge branch 'master' into prometheus_integration

bf897dc

Fixed pre-commit

4b24dc0

danyi1212 merged commit 141a4a6 into permitio:master Dec 11, 2024
9 of 12 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OpenTelemetry integration #699

OpenTelemetry integration #699

psardana commented Nov 11, 2024 •

edited

Loading

netlify bot commented Nov 11, 2024 •

edited

Loading

danyi1212 commented Nov 18, 2024 •

edited

Loading

psardana commented Nov 18, 2024

danyi1212 left a comment

danyi1212 left a comment

danyi1212 left a comment

OpenTelemetry integration #699

OpenTelemetry integration #699

Conversation

psardana commented Nov 11, 2024 • edited Loading

Fixes Issue

Changes proposed

Check List (Check all the applicable boxes)

Screenshots

Note to reviewers

netlify bot commented Nov 11, 2024 • edited Loading

✅ Deploy Preview for opal-docs ready!

danyi1212 commented Nov 18, 2024 • edited Loading

psardana commented Nov 18, 2024

danyi1212 left a comment

Choose a reason for hiding this comment

danyi1212 left a comment

Choose a reason for hiding this comment

danyi1212 left a comment

Choose a reason for hiding this comment

psardana commented Nov 11, 2024 •

edited

Loading

netlify bot commented Nov 11, 2024 •

edited

Loading

danyi1212 commented Nov 18, 2024 •

edited

Loading