[Jaeger v2] Run benchmarks

[yurishkuro]

Part of #4843

Objective

• Measure performance differences between traditional Jaeger v1 setup and jaeger-v2 deployed in similar configuration
  • CPU utilization
  • Memory utilization
  • GC pressure / frequency
• Document / publish results

Steps

• Design testing setup
  • tracegen, collector, some storage backend (preferably remote like Elasticsearch, to avoid noisy neighbor effect)
  • how long do we need to run, with which throughput determined by tracegen
  • May use this blog as a blueprint: https://medium.com/jaegertracing/making-design-decisions-for-clickhouse-as-a-core-storage-backend-in-jaeger-62bf90a979d (but the objective in this ticket is simpler, we're not trying to find saturation points, just compare relative performance of v1 and v2)
• Prepare setup (scripts) to start required components and monitoring (e.g. prometheus + grafana, unless there is simpler way to measure)
• Run locally, capture results, publish on this ticket
• Run on a HW cluster (I can do it if the scripts are easily reusable)

This is not a very straightforward ticket, but hey that's how we learn, so I am still going to tag it as good-first-issue,

[Pushkarm029]

👀giving this a try

[Pushkarm029]

Hey @yurishkuro! Just wanted to confirm if we can start working on this issue now or if we need to wait until the other issues are resolved in #4843. Can you please let me know?

[yurishkuro]

It does not need to block, it's actually a part of #4843 scope.

[h4shk4t]

Hey @yurishkuro, I started a WIP repository on running benchmarks on Jaeger - benchmarks. I was thinking of running a docker-compose file for all the individual components (along with Prometheus and Grafana) for easily setting up the different services and getting the metrics. Please let me know if there is a better alternative.

[yurishkuro]

@h4shk4t docker-compose sounds good. I would rather not have a separate repository, but a benchmarks top directory in this repo, and you can keep a PR open for discussion. Before setting up anything though I suggest writing down a plan of what and how you want to setup & benchmark. That plan by itself would be a valuable artifact already.

[h4shk4t]

Hello @yurishkuro I wrote a document on the background and plan of action for running benchmarks. Kindly have a look: https://docs.google.com/document/d/1iDl8LaAJbCjp1qliECdTQvjEcJ8mfBsIII7J0RH7l6I/edit?usp=sharing

[yurishkuro]

Replacing this with #5768