Periodic Metric exporter unable to collect data points on Passenger-based servers

[divyam-rai]

Description of the bug

Due to Passenger's smart spawning mechanism which creates worker processes by forking an initial pre-loader process, we notice that the thread for the periodic metric exporter is only attached to the pre-loader process and as such does not share memory with the other child processes which handle the actual incoming requests.

This causes the periodic metric reader to never have any data points since the metrics are stored in the memory of the child processes and are not shared with the thread which is attached to the pre-loader process.

The spawning mechanism, as well as its caveats, are described here: https://www.phusionpassenger.com/library/indepth/ruby/spawn_methods/#the-smart-spawning-method

To resolve this, the periodic metric reader needs to be initialised for each of the forked worker processes instantiated by Passenger, which was done as followed based on events that are made available within the internals of Passenger. This allowed us to send metrics as required.

if defined?(PhusionPassenger)
    PhusionPassenger.on_event(:starting_worker_process) do |forked|
      if forked
        metric_reader = OpenTelemetry::SDK::Metrics::Export::PeriodicMetricReader.new(
          exporter: OpenTelemetry::Exporter::OTLP::Metrics::MetricsExporter.new,
        )

        OpenTelemetry.meter_provider.add_metric_reader(metric_reader)
      end
    end
end