Add prometheus metrics endpoint with proper tag support

[kroepke]

What?

Add a Prometheus metrics endpoint to the standard of Graylog.
The exposed metrics should support tags for time series wherever they make sense, to more cleanly map to what Prometheus users expect to see.
It should be possible to bind the metrics endpoint to a separate address/port than the API, to allow for unauthenticated scraping from exporters/Prometheus. Oftentimes this will be either an internal monitoring network or localhost.

Why?

Exposing metrics in a flat way makes it difficult to work with using Prometheus and it is easier to flatten them for systems that do not have tag support, than the other way around.
Prometheus is a very common choice for collecting metrics, and Graylog should support it out of the box, without having to install an extra plugin that only solves half the problem.

Related issues:

graylog-labs/graylog-plugin-metrics-reporter#29
graylog-labs/graylog-plugin-metrics-reporter#35
graylog-labs/graylog-plugin-metrics-reporter#28

Your Environment

• Elasticsearch Version: 4.0

[mpfz0r]

It might make sense to consider using https://micrometer.io/ instead of coding this directly for prometheus

[jalogisch]

[z#844502]

[kroepke]

We should definitely consider various options, yes!

[l0calhost]

+1 for micrometer consideration, metric push mechanism would be a second big benefit to the tags.