syslog-ng custom log metrics

[pepov]

SyslogNG custom source and output metrics

Source metrics are put after the json parser.

kind: Logging
apiVersion: logging.banzaicloud.io/v1beta1
metadata:
  name: logging
spec:
  controlNamespace: default
  fluentbit: {}
  syslogNG:
    metrics: {}
    sourceMetrics:
      - key: custom_input
        labels:
          test: asd

Results in:

source "main_input" {
    channel {
        source {
            network(flags("no-parse") port(601) transport("tcp") max-connections(100) log-iw-size(10000));
        };
        parser {
            json-parser(prefix("json."));
            metrics-probe(key("custom_input") labels(
                "logging" => "logging"
                "test" => "asd"
            ));
        };
    };
};

Output metrics are put before the log reaches the destination and is decorated with the output metadata like: name, namespace and scope which holds whether the output is a local or global one.

apiVersion: logging.banzaicloud.io/v1beta1
kind: SyslogNGFlow
metadata:
  name: all1
spec:
  match: {}
  outputMetrics:
    - key: custom_output
      labels:
        flow: all1
  localOutputRefs:
    - http
  globalOutputRefs:
    - http2

Becomes:

filter "flow_default_all1_ns_filter" {
    match("default" value("json.kubernetes.namespace_name") type("string"));
};
log {
    source("main_input");
    filter("flow_default_all1_ns_filter");
    log {
        parser {
            metrics-probe(key("custom_output") labels(
                "flow" => "all1"
                "logging" => "logging"
                "output_name" => "http2"
                "output_namespace" => "default"
                "output_scope" => "global"
            ));
        };
        destination("clusteroutput_default_http2");
    };
    log {
        parser {
            metrics-probe(key("custom_output") labels(
                "flow" => "all1"
                "logging" => "logging"
                "output_name" => "http"
                "output_namespace" => "default"
                "output_scope" => "local"
            ));
        };
        destination("output_default_http");
    };
};

Metrics:

k curl logging-syslog-ng-0:9577/metrics  | grep custom_
# TYPE syslogng_custom_output gauge
syslogng_custom_output{flow="all1",logging="logging",output_name="http2",output_namespace="default",output_scope="global"} 42
syslogng_custom_output{flow="all1",logging="logging",output_name="http",output_namespace="default",output_scope="local"} 42
syslogng_custom_output{flow="all2",logging="logging",output_name="http2",output_namespace="default",output_scope="global"} 154
# TYPE syslogng_custom_input gauge
syslogng_custom_input{logging="logging"} 154

[siliconbrain]

lgtm

[pepov]

Update: based on @tarokkk's request logging name is now part of the automatic metric labels