Kubernetes Logs with GELF

[badokun]

I have followed the guide documented here: https://docs.datalust.co/docs/collecting-kubernetes-logs

I do see the logs come into Seq, but I can't seem to figure out how the raw_payload can be expanded into their own properties.

My configuration is exactly as described in the article. Any help will be greatly appreciated (I'm sure it's just some missing config)

[nblumhardt]

Hi @badokun, thanks for reaching out. The properties in raw_payload are already visible on the event (they're in the list above it, and as the event message - GELF adds leading underscores to property names as a convention).

To get rid of the raw_payload field, you can un-check the "Enable diagnostics" option in the GELF input's settings. Let me know if this helps!

Best regards,
Nick

[badokun]

Thanks @nblumhardt for getting back to me. I now realize that all the properties are indeed already visible on the event with the exception of the short_message, which appears to be displayed by default as the message in the UI. This is great, however ideally I'd like that message to be dissected into its own segments/components. When looking at the short_message we can ignore the leading timestamp, but the WARNING should be a property, say level. The text follow that is what I'd like to see in the UI as the message.

Here's another example:

By looking at the short_message we can see there are various properties combined into one message. I'd like to break them apart, and not sure how to go about it. Is it a matter of creating a PARSER for it? Since I'm already using the input-kubernetes.conf which points to docker as the parser, how could I introduce another to deal with, in the example above longhorn.

I'll share my configuration for clarity:

    fluent-bit.conf: |
      [SERVICE]
          Flush         1
          Log_Level     info
          Daemon        off
          Parsers_File  parsers.conf
      @INCLUDE input-kubernetes.conf
      @INCLUDE filter-kubernetes.conf
      @INCLUDE output-gelf.conf
    input-kubernetes.conf: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Parser            docker
          DB                /var/log/flb_kube.db
          Mem_Buf_Limit     5MB
          Skip_Long_Lines   On
          Refresh_Interval  10
    filter-kubernetes.conf: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Kube_URL            https://kubernetes.default.svc:443
          Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
          Kube_Tag_Prefix     kube.var.log.containers.
          Merge_Log           On
          Merge_Log_Key       log_processed
          K8S-Logging.Parser  On
          K8S-Logging.Exclude Off
    output-gelf.conf: |
      [OUTPUT]
          Name                    gelf
          Match                   kube.*
          Host                    ${FLUENT_GELF_HOST}
          Port                    ${FLUENT_GELF_PORT}
          Mode                    ${FLUENT_GELF_PROTOCOL}
          Gelf_Short_Message_Key  log
    parsers.conf: |
      [PARSER]
          Name   apache
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache2
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache_error
          Format regex
          Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
      [PARSER]
          Name   nginx
          Format regex
          Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   json
          Format json
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name        docker
          Format      json
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On
      [PARSER]
          Name        syslog
          Format      regex
          Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
          Time_Key    time
          Time_Format %b %d %H:%M:%S

Thanks for helping out

[nblumhardt]

Hi! I think it will be a parser, though I'm not well-versed enough in Fluent Bit to get any further than that without re-reading the docs, unfortunately. Keen to hear how you go!

[badokun]

I tried adding configuration as specified here but still no luck.

What baffles me is how this doesn't affect everyone using Seq logging in Kubernetes. There's nothing special about my installation. I thought fluent bit is the recommended approach

# CRI Parser
[PARSER]
    # http://rubular.com/r/tjUt3Awgg4
    Name cri
    Format regex
    Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%dT%H:%M:%S.%L%z

[INPUT]
    Name tail
    Path /var/log/containers/*.log
    Parser cri
    Tag kube.*
    Mem_Buf_Limit 5MB
    Skip_Long_Lines On

[nblumhardt]

Hi! I dug around a bit and found:

fluent/fluent-bit#2095

It suggests that the docs are outdated, and you should use:

^(?<time>.+) (?<stream>stdout|stderr) (?<_p>F|P) (?<log>.*)$

as the cri parser's Regex.

There's a bit more discussion and some other options thrown around in the ticket.

Let me know if this one does the trick!

Best regards,
Nick

[badokun]

Thanks for checking. I tried that but had no impact. I don't really have more time to spend on this, really unfortunate.

I'll leave my config here in case someone stumbles upon this thread with some insights.

# config.yaml
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
apiVersion: v1
data:
    fluent-bit.conf: |
      [SERVICE]
          Flush         1
          Log_Level     info
          Daemon        off
          Parsers_File  parsers.conf
      @INCLUDE input-kubernetes.conf
      @INCLUDE filter-kubernetes.conf
      @INCLUDE output-gelf.conf
    input-kubernetes.conf: |
      [INPUT]
          Name              tail
          Tag               kube.*
          Path              /var/log/containers/*.log
          Parser            docker
          DB                /var/log/flb_kube.db
          Mem_Buf_Limit     5MB
          Skip_Long_Lines   On
          Refresh_Interval  10
    filter-kubernetes.conf: |
      [FILTER]
          Name                kubernetes
          Match               kube.*
          Kube_URL            https://kubernetes.default.svc:443
          Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
          Kube_Tag_Prefix     kube.var.log.containers.
          Merge_Log           On
          Merge_Log_Key       log_processed
          K8S-Logging.Parser  On
          K8S-Logging.Exclude Off
    output-gelf.conf: |
      [OUTPUT]
          Name                    gelf
          Match                   kube.*
          Host                    ${FLUENT_GELF_HOST}
          Port                    ${FLUENT_GELF_PORT}
          Mode                    ${FLUENT_GELF_PROTOCOL}
          Gelf_Short_Message_Key  log
    parsers.conf: |
      [PARSER]
          Name   apache
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache2
          Format regex
          Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   apache_error
          Format regex
          Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$
      [PARSER]
          Name   nginx
          Format regex
          Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name   json
          Format json
          Time_Key time
          Time_Format %d/%b/%Y:%H:%M:%S %z
      [PARSER]
          Name        docker
          Format      json
          Time_Key    time
          Time_Format %Y-%m-%dT%H:%M:%S.%L
          Time_Keep   On
      [PARSER]
          Name        syslog
          Format      regex
          Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
          Time_Key    time
          Time_Format %b %d %H:%M:%S