Vector Routing and GEOIP

[tastyfrankfurt]

Hey Guys,

Trying to convert logstash to vector having a few issues with the routing transform and geoip logic. Basically the transform below is not matching the check statements even though the fields are there i. Yes we have a parsing map above that i have tested by deleting and adding fields. I have tested each below with vector vrl and it works fine.

t_kfk_parsed_route_geoip:
type: route
inputs: [t_kfk_parsed_route.parsesuccess]
route:
"geoipsrc": exists(.source.ip) && !match!(.source.ip,r'(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)|(^22[4-9]{1}.)|(^23[0-9]{1}.)|(^255.255.255.255)')
"geoipdst": exists(.destination.ip) && !match!(.destination.ip,r'(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)|(^22[4-9]{1}.)|(^23[0-9]{1}.)|(^255.255.255.255)')
"geoipsrcnat": exists(.source.ip_nat) && !match!(.source.ip_nat,r'(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)|(^22[4-9]{1}.)|(^23[0-9]{1}.)|(^255.255.255.255)')
"geoipdstnat": exists(.destination.ip_nat) && !match!(.destination.ip_nat,r'(^127.0.0.1)|(^10.)|(^172.1[6-9].)|(^172.2[0-9].)|(^172.3[0-1].)|(^192.168.)|(^169.254.)|(^22[4-9]{1}.)|(^23[0-9]{1}.)|(^255.255.255.255)')

Additionally because of the way the geoip transform works we have to run it back through multiple geoip transforms as some logs have multiples of the fields above, ie .destination.ip and .source.ip exist in the same log message. It would be great in geoip could be incorporated into the vector vrl remap implementation.

[tastyfrankfurt]

I have identified why the routing above is not working as its a problem with json, current json is looking for the value represented as follows,
"source":{"ip":"127.0.0.1"}

But the value in the json we are handling is stored as follows, is there a way to cater for the '.' as a string value rather than a sub element?
"source.ip":"127.0.0.1"

[jszwedko]

Hi @tastyfrankfurt ! You can use "s around path keys if they contain special characters like .. In this case that would look like ."source.ip". You can see an example here: https://vector.dev/docs/reference/vrl/expressions/#path-example-quoted-path

[tastyfrankfurt]

@jszwedko managed to get around this using the quoted path as stated, i also had to daisy chain the geoip to make sure that each field was enriched correctly. But noted that the GEOIP didnt support quoted paths as well as the output to index when using the {{ fieldname }}. So i wrote some VRL at the start to duplicate these fields but use underscores instead to evaluate and then cleaned up the unneeded fields at the end, with the exception for the ones i used for the elasticsearch sink.
if exists(."destination.ip") {
.destination_ip = ."destination.ip"
}
Its not the most efficient way to do it but it is working.

[jszwedko]

Ah, yes, that is a common problem when using Elasticsearch. We are thinking about ways to make that easier to deal with in #3588