Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WARN log messages for failed matches are noisy #65

Open
joshuar opened this issue Sep 20, 2018 · 2 comments
Open

WARN log messages for failed matches are noisy #65

joshuar opened this issue Sep 20, 2018 · 2 comments
Assignees

Comments

@joshuar
Copy link

joshuar commented Sep 20, 2018

When using a lot of different dissect filters, in particular, where all of these are matching against a single event, the warning messages from failed matches spams the log with messages of the following:

[2018-09-20T14:00:14,305][WARN ][org.logstash.dissect.Dissector] Dissector mapping, field not found in event {"field"=>"somefield", "event"=>{"host"=>"ludicra", "@timestamp"=>2018-09-20T04:00:14.108Z, "@version"=>"1", "message"=>"sdgasdg"}}

There is one of these messages for each failed match. On a busy server with lots of patterns, this adds up. Can we reduce the level of these messages to DEBUG to avoid spamming the logs?

@guyboertje
Copy link

Sure. I can do that. I think it is a bug as that warning is supposed to be rare. I fixed it while implementing a POC of sequencing.
Be aware, though, the addition of sequencing should ameliorate the need for multiple dissects.

@guyboertje guyboertje self-assigned this Sep 24, 2018
@seang-es
Copy link

This has a performance impact as well. Some users are seeing a perf hit from all the additional logging. When the Logstash logs were diverted to /dev/null, dissect was much faster than grok, but when the failures were sent to the log file they were about the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants