-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
output/eve: reduce fflush call count #12288
Conversation
Issue: 3449 Add a flush function to packet logger registration and collapse the parameter count for registration functions.
This commit adds 2 EVE output buffering settings - buffer-size value which specifies the amount of buffering, if any, for regular/file output types. - flush-interval Specifies the cadence at which Suricata will direct detect threads to flush EVE output. Issue: 3449
Issue: 3449
Issue: 3449 Add flushing functions and infrastructure. This includes: - Flushing functions for packet loggers - Log file flushing support
Issue: 3449 Add a flush directive to the packet that is distinct from the existing "log flush" flag as the new flag is to distinguish between the 2 use cases.
Issue: 3449
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #12288 +/- ##
==========================================
- Coverage 83.22% 83.21% -0.02%
==========================================
Files 912 913 +1
Lines 257311 257891 +580
==========================================
+ Hits 214154 214600 +446
- Misses 43157 43291 +134
Flags with carried forward coverage won't be shown. Click here to find out more. |
Information: QA ran without warnings. Pipeline 23957 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Needs a rebase
For the design :
I also do not understand why we need a new general mechanism for event scheduling, when we have already FlowRecycler and/or FlowManager that do such things.
I think you have a good reason for that, but I do not see it explicited
And why do we need flushing at all ? Will things not get flushed anyways later at last when we close the file if we did not fill the buffer ?
Contined in #12525 |
Continuation of #12135
Reduce fflush calls on output streams (regular files only).
Output can be buffered, specify the buffer-size with
outputs.<type>.buffer-size
. A value of 0 selects no buffering; otherwise, up to the buffer-size value can be buffered. Note that this buffering is part of the stdio library.Since output can be buffered, a mechanism that periodically flushes the output streams has been added. The
heartbeat.output-flush-interval
configuration setting specifies at what interval the output should be flushed. A value of 0 means never flush.Link to redmine ticket: 3449
Describe changes:
buffer-size
. When 0, unbuffered I/O is used; other values are used to set the stdio buffer size. The value isoutputs.eve-log.buffer-size
heartbeat.output-flush-interval
-- to set cadence for Suricata periodically directing detect threads to flush EVE output. To be used in conjunction withbuffer-size
. Setheartbeat.output-flush-interval
to the number of seconds Suricata should periodically cause the EVE output to be flushed. The default value is0
which instructs Suricata never to cause the EVE output to be flushed.heartbeat.output-flush-interval
heartbeat.output-flush-interval
is between 1 and 60 in seconds).Updates:
Suricata build information
Benchmarks/Measurements
Hyperfine was used to measure results with my pcap collection and ET Pro
Summary: Buffering had the biggest impact; using the flushing mechanism had little impact but is necessary for integrity.
Recommendation:
eve-log.buffer-size
: TBDheartbeat.output-flush-interval
TBDPermutations benchmarked for
buffer-size
andoutput-flush-interval
Hyperfine output