tetragon: Add cgroup rate support #2177
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for tetragon ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
from
9ef5863 to
9087d81
Compare
olsajiri
added
the
release-note/minor
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
6 times, most recently
from
7c42c49 to
0745c09
Compare
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
4 times, most recently
from
1504be7 to
21d0856
Compare
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
6 times, most recently
from
d61b1f4 to
362bca3
Compare
kkourt
reviewed
Mar 15, 2024
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
8 times, most recently
from
ff897c8 to
7a0bc3f
Compare
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
3 times, most recently
from
5aa8c90 to
8f9f4c8
Compare
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
2 times, most recently
from
0985904 to
3ee02af
Compare
Adding throttle message that is going to be triggered when
the cgroup cross the rate limit or comes back.
message ProcessThrottle {
// Throttle type
ThrottleType type = 1;
// Cgroup name
string cgroup = 2;
}
Signed-off-by: Jiri Olsa <[email protected]>
Adding cgrouprate ebpf code that keeps track of cgroup rate in per cpu hash. We keep track of received events in 2 intervals and compute slide window rate based on current time. If the limit is crossed we throttle the cgroup and in following changes we will also send an event to user space. Signed-off-by: Jiri Olsa <[email protected]>
Adding --cgroup-rate=options to configure exec/fork/exit allowed cgroup rates. The option is in following format: 'events,interval' Example: --cgroup-rate="1000,1s" # 1000/second rate --cgroup-rate="10000,5s" # 10000/5 seconds rate Signed-off-by: Jiri Olsa <[email protected]>
Adding cgroup rate rmdir sensor to clean up cgroup_rate_map entries. At the moment it's only removal of the cgroup_rate_map entry, so doing it unconditionally, because the overhead is negligent. Signed-off-by: Jiri Olsa <[email protected]>
Adding base cgrouprate object which will be updated in following changes Signed-off-by: Jiri Olsa <[email protected]>
Adding cgrouprate logic for checking on cgroup rate limit.
- cgrouprate channel is fed with cgroups that crossed the
rate limit through Check function
- cgrouprate keeps list of active (throttled) cgroups and
periodically checks if the cgroup rate dropped down
- throttle stop event is sent if the rate drops down under the
limit or the cgroup has no traffic for more than 5 seconds
Signed-off-by: Jiri Olsa <[email protected]>
Adding throttle event that is sent from cgroup_rate function when the rate limit is crossed and the cgroup is not throttled. Signed-off-by: Jiri Olsa <[email protected]>
Adding several metrics to help debug/diagnose the cgrouprate code. Signed-off-by: Jiri Olsa <[email protected]>
Create cgrouprate instance for tetragon and for observer test helper plus adding and configuring related maps in base sensor. Signed-off-by: Jiri Olsa <[email protected]>
Adding support to control cgroup rate for execve events. Signed-off-by: Jiri Olsa <[email protected]>
Adding support to control cgroup rate for clone events. Signed-off-by: Jiri Olsa <[email protected]>
Adding support to control cgroup rate for exit events. Signed-off-by: Jiri Olsa <[email protected]>
The base sensor now depends on config options, but the option can be enabled for specific test and base sensor is not recalculated. Adding base sensor for observer (sensorTest) which contains all possible programs and maps. Signed-off-by: Jiri Olsa <[email protected]>
Adding throttle exec/fork event test that sets cgroup rate limit and spawns bit more processes per second and triggers throttle start event, then wait for throttle stop event. Signed-off-by: Jiri Olsa <[email protected]>
Adding throttle processCgroup test that models possible cases and checks the throttle stop event is properly sent or not. Signed-off-by: Jiri Olsa <[email protected]>
Adding test for ParseCgroupRate function. Signed-off-by: Jiri Olsa <[email protected]>
Adding support to display throttle events in getevents like: 💥 exit ubuntu-22 /usr/bin/sleep 0.1s 0 🚀 process ubuntu-22 /usr/bin/sleep 0.1s 🧬 throttle START session-130.scope-10741 🧬 throttle STOP session-130.scope-10741 🚀 process ubuntu-22 /usr/bin/git diff 🚀 process ubuntu-22 /usr/bin/pager Signed-off-by: Jiri Olsa <[email protected]>
olsajiri
force-pushed
the
pr/olsajiri/cgroup_rate
branch
from
3ee02af to
35dcbb8
Compare
tixxdz
approved these changes
May 6, 2024
|
I think we could config the merge commit from PR to always squash for a better look :D 
|
|
@Trung-DV We intentionally split changes into multiple patches. This makes things easier to understand when looking at git history. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adding support to monitor cgroup rate for exec, clone and exit events. The rate is configurable.