Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

grafana-agent consumes a substantial amount of CPU when log folders have more than 100K files. #164

Open
drencrom opened this issue Aug 7, 2024 · 1 comment
Open

grafana-agent consumes a substantial amount of CPU when log folders have more than 100K files. #164

drencrom opened this issue Aug 7, 2024 · 1 comment
Labels
Status: Triage Type: Enhancement

Comments

@drencrom
Copy link

drencrom commented Aug 7, 2024

Enhancement Proposal

When grafana-agent is scraping log folders with hundreds of thousands of files it creates thousands of threads and consumes a lot of CPU time. It would be desirable to be able to customize the operator so that some log folders can be excluded (using the __path_exclude__ directive for example).
This problem was observed in particular with the /var/log/libvirt/qemu/ folder. There is a related enhancement proposal in loki (canonical/loki-k8s-operator#439) regarding the huge cardinality of the scraped data that might be tackled together with this one.
@drencrom drencrom changed the title grafana-agent consumes a substantial amount of CPU when log folders have more tha 100K files. grafana-agent consumes a substantial amount of CPU when log folders have more than 100K files. Aug 7, 2024
@sed-i
Copy link
Contributor

sed-i commented Nov 29, 2024

Duplicates: #48

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Triage Type: Enhancement
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@drencrom @sed-i