Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loki bloom buiilder ignores bloomshipper cache size limit settings #15909

Open
davidk-zx opened this issue Jan 23, 2025 · 0 comments
Open

Loki bloom buiilder ignores bloomshipper cache size limit settings #15909

davidk-zx opened this issue Jan 23, 2025 · 0 comments

Comments

@davidk-zx
Copy link

Describe the bug
We have configured loki helm chart to run with these settings regarding bloomshipper cache storage

storage_config:
  bloom_shipper:
    blocks_cache:
      # cleanup old blocks
      ttl: 30m
      soft_limit: 12GiB
      hard_limit: 25GiB

Pod was running 19 hours and is well over the storage limits

Our ephemeral cache size limit is 30G from k8s side, so this causes pods to crash eventually

To Reproduce
Run loki helm chart 6.23.0 with bloom filters enabled

Expected behavior
Bloom builder should contain its data within the expected range, usually under 12GiB specified

Environment:

  • EKS kubernetes cluster, deployed with loki helm chart 6.23.0

Screenshots, Promtail config, or terminal output

Bloomshipper size after 19h of pod runtime

/ $ du -h /var/loki | tail
420.0K	/var/loki/data/bloomshipper/bloom/loki_index_20111/fake/blocks/fa453f1058d6a7fa-fa749fd290872aba
40.0K	/var/loki/data/bloomshipper/bloom/loki_index_20111/fake/blocks/fd88138c5e21fc61-fdb8878c635afec5/1737587400671-1737617602900-e70ec597
40.0K	/var/loki/data/bloomshipper/bloom/loki_index_20111/fake/blocks/fd88138c5e21fc61-fdb8878c635afec5
4.8G	/var/loki/data/bloomshipper/bloom/loki_index_20111/fake/blocks
4.8G	/var/loki/data/bloomshipper/bloom/loki_index_20111/fake
4.8G	/var/loki/data/bloomshipper/bloom/loki_index_20111
29.9G	/var/loki/data/bloomshipper/bloom
29.9G	/var/loki/data/bloomshipper
29.9G	/var/loki/data
29.9G	/var/loki
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant