Skip to content

Latest commit

 

History

History
103 lines (76 loc) · 6.43 KB

README.md

File metadata and controls

103 lines (76 loc) · 6.43 KB

TGIR S01E02: Help! RabbitMQ ate my RAM!

In this episode we will try to understand what happens under the Erlang hood when RabbitMQ is using a lot of memory and what we can do about it. We will be covering the interactions between RabbitMQ, Erlang Memory Allocators and the Linux Resident Set Size memory.

There is an unexpected twist - a.k.a. The One Hour Gap - as well as 2 short follow-ups to last month's episode.

As always, you can follow along - all commands are available as make targets in this episode's directory.

Timeline

  • 00:00:00 - Today's topic
  • 00:02:29 - RabbitMQ-Overview for 50 mil persistent messages in durable queues
  • 00:05:38 - Erlang-Memory-Allocators Grafana dashboard
  • 00:06:50 - Resident Set Size (RSS) vs Erlang Allocated Used & Unused
  • 00:10:00 - The most important Erlang memory allocators: binary, ETS & eheap
  • 00:14:00 - Multiblock vs Singleblock Carriers
  • 00:16:26 - Why does Allocated Unused memory keep growing?
  • 00:19:11 - Memory & disk considerations for recovering large backlogs of persistent messages
  • 00:25:55 - Memory profile for 50 mil persistent messages in lazy queues
  • 00:30:07 - Recommendations for @Farkie - #5
  • 00:31:54 - Recommendations for @lyrixx - rabbitmq/discussions#59
  • 00:34:34 - TGIR S01E01 follow-up: RabbitMQ 3.6 to 3.8 in-place upgrade
  • 00:36:16 - TGIR S01E01 follow-up: RabbitMQ node startup with many vhosts
  • 00:37:53 - Marques Johansson is paying attention! TGIR is also inspired by TBS, not just TGIK 🙌

Links

Screenshots

Find more screenshots in the directory.

Make targets

backlog-%                   Simulate message backlog
bash-%                      Open a shell session on instance
ctop-%                      Open ctop session
delete-%                    Delete instance
deps                        Resolve all dependencies
htop-%                      Open htop session
instances                   List all instances
management-%                Open RabbitMQ Management
netdata-%                   Start netdata & open dashboard
prometheus-%                Open RabbitMQ Prometheus
rabbitmq-%                  Create RabbitMQ server
ssh-%                       Open SSH session
workload-%                  Simulate workload

To reproduce the first setup you will need a GCP account & Google Cloud SDK, then just run:

make deps
make rabbitmq-durable backlog-durable workload-durable
make netdata-rabbitmq-durable
make management-rabbitmq-durable
make ctop-rabbitmq-durable

Replace durable with lazy in the above commands to reproduce the second setup.

To delete all instances, run:

make delete-rabbitmq-durable delete-backlog-durable delete-workload-durable
make delete-rabbitmq-lazy delete-workload-lazy