Recycle serialization buffers on transmission

[fuzzypixelz]

Adds a LIFO buffer pool in the context to reuse buffers allocated on serialization. The aim is not (only) to avoid the overhead of dynamic allocation but rather to enhance the cache locality of serialization buffers.

[fuzzypixelz]

The aim of this pull request is to fix high latency for relatively large topics (e.g. 1 MB tagus) in the single-process iRobot benchmark with --ipc off. Results were obtained using the Mont Blanc topology on an Ubuntu 24.04.1 LTS machine with an Intel(R) Core(TM) i5-6500 CPU @ 3.20GHz.

dd82e84

node	topic	size_b	mean_us	min_us	max_us
ponce	tagus	1000000	3016	2512	5705
geneva	tagus	1000000	3513	2997	6457
mandalay	tagus	1000000	2678	2214	5233

401016c

node	topic	size_b	mean_us	min_us	max_us
ponce	tagus	1000000	1947	1476	2788
geneva	tagus	1000000	2200	1667	3063
mandalay	tagus	1000000	1603	1225	1997

401016c + eed223a

node	topic	size_b	mean_us	min_us	max_us
ponce	tagus	1000000	393	257	1301
geneva	tagus	1000000	657	438	1701
mandalay	tagus	1000000	108	68	650

[clalancette]

All right, now that we've merged in #327 , we can consider this one. Please rebase this onto the latest, then we can do a full review of it. Until then, I'll mark it as a draft.

[ahcorde]

There are also many changes unrelated with the goal of this PR

[fuzzypixelz]

There are also many changes unrelated with the goal of this PR

There was a formatting error from my IDE. I've restored the files and manually re-applied the patches.

[clalancette]

Besides the comments inline, do you have any updated performance numbers here?

[clalancette]

This is going to require some additional plumbing, but I think we should respect the allocator passed in via the options during rmw_init. To get to that, we'll have to change the constructor of rmw_context_impl_s to pass that into the BufferPool constructor, and then we can store the pointer in this class and use it as necessary.

[fuzzypixelz]

I don't think we should be using the allocator passed in rmw_init_options_s for all allocation in the RMW.

1. According to the documentation, rmw_init_options_s::allocator is "[The] allocator used during internal allocation of init options, if needed.". So by the same rmw_init is called, we have no business allocating through it.
2. Other RMW implementations don't use rmw_init_options_s::allocator either. Instead they use rcutils_get_default_allocator and rmw_allocate.
3. The only place where this allocator seems useful is in rmw_init_options_fini.

We also allocate many std::vector and std::string without the default allocator anyway, which seems wrong.

I can make a subsequent pull request addressing this issue.

[fuzzypixelz]

Besides the comments inline, do you have any updated performance numbers here?

The following results were obtained using the Mont Blanc topology (with a 1 MiB tagus topic size) on an Ubuntu 24.04.1 LTS machine with an 12th Gen Intel(R) Core(TM) i5-1240P (different machine).

57a6b4b

node	topic	size_b	mean_us	sd_us	min_us	max_us
ponce	tagus	1000000	839	255	235	1227
geneva	tagus	1000000	1053	285	327	1490
mandalay	tagus	1000000	643	252	121	1011

8ba5278

node	topic	size_b	mean_us	sd_us	min_us	max_us
ponce	tagus	1000000	545	192	173	1091
geneva	tagus	1000000	702	193	253	1326
mandalay	tagus	1000000	342	189	86	771

cebb972

node	topic	size_b	mean_us	sd_us	min_us	max_us
ponce	tagus	1000000	368	162	87	879
geneva	tagus	1000000	522	162	127	1071
mandalay	tagus	1000000	188	159	38	646

cebb972 + 8ba5278

node	topic	size_b	mean_us	sd_us	min_us	max_us
ponce	tagus	1000000	224	31	60	313
geneva	tagus	1000000	379	52	99	528
mandalay	tagus	1000000	58	10	20	87