docs: investigation of mongodb unavailability on 2023/10/14

[stephanegigandet]

The reason for the unavailability of MongoDB this weekend seems to be something written in Python running on ovh2.

[sweep-ai]

Apply Sweep Rules to your PR?

• Apply: Leftover TODOs in the code should be handled.
• Apply: All new business logic should have corresponding unit tests in the tests/ directory.
• Apply: Any clearly inefficient or repeated code should be optimized or refactored.

[raphael0202]

Pymongo 3.12.3 and Python 3.11 is definitely Robotoff. I'm looking into it.

[raphael0202]

After some investigation, it's not clear why there were so many connections attempts from pymongo. It's clear however that it tried to reconnect many times to MongoDB: https://openfoodfacts.sentry.io/issues/4339801224/events/92bac62eb9ea4ad1a38cb207af088060/?project=1415205

Connection pooling is handled by Pymongo entirely, and we're currently respecting client creation best practices (not creating the client before forking).
In Pymongo 4.0, one of the improvement is a better connection management to avoid "connection storms" (https://pymongo.readthedocs.io/en/stable/changelog.html#breaking-changes-in-4-0), upgrading from 3.12 to 4.5 would possibly help.

[alexgarel]

I added the remarks of Raphaël on robotoff side.

I think we can merge. Alright @stephanegigandet ?