Use fail2ban to ban ip that hits a lot the rate limiter

[alexgarel]

We can use fail2ban on the proxy to ban ips that are getting too many 429 (rate limiting).
a rule like more than x 429 in y minutes.
It will ban bots without banning legitimate users.

[alexgarel]

This is not so easy as it looks.

Currently fail2ban is only activated on the reverse proxy where we don't have access log, so we are unable to look for 469 (or 444) return code.

More-over the fail2ban included filter to ban on too may rate limit hits, is based upon the nginx error log (which make sense to avoid parsing to many logs), so it have to be set up on the off server.

[alexgarel]

Using fail2ban on the nginx proxy will not resolve the problem as it banning the real incoming ip will still allow requests from it, as it goes through the reverse proxy.

One possible solution is to mount logs from the o*f servers (which are in a separate dataset) in the proxy container, to make them accessible to fail2ban.
It then only requires a changes in the jail definition to grep for the right file.

But I think there is a better solution.

We should follow this best practice on reducing parasitic log on the proxy and add a log which is active only on certain type of status response ($status in nginx).

This way we can only log 401, 403, 404, 469, etc... in special log files (supicious_*.log) and have fail2ban watch those files.