Nginx may be missing a restart on worker refresh?

[PietroPasotti]

Bug Description

when you deploy cos Pro and refresh the worker, tempo's coordinator fails to route to the /ready endpoint

an nginx restart seems to fix it, which is odd because the FQDNs haven't changed.

To Reproduce

deploy cos-pro
refresh tempo-worker

Environment

n/a

Relevant log output

n/a

Additional context

No response

[PietroPasotti]

probably an instance of https://tenzer.dk/nginx-with-dynamic-upstreams/

[PietroPasotti]

looks like adding a resolver: ns.dns.cluster.local statement to each route fixes it (updates the IP within 5 seconds).
TBD: can we assume this dns address is stable?

if not, we could use this approach:

dig NS <my fqdn>

(we'd need to add dig to our rocks/images or apt install it)

[PietroPasotti]

or:

# cat /etc/resolv.conf                                           
search foo.svc.cluster.local svc.cluster.local cluster.local home
nameserver 10.152.183.10                                         

[PietroPasotti]

PRs to fix this issue:

• added resolver directive tempo-coordinator-k8s-operator#66
• add dns resolver to nginx config mimir-coordinator-k8s-operator#83
• add dns resolver to nginx config loki-coordinator-k8s-operator#18

[michaeldmitry]

Reopening this issue, as we're still experiencing this issue with tempo-coordinatior-k8s latest edge 46

[michaeldmitry]

It seems that adding the dynamic resolver alone won't fix the issue.
See more in https://www.f5.com/company/blog/nginx/dns-service-discovery-nginx-plus

Solutions considered:

1. Adding a dynamic resolver fetched from /etc/resolv.conf → Nginx will only do lookups at startup and cache the result
2. Adding a dynamic resolver + using proxy_pass with variables → This will enable DNS lookup during runtime. However, this will not work for our case because it doesn't work with upstream groups. So, we can’t leverage the load balancing functionality of upstream groups.
3. Use Nginx 1.27.3 to leverage the Nginx-plus functionality. they pulled into opensource nginx.

Solution 3 works best for our use-case. However, as of today, there's no Ubuntu-provided nginx image for 1.27.3

Sub-solutions considered:
A. Going ahead of Debian in adding an nginx package for plucky 25.04 with a development release (1.27.x)
B. Create our custom rock

Sub-solution A will probably not be an option since nginx odd releases are development releases, and we can't take 1.27.x for 25.04 and not know if 1.28.x (the stable release) will be released in time for 26.04 (the LTS)

This leaves us with sub-solution B, where we could create a rock.