Clean up and fix for NIC Pod failing to bind when NGINX exits unexpectedly

[AlexFenlon]

Proposed changes

This fix removes any.sock files in the /var/lib/nginx directory after nginx is killed if the pod is restarted. This issue does not persist if the pod is deleted and recreated.

Before these files were removed we would get log spam similar to the example in #4604 or if readOnlyRootFileSystem is enabled.

Example of what is now removed

2025/01/13 15:50:56 [emerg] 32#32: bind() to unix:/var/lib/nginx/nginx-plus-api.sock failed (98: Address already in use)
2025/01/13 15:50:56 [emerg] 32#32: bind() to unix:/var/lib/nginx/nginx-config-version.sock failed (98: Address already in use)
2025/01/13 15:50:56 [emerg] 32#32: bind() to unix:/var/lib/nginx/nginx-418-server.sock failed (98: Address already in use)
2025/01/13 15:50:56 [notice] 32#32: try again to bind() after 500ms

Fixes: #6752 & #4604

Checklist

Before creating a PR, run through this checklist and mark each as complete.

• I have read the CONTRIBUTING doc
• I have added tests that prove my fix is effective or that my feature works
• I have checked that all unit tests pass after adding my changes
• I have updated necessary documentation
• I have rebased my branch onto main
• I will ensure my PR is targeting the main branch and pulling from my branch from my own fork

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 12 lines in your changes missing coverage. Please review.

Project coverage is 52.73%. Comparing base (80f67d3) to head (5c7f933).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
cmd/nginx-ingress/main.go	0.00%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7121      +/-   ##
==========================================
- Coverage   52.76%   52.73%   -0.04%     
==========================================
  Files          89       89              
  Lines       20831    20843      +12     
==========================================
  Hits        10992    10992              
- Misses       9382     9394      +12     
  Partials      457      457              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.