Should we use a custom nginx container?

[sjberman]

Right now our deployment simply uses the publicly available nginx image. While convenient, there are some downsides of this.

• Users have to create a ConfigMap containing our custom http matching module, which gets mounted as a volume into the nginx container. This pattern would be repeated for any new modules that we would add. If the ConfigMap is forgotten, the Pod fails to start.
• We have to run an init container to set the proper permissions on the /etc/nginx directory so that it can be written to.

The remedy for this is to package and distribute our own data plane container image. We would use the public nginx image as the base, and build on top of that.

Pros:

• Custom modules are packaged directly into the image, removing the need for ConfigMaps and volume mounts
• Directory permissions are defined in the image itself, removing the need for an init container

Cons

• another container that we have to manage
• user loses control over the nginx version; we control the version and it's only updated when we release.

The cons may not be a big deal, but worth thinking about.

Considerations:

• Does anything change when we separate the control planes and data planes?

[brianehlert]

The existing NIC project builds on top of the NGINX open source container as well as supports NGINX Plus through its installation process.
This is all provided in a Dockerfile that customers can and do use to build their own container images, when they choose not to use the images provided by the project.
This allows customers to use hardened OS settings, as well as 3rd party modules if they so choose.

NGINX has established this pattern over the 4+ years of the NGINX Ingress Controller project and it has proven successful. I would say that it is better received by the community than using init containers in general practice. These generally require elevated privileges and thus cause scanning engine alerts from tools such as Azure Defender which in turn causes support churn and exceptions taht customers would rather not implement.

The NIC project has only introduced pre-built container images a bit over 1 year ago. Prior to this, customer always had to build their own container images. While customers have asked for the convenience of pre-built images - the NIC project provides one with NGINX OSS that is available through DockerHub as well as other container registries and is very popular.
Customers who subscribe also have the option of downloading pre-built container images that include NGINX Plus.

At some time, this project will move to supporting NGINX Plus. And at that time will have to provide a method for a customer to build an image or for customers to exclusively pull images.

Again, I recommend considering the pattern that NIC has today. As it has proven to be a good balance between flexibility and capability. And while there are customers who prefer not to built their own images, the project can offer a pre-built image like NIC does and close that gap.

[sjberman]

As an open source project, users certainly always have the option of building the images themselves, which then gives them the freedom of controlling what they need (like nginx version).

The cons I list are targeted at the pre-built image option, since that's the most convenient (and probably more popular) option for users.

[brianehlert]

Maintaining pre-built containers is not horrible. At some point there will be nightly and release. Out of convenience to the team if not for the customers.
In the end, the convenience of pre-built containers is great for customers. The ability to deploy and go. That is a huge driver of adoption.

In my experience, most customers want the latest version of NIC, just like they want the latest version of 3rd party libraries.

NIC don't not continually update its images. They are fully patched and the latest release of NGINX at the time of a release. NKG should do the same.
If NGINX updates, such as issue a patch, we should try to also patch and update the images. But not for 3rd party libraries, unless it is a pretty severe CVE.

Customers who want an older version of NGINX would run an older release of the project. This is unlikely but possible.

I would not overthink it. Like I say, we have patterns and practices for this already with the sister NIC project. I see this as no different in intention or deliverable.

[brianehlert]

Honestly, I would rather get rid of the init container.
And be able to support NGINX Plus.
And be able to support capabilities such as readOnlyRootFilesystem which really require us to build the container.

[brianehlert]

My other comment on this is that pre-built images (even open source ones) also give us a mechanism to track pulls.
Such as here: https://hub.docker.com/v2/repositories/nginx/nginx-ingress/

As you can see from the pull count. It is pretty popular and impactful.

[pleshakov]

I don't think auxiliary ConfigMaps that we have is a problem at all. Once we have a helm chart -- which is the most common installation method -- it will create those ConfigMaps. There is also a related issue to simplify manifest installation #706

I like the fact we don't need to manage additional image. However, on top of the mentioned need for the need container, I see the following issues:

• the official image runs NGINX as root user, which goes against security best practices.
• because we use process namespace sharing, a side effect of that is containers see each others filesystems, not only shared volumes. Because of that, NGINX has access to the service account token (which has access to all cluster secrets and other resources) , which means malicious config can expose that token via NGINX web serving capabilities. To fix this, we can have a custom NGINX image with a light-weight agent that exposes a simple API to reload NGINX, so there is no need for process namespace sharing, while we continue to use shared volumes for configs. And the NKG pod will only mount service account credentials (we will need to opt out mounting and then mount only in NKG container https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#opt-out-of-api-credential-automounting )

[sjberman]

The problem with the ConfigMap we have now is that it requires us to copy a file from our source into there. When we create a helm chart, we're probably going to have to literally copy the code into the ConfigMap so when installing from the helm repo it has access to the module. Which means we're duplicating code.

Also, the nginx OSS otel module is currently not available via a package manager, which means you have to manually compile it and include it into your nginx image. While it may eventually be available via a package manager, who knows when that might be (or if it ever will be). So when we support tracing, we have to build the module ourselves. (EDIT: now that I think about it, even when it's available from a package manager, we would still have to build it into the image)

I like the idea of only having to manage a single container, but Brian makes a good point that ultimately our pipelines will handle all the management of building and publishing it. I don't think it's outlandish to have separate data plane and control plane containers that we ship. That's also coming from the NSM perspective where we were shipping a lot of containers.

Regarding the filesystem sharing, that should go away when we split CP and DP, right?

[pleshakov]

The problem with the ConfigMap we have now is that it requires us to copy a file from our source into there. When we create a helm chart, we're probably going to have to literally copy the code into the ConfigMap so when installing from the helm repo it has access to the module. Which means we're duplicating code

duplication will be our problem, not the user problem though. I think we can mitigate it with some automation - like a check that one file matches the other.

(EDIT: now that I think about it, even when it's available from a package manager, we would still have to build it into the image)

yeah, probably

Regarding the filesystem sharing, that should go away when we split CP and DP, right?

yep. see also https://github.com/nginxinc/nginx-kubernetes-gateway/blob/main/design/control-data-plane-separation/design.md#handling-users-secret-data

[sjberman]

Due to the second point about the otel module (and potential future modules), we'll probably need a custom container regardless. So in preparation for helm charts, we either need to build that custom container beforehand, or we'll have to write a temporary automation job to verify files match. I'm not a huge fan of doing that automation, I would prefer to get the container work done and then we don't have to worry about the extra work since we'll have to do it anyway. Also gives us time to ensure that our second image is released properly before we get too close to future deadlines.

[brianehlert]

I am going to bring this one point up again.
At some point in the not too distant future, this project will need to support NGINX Plus and not exclusively open source NGINX.
That is not an image that can be pulled on demand and injected via an init container.
This will come as customers want to unlock use cases that are exclusive to NGINX Plus. OIDC, JWT, state sharing, NAP WAF, etc.

[sjberman]

Issue created: #798