Where should control plane configuration live?

[mpstefan]

As we progress NKG beyond just the Gateway API, we need a place to put additional configuration features for the control plane. We've had less initially discussion on this, but this may be related to how to implement configuration for data plane specific features.

Questions to prompt discussion:

• Where would control plane configurations live?
• Should they be in the same place as data plane configurations?
• How would we continue to add to the elements you can configure for the control plane?

[brianehlert]

Are you referring to settings that are generic to the control plane itself?
Such as where a paying customer enters their ID, or where a dark customer needs to send their licensing data?
Things beyond the scope of the deployment and service definitions.

[sjberman]

One example that brought this topic up is configuring the logging level. And being able to reconfigure it at runtime.

[pleshakov]

I see two groups:

1. Things that cannot be changed on the fly Flags > Env Vars >
2. Things that can be changed Config Files > Defaults

[brianehlert]

This 1 and 2 are excellent points.
It is beneficial to the customer to enable as many things as possible via configuration files.
The stance on this is anything that does not require a stop and start of nginx should be possible through configuration files.
If a stop and start of nginx is required, then options such as deployment arguments make the impact very clear to the customer.

We tend to think of an nginx reload as not causing an outage, because new worker processes are spawned for the new configuration.
Anytime that is possible, we should consider the configuration parameter safe for a configuration file.
Anytime that is not possible we should move that configuration option to a deployment flag / env var.

I feel I am simply adding more description to what Michael stated above.

[f5yacobucci]

We were talking about this out-of-band and were discussing a split between control plane configuration and data plane configuration. Then a tier of data plane configuration that can be merged through the strata.

This doesn't necessarily address what's immutable and mutable, but very good points above, and a good shorthand for why one element belongs somewhere and not another.

Control plane config would be: controller log level, controller telemetry endpoints (prometheus, otel config for its data), licensing endpoints.

Data plane configuration is anything that would make it to nginx conf files. Main config (worker processes), block config (stream, http), server, location. Different layers allow different config.

A control plane CRD should be entirely mutable data, the control plane should watch and respect updates to this "file".

The data plane configuration templates might not be mutable at runtime but is something the controller should apply at a bootstrap stage. After starting this post, I think the DP templates would be shaped by how dynamic the controller gatewayclass support is, how its managing data planes instances, and might be better introduced when the controllers and data planes get separated - so this may be a pretty speculative structure right now.

The API configuration is entirely mutable and directed from the gateway API now and applied to the data planes.

Global Tier
------------------------------------------------------------------
NKG Config              -> NKG-CRD:
                           spec:
                             control-plane:
                               log-level: error,
                               ...
                             data-plane-template:
                               type: nginx
                               config:
                                 <almost anything from core / main conf>
                                 ssl_engine
                                 error_log
                                 load_module

Class Tier
------------------------------------------------------------------
GatewayClass Config     -> GatewayClass :: ParametersRef :: ConfigMap / CRD
                           spec:
                             data-plane-template:
                               type: nginx
                               config:
                                 <almost anything from core / main conf>
                                 ssl_engine
                                 error_log
                                 load_module

Configuration merged from above and overrides higher level template config.

API Tier
------------------------------------------------------------------
Gateway + Route + Polic -> http / stream block, server, location generated configuration for data plane

[pleshakov]

@f5yacobucci +1
but could you explain what would be the benefits for having data-plane-template in the NKG config? why not just leave it to the GatewayClass only

[pleshakov]

NKG already handles multiple GatewayClasses (it uses one and rejects the rest) provided that their controllerName matches NKG controllerName (configured via cli arg)

So I think it makes sense to divide control plane configuration from data plane configuration. For data plane configuration, configure it via GatewayClass extension points and Gateway (once it is part of the API https://gateway-api.sigs.k8s.io/geps/gep-1867/ )

Also, consider that we plan to eventually split control plane from data plane, with control plane configuring multiple data planes. In this case, bundling control plane and data plane doesn't look right.

[mpstefan]

With a specific recommendation for data plane, do we want to mention this option on our discussion for the data plane?

I think I would agree we'd want to separate the two, as they are going to expose two very different sets of features comparatively.

[mpstefan]

The team met today and decided to move forward with this in an enhancement proposal for dynamic control plane configuration. In this proposal, we'll be moving forward with configuration control plane configuration in its own CRD.