Don't require a privileged security context or the NET_ADMIN capability

[devjoes]

Is your feature request related to a problem? Please describe.

I'm trying to set this up on a multi-tenanted EKS cluster with around 300 namespaces. We enforce tenant boundaries quite strictly and lock everything down with gatekeeper. I'm running in to an issue where the controller can't inject the init container into pods due to its security context, I don't see anything about this in the controller logs. But I encounter this issue if I manually apply a pod manifest with the init and sidecar containers populated. If I bypass the gatekeeper rule then the pod is applied correctly and I can assume managed identities in Azure.

Describe the solution you'd like

Other components (like Istio's Envoy sidecar) reconfigure a pod's networking using CNI plugins and don't require a special security context. Is this an approach that you'd consider implementing?

Describe alternatives you've considered

We could potentially replicate the iptables configuration using an envoy filter.

Additional context

Applying:

apiVersion: v1
kind: Pod
metadata:
  name: test
  namespace: test-ns
  labels:
    azure.workload.identity/use: 'true'
    sidecar.istio.io/inject: 'false'
spec:
  volumes:
    - name: kube-api-access-vwrcf
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  initContainers:
  - name: init-networking
    image: mcr.microsoft.com/oss/azure/workload-identity/proxy-init:v1.3.0
    securityContext:
      capabilities:
        add:
        - NET_ADMIN
        drop:
        - ALL
      privileged: true
      runAsUser: 0
    env:
    - name: PROXY_PORT
      value: "8000"
  containers:
    - name: test
      image: xxxxxxxxxxxx.dkr.ecr.eu-west-1.amazonaws.com/yyyy/zzzz:stable
      env:
        - name: AZURE_CLIENT_ID
          value: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        - name: AZURE_TENANT_ID
          value: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
        - name: AZURE_FEDERATED_TOKEN_FILE
          value: /var/run/secrets/azure/tokens/azure-identity-token
        - name: AZURE_AUTHORITY_HOST
          value: https://login.microsoftonline.com/
      volumeMounts:
        - name: kube-api-access-vwrcf
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
        - name: azure-identity-token
          readOnly: true
          mountPath: /var/run/secrets/azure/tokens
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
      stdin: true
      stdinOnce: true
      tty: true
    - name: proxy
      image: mcr.microsoft.com/oss/azure/workload-identity/proxy:v1.3.0
      ports:
      - containerPort: 8000
  restartPolicy: Never
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  serviceAccountName: identity-test
  serviceAccount: identity-test
  nodeName: xxx.eu-west-1.compute.internal
  securityContext: {}
  imagePullSecrets:
    - name: image-pull-secret
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priorityClassName: default
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority

Results in:

Error from server (Forbidden): error when creating "test.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [default-privileged-restriction] Privileged container is not allowed: init-networking, securityContext: {"capabilities": {"add": ["NET_ADMIN"], "drop": ["ALL"]}, "privileged": true, "runAsUser": 0}
[default-capability-restriction] init container <init-networking> has a disallowed capability. Allowed capabilities are ["NONE"]