If I'm using Helm and have 3 services each running in prod + staging namespaces on 3 domains, how should I deal with external-dns' manifests?

[nickjj]

I am using Helm to abstract away the idea of a "web service" and I'm using Route53 on AWS which requires setting a --txt-owner-id to match your Zone ID on AWS, but you can only have 1 root domain name per zone so that makes me think no matter what I will need at least 1 external-dns deployment + rbac manifest for each service that uses its own domain name.

Helm lets you pretty easily spin up something in a specific namespace by running helm -n prod upgrade --create-namespace --install myservice . and then you can use values files to merge and customize things for each namespace, for example that command could be modified to helm -n staging upgrade --create-namespace --install myservice . -f values-staging.yaml and now it'll spin that service up in its own namespace.

This lead me to having a values file for each service / namespace that has this along with some ingress rules:

externalDNS:
  version: "v0.7.6"
  upsertOnly: true
  args:
    - "--txt-owner-id=ZABC123"

Then in Helm, I created a library chart which has everything a web service needs to run like the deployment, service, ingress and it also has external-dns too which I defined as:

{{- define "shared-lib.external-dns" -}}
apiVersion: "rbac.authorization.k8s.io/v1"
kind: "ClusterRole"
metadata:
  name: "external-dns"
  namespace: {{ .Release.Namespace | quote }}
rules:
- apiGroups: [""]
  resources: ["services","endpoints","pods"]
  verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
  resources: ["ingresses"]
  verbs: ["get","watch","list"]
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["list","watch"]
---
apiVersion: "rbac.authorization.k8s.io/v1"
kind: "ClusterRoleBinding"
metadata:
  name: "external-dns-viewer"
  namespace: {{ .Release.Namespace | quote }}
roleRef:
  apiGroup: "rbac.authorization.k8s.io"
  kind: "ClusterRole"
  name: "external-dns"
subjects:
- kind: "ServiceAccount"
  name: "external-dns"
  namespace: {{ .Release.Namespace | quote }}
---
apiVersion: "apps/v1"
kind: "Deployment"
metadata:
  name: "external-dns-{{ include "shared-lib.fullname" . }}"
spec:
  strategy:
    type: "Recreate"
  selector:
    matchLabels:
      app: "external-dns-{{ include "shared-lib.fullname" . }}"
  template:
    metadata:
      labels:
        app: "external-dns-{{ include "shared-lib.fullname" . }}"
    spec:
      serviceAccountName: "external-dns"
      containers:
      - name: "external-dns-{{ include "shared-lib.fullname" . }}"
        image: "k8s.gcr.io/external-dns/external-dns:{{ .Values.externalDNS.version }}"
        args:
          - "--source=service"
          - "--source=ingress"
          - "--provider=aws"
          - "--aws-zone-type=public"
          - "--registry=txt"
          {{- if .Values.externalDNS.upsertOnly }}
          - "--policy=upsert-only"
          {{- end }}
          {{- if .Values.externalDNS.args }}
            {{- range .Values.externalDNS.args }}
          - {{ . | quote }}
            {{- end }}
          {{- end }}
{{- end }}

And this is where the problem begins. This works fine as long as I run 1 server in 1 namespace (let's say prod) but when I try to install this using Helm in the staging namespace then I get an error of:

Error: rendered manifests contain a resource that already exists.
Unable to continue with install: ClusterRole "external-dns" in namespace "" exists
and cannot be imported into the current release: invalid ownership metadata;
annotation validation error: key "meta.helm.sh/release-namespace" must equal "staging": current value is "prod"

I'm not sure how to fix that, but let's say it's fixable. This leads us down a path of having (6) external-dns deployments running with 3 services across 2 environments. Is that going to be an issue?

What if I end up creating sub-domains for each PR for these services and run each one in its own namespace? This could end up in a situation where you have dozens of copies of external-dns running.

I'm not really sure how to solve this in a clean way. What do you suggest?

Ideally I'd like to keep everything needed for a web service in this Helm library chart so I can continue running 1 Helm command to deploy / update it. This is especially nice because then all of the related service configuration is in that Helm values file too.