bug: unable to rerun gen-secrets job

[venkatamutyala]

What happened?

It seems like this isn't a way to have the job that generates the secrets run again and overwrite the existing secret:

Error: secrets "bootstrap" already exists2023/06/26 01:05:00 secrets "bootstrap" already exists

As I was digging around to see if there was a flag for this, I found that it should probably just work per: https://github.com/pomerium/ingress-controller/blob/main/deployment.yaml#L227

What did you expect to happen?

I was expecting the job that bootstraps secrets to just overwrite the existing secret. I also was expecting some breakage may occur where I have to restart pods after the generation of new secrets.

How'd it happen?

1. Deleted the cronjob resource
2. I reapplied the cronjob resource
3. I let the Job run and got this error from gen-secrets:

Error: secrets "bootstrap" already exists2023/06/26 01:05:00 secrets "bootstrap" already exists

What's your environment like?

• I am using the pomerium ingress-controller image: docker pull pomerium/ingress-controller:sha-dd49d67
• kubernetes

What's your config.yaml?

I have a very basic installation that leverages your kustomize file. The only thing missing in this file is the override I pass in to use sha-dd49d67:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - git::https://github.com/pomerium/ingress-controller.git//config/default?ref=v0.22.2

I tried digging up the config but i'm unclear on where the default config is stored in pomerium/ingress-controller:sha-dd49d67

What did you see in the logs?

Error: secrets "bootstrap" already exists2023/06/26 01:05:00 secrets "bootstrap" already exists

Additional context

Per: #477 this may also be the expected behavior as it was only intended to run once. So perhaps rotation isn't an option unless i manually delete the secret first?

[wasaga]

This is indeed an expected behaviour. This job only runs once and would not overwrite secrets again.
Rotating secrets is organization-specific, and you have to come up with your own policy of doing so.

[venkatamutyala]

Thanks for confirming @wasaga! I am closing this issue.

[venkatamutyala]

For anyone coming across this issue, I am just manually deleting the secret using kubectl delete secret and then recreate the cronjob so that it runs the job again to generate the secret. It seems to work fine. I also kill all of my pomerium pods to be on the safe side but if you can't tolerate an outage then you may want to look at another approach.