How can I deploy/sync a Job resource with Argo CD?

[wolframkleis]

My question is: Can we sync Job resources with Argo CD? If yes, do you have any hints or links to samples for how to do it?
I found nothing in documentation or with Google search, only about hook-related jobs.
But this is not about hook jobs. In my scenario, the primary resource I want to deploy and manage is a Job.
I have, for example, a job that does a database migration. This job needs to re-run whenever the image name is updated in the source repo.

I tried with Argo CD

• First time I tried it with default sync options
  • initially, the Job was created by Argo executed as expected
  • then I changed the job definition in Git and synched the Argo application
  • I got this error:
    spec.template: Invalid value: core.PodTemplateSpec{ ...<my whole Job definition> .....} : field is immutable
  • I understand that this attempt fails, because you can't update a Job, you must delete it and create it a again
• Next time, I tried with Replace=true in syncOptions
  • I hoped that Argo would delete the old job and create the new one with the new image
  • After changing the Job in the source Git repo, I synched the argo application again
  • This time I got different errors:
    • spec.selector: Required value
    • spec.template.metadata.labels: Invalid value: map[string]string(nil): selectordoes not match templatelabels`
    • spec.selector: Invalid value: "null": field is immutable
    • spec.template: Invalid value: core.PodTemplateSpec{ ...<my whole Job definition> .....} : field is immutable

Any ideas what I am doing wong? (I am still new to Argo).

Argo version: v2.1.7

here is the simple Job I used for testing this

apiVersion: batch/v1
kind: Job
metadata:
  name: jobtest
spec:
  backoffLimit: 1
  completions: 1
  template:
    spec:
      containers:
      - name: jobtest
        image: busybox
        command: ["echo", "This is the changed test job"]
      restartPolicy: Never

here is my application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: debug-job
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: explorer
  destination:
    namespace: dev
    server: https://kubernetes.default.svc
  source:
    path: debug-job
    repoURL: https://github.example.com/myrepo.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    syncOptions:
      - Replace=true
    retry:
      limit: 1 
      backoff:
        duration: 5s 
        factor: 2 
        maxDuration: 30s 

Here is the error of the 2nd attempt:

error when replacing "/dev/shm/526139654": 
Job.batch "jobtest" is invalid: 
[ 	spec.selector: Required value, 
	spec.template.metadata.labels: 
		Invalid value: map[string]string(nil): `selector` does not match template `labels`, 
	spec.selector: Invalid value: "null": field is immutable, 
	spec.template: Invalid value: core.PodTemplateSpec{
		ObjectMeta:v1.ObjectMeta{
			Name:"", 
			GenerateName:"", 
			Namespace:"", 
			SelfLink:"", 
			UID:"", 
			ResourceVersion:"", 
			Generation:0, 
			CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:0, loc:(*time.Location)(nil)}}, 
			DeletionTimestamp:(*v1.Time)(nil), 
			DeletionGracePeriodSeconds:(*int64)(nil), 
			Labels:map[string]string(nil), 
			OwnerReferences:[]v1.OwnerReference(nil), 
			Finalizers:[]string(nil), 
			ClusterName:"", 
			ManagedFields:[]v1.ManagedFieldsEntry(nil)}, 
			Spec:core.PodSpec{
				Volumes:[]core.Volume(nil), 
				InitContainers:[]core.Container(nil), 
				Containers:[]core.Container{core.Container{
					Name:"jobtest", 
					Image:"busybox", 
					Command:[]string{"echo", "This is the changed test job"}, 
					Args:[]string(nil), WorkingDir:"", 
					Ports:[]core.ContainerPort(nil), 
					EnvFrom:[]core.EnvFromSource(nil), 
					Env:[]core.EnvVar(nil), 
					Resources:core.ResourceRequirements{Limits:core.ResourceList{
					  "cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, 
					    d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, 
					  "memory":resource.Quantity{i:resource.int64Amount{value:134217728, scale:0}, 
					    d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}, 
					Requests:core.ResourceList{
					  "cpu":resource.Quantity{i:resource.int64Amount{value:100, scale:-3}, 
					    d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"100m", Format:"DecimalSI"}, 
					  "memory":resource.Quantity{i:resource.int64Amount{value:67108864, scale:0}, 
					    d:resource.infDecAmount{Dec:(*inf.Dec)(nil)}, s:"", Format:"BinarySI"}}}, 
					VolumeMounts:[]core.VolumeMount(nil), 
					VolumeDevices:[]core.VolumeDevice(nil), 
					LivenessProbe:(*core.Probe)(nil), 
					ReadinessProbe:(*core.Probe)(nil), 
					StartupProbe:(*core.Probe)(nil), 
					Lifecycle:(*core.Lifecycle)(nil), 
					TerminationMessagePath:"/dev/termination-log", 
					TerminationMessagePolicy:"File", 
					ImagePullPolicy:"Always", 
					SecurityContext:(*core.SecurityContext)(nil), Stdin:false, StdinOnce:false, TTY:false}}, 
					EphemeralContainers:[]core.EphemeralContainer(nil), 
					RestartPolicy:"Never", 
					TerminationGracePeriodSeconds:(*int64)(0xc01b3d0c18), 
					ActiveDeadlineSeconds:(*int64)(nil), 
					DNSPolicy:"ClusterFirst", 
					NodeSelector:map[string]string(nil), 
					ServiceAccountName:"", 
					AutomountServiceAccountToken:(*bool)(nil), 
					NodeName:"", 
					SecurityContext:(*core.PodSecurityContext)(0xc02e903a80), 
					ImagePullSecrets:[]core.LocalObjectReference(nil), 
					Hostname:"", 
					Subdomain:"", 
					SetHostnameAsFQDN:(*bool)(nil), 
					Affinity:(*core.Affinity)(nil), 
					SchedulerName:"default-scheduler", 
					Tolerations:[]core.Toleration(nil), 
					HostAliases:[]core.HostAlias(nil), 
					PriorityClassName:"", 
					Priority:(*int32)(nil), 
					PreemptionPolicy:(*core.PreemptionPolicy)(nil), 
					DNSConfig:(*core.PodDNSConfig)(nil), 
					ReadinessGates:[]core.PodReadinessGate(nil), 
					RuntimeClassName:(*string)(nil), 
					Overhead:core.ResourceList(nil), 
					EnableServiceLinks:(*bool)(nil), 
					TopologySpreadConstraints:[]core.TopologySpreadConstraint(nil)}}
		: field is immutable]

[jannfis]

Hm. I'd dare to say that Kubernetes Jobs are classical one time events. You deploy a Job into your cluster, it will run and afterwards be terminated - it can't be re-run. Probably due to this fact, lots of fields in the Job spec are immutable and can't be changed once the resource exists in your cluster. Also, this makes it a quite ungrateful resource for being managed in a GitOps way.

I'd suggest to make your Job a sync hook, with a proper deletion policy (e.g. BeforeHookCreation), so that the Job will be deleted and re-created on every sync. If you have no other resources for your application that could sync, maybe you can just create a ConfigMap (with a timestamp or similar property) for kicking off the sync.

[tooptoop4]

problem i found is that if using a postsync with BeforeHookCreation that the job will retry a lot if it failed. i can't seem to make it not retry

[neiser]

Another suggestion, as Pre/PostSync Hooks in ArgoCD are not very well supported IMHO and sometimes hard to debug in my experience, you might be able to leverage Helm and add a hash'd suffix to the Job resource name. The hash should only depend on the Job resource content, which can be achieved with the following Helm template magic:

{{- define "jobYaml" }}
{{- $jobName := printf "my-cool-job-%s" .jobSuffix }}
---
apiVersion: batch/v1
kind: Job
metadata:
  name: {{ $jobName }}
// continue writing down your cool Job resource using ".root" instead of "$" when templating
{{- end }}
{{- $jobSuffix := include "jobYaml" (dict "root" $) | toYaml | sha256sum | trunc 10 }}
{{- include "jobYaml" (dict "root" $ "jobSuffix" $jobSuffix) }}

This way, as every Job has a new distinct name, ArgoCD will delete the old Job and add a new Job on every sync, but also does not touch the Job if the hash is unchanged.

[milank78git]

@neiser I have a question about this. If I set up a Job according to your configuration, the Job indeed doesn't change during a sync or sync prune, but if I call a Hard-refresh and then a sync-prune, the Job gets deleted. How can this be resolved?

[neiser]

@milank78git I don't do hard refreshes very often and haven't tried what happens in this case. You'd probably need to check Argo CD source code to figure out what exactly is happening. For my purposes, I'm happy with the random job suffix and its ramifications. You may also consider using ArgoCD Workflows.

[gregwhorley]

problem i found is that if using a postsync with BeforeHookCreation that the job will retry a lot if it failed. i can't seem to make it not retry

That might be the backoffLimit setting in the Job itself: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy. FYI there is a beta k8s feature that allows you to set a pod failure policy so job failures can be controlled better than the backoff failure policy: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-failure-policy

[N214]

Using sync hook(Sync) with deletion policy(Success) do work but it also delete the job from the ArgoUI, let's say, I have multiple jobs from one application and I want only rerun one of the jobs from the UI, how can I do that?

[andrii-korotkov-verkada]

I'll close this as the original question is quite old and got answered back then. Please, open new questions instead.

[N214]

The age of the question should not be a concern before a proper solution is proposed...
I can also open another question/issue but there are plenty of issues regarding this already... And the answer above seems more like a workaround to me, the real issue has never been answered... Do Argo CD or Argo CD with Argo Workflow has a more appropriate solution?

[andrii-korotkov-verkada]

Okay, let me re-open.