Old JobSink secrets lead to processing old events again while new events are lost

[tikr7]

First of all, I really like the new feature JobSink in Knative Eventing because it helps to execute long running processes while being fully compatible with the eventing stack!

Describe the bug
JobSink Secret mounted to the Job is not cleaned up after deleting the Job.
There is even no update on that Secret with new CloudEvent information.
This leads to processing old events again and the new event is lost.

Expected behavior
Either clean up Kubernetes Secret after Kubernetes Job has been deleted.
Or overwrite the existing Kubernetes Secret with the new CloudEvent before it gets executed.

To Reproduce
Lets assume, I trigger 3 Jobs via eventing (names of Jobs are somehow autogenerated):

job-sink-logger
job-sink-logger0
job-sink-logger0xgzzy

Lets clean them up (imagine we have 1000 of those):

kubectl delete job job-sink-logger job-sink-logger0 job-sink-logger0xgzzy

We still have the Secrets with the CloudEvent inside:

kubectl get secret
job-sink-logger
job-sink-logger0
job-sink-logger0xgzzy

Now lets assume, I trigger 3 jobs via eventing again (names of Jobs are somehow autogenerated):

job-sink-logger0
job-sink-logger1z
job-sink-logger3mpyh

Secret job-sink-logger0 was not updated and still has the old CloudEvent, hence the Job job-sink-logger0 processes the old event and the new event never gets processed.

Knative release version v1.15.4 (I assume also v1.16.0)

Additional context
If something is unclear, just let me know.

[tikr7]

@pierDipi could you help here? :)

[pierDipi]

@tikr7 thanks for reporting, I'll take a look

[pierDipi]

With the associated PR #8331:

$ k get jobs,secrets 
NAME                                COMPLETIONS   DURATION   AGE
job.batch/job-sink-successh1dy3pu   12/12         37s        48s

NAME                             TYPE     DATA   AGE
secret/job-sink-successh1dy3pu   Opaque   1      48s

$ k delete jobs job-sink-successh1dy3pu 
job.batch "job-sink-successh1dy3pu" deleted

$ k get jobs,secrets 
No resources found in default namespace.

One additional note

A job and secret is created per unique event id and source, so if 2 events have the same id and source, then the event is considered the "same" event and a new job + secret will not be created if a job already exists.

This is following the CloudEvents specifciation: https://github.com/cloudevents/spec/blob/main/cloudevents/spec.md

Producers MUST ensure that source + id is unique for each distinct event.

[pierDipi]

This will be released next Tuesday in a patch release for both 1.15 and 1.16