I tried adding this probe to OAB:

livenessProbe: 
    exec:
      command:
        - /bin/sh
        - -c
        - celery --workdir src --app openarchiefbeheer.celery inspect active
    initialDelaySeconds: 60
    periodSeconds: 10
    timeoutSeconds: 10
    failureThreshold: 3
    successThreshold: 1

This seemed to not interrupt the long running tasks and when I shut down redis inside the pod from the redis-cli, after the redis pod was restarted, the worker was also restarted.

The command celery inspect active returns a 0 exit code if there are active workers and a non-zero exit code otherwise. So this seems to be a good possibility for a liveness probe?

@alextreme Lets discuss

Discussed, both the inspect active and ping healthchecks are necessary as they monitor and avoid 2 different failure scenario's (if the task queue is active and reachable vs if tasks can be performed)

I'd like to have both livenessprobes in each chart with celery, and have higher timeouts for the ping-healthcheck if there are longrunning celery tasks

I am not sure I understand why the extra ping is needed. If the worker isn't working properly the active command will return a non-zero exit code. The only case I can think of where ping would fail and not active is if the worker is processing long tasks or there is a task that is hanging and the worker cannot process new tasks (so it can't respond to the ping).
But I think that this last case where a task is "hanging" should be handled by the celery timeout 🤔 Or am I missing something?

The only case I can think of where ping would fail and not active is if the worker is processing long tasks or there is a task that is hanging and the worker cannot process new tasks (so it can't respond to the ping). But I think that this last case where a task is "hanging" should be handled by the celery timeout 🤔 Or am I missing something?

No you are not missing something, this is exactly the scenario that I have in mind. In these cases there should be a celery soft/hard timeout on the task ( https://docs.celeryq.dev/en/stable/userguide/workers.html#time-limits ), but just like setting a timeout on requests-calls it is easy to forget and has lead to issues in the past that celery workers required manual restarting.

And you can neatly put timeouts on your tasks, however third-party celery tasks can also be enabled and we don't vet these tasks for having timeouts set

I see 🤔 but if it is forgotten, it is not difficult to add to the projects? And with the project level settings, you can set the timeout for all the tasks (so I think that this would also take care of the 3rd party libraries who run tasks?).

I am just a bit reluctant to add the ping to all projects as a blanket, because projects like OAB who have long running tasks then would struggle with the ping probe 🤔

Well, I guess it depends how long the tasks are expected to run.

With OIP we have a 15-minute hard task limit ( https://github.com/maykinmedia/open-inwoner/blob/fbae2b82d82107a9e1e615c7ff801f851533e8c7/src/open_inwoner/conf/base.py#L687 ), however the 15-minute limit is configurable via an envvar.

Then again, due to this the need for a ping healthcheck is indeed zero.

So lets add the inspect active healthcheck to all celery-using components, and decide on a component-per-component basis which of these situations apply: a global hard task limit, or long running tasks which duration can't be properly determined a priori. If either of these are applicable, then the ping healthcheck is to be removed

Sounds good, I will add them 👍