[#148] Update liveness probes #164
Conversation
SilviaAmAm
force-pushed
the
feature/148-liveness-probes
branch
2 times, most recently
from
f484d0e to
d1f1bfe
Compare
|
Is to be verified first on our test-cluster before approval/merging |
|
Yes, I agree with the testing on our cluster. I will try to do it. |
|
Deployed on our test cluster:
I tested adding the livenessProbe to the worker with |
|
I attempted to resolve the merge conflicts however the linter failed, PR can be merged after resolving this properly |
OAB has long running tasks (>2h), so only the celery active probe is added
The object API has a celerty task limit of 5 min (soft), 15 min (hard). So no ping was added.
Open forms has task limits (5 (soft), 15 (hard) minutes) so no ping was added
OIP has a hard task limit of 15 min, so no ping probe was added
Timeouts added in maykinmedia/open-klant#316
Open Zaak has one long running tasks of which we don't know how long it will take, so no ping probe was added. The task: https://github.com/open-zaak/open-zaak/blob/70449f11a31a0e8a06d9e47d49e095b31d1068f6/src/openzaak/components/documenten/tasks.py#L333
SilviaAmAm
force-pushed
the
feature/148-liveness-probes
branch
from
3bf9fb2 to
e7e49ed
Compare
|
I will investigate! |
|
The problem was an issue in the bitnami charts (bitnami/charts#31348). It is now resolved. |
| - /app/bin/check_celery_worker_liveness.py | ||
| - /bin/sh | ||
| - -c | ||
| - celery --workdir src --app openforms.celery inspect --destination celery@${HOSTNAME} active |
There was a problem hiding this comment.
This is a bit too naive - the worker host name can be set via envvars and is especially useful when using different queues (short lived / prio / long-running tasks). The celery worker script in OF has:
QUEUE=${1:-${CELERY_WORKER_QUEUE:=celery}}
WORKER_NAME=${2:-${CELERY_WORKER_NAME:="${QUEUE}"@%n}}which indeed defaults to celery@$host, but is not guaranteed to be that
Fixes #148