[CRITICAL] WORKER TIMEOUT-Unable to scale the number of PODs

[johnson-prabu]

Hi Team,

I am unable to bring more web pods, It is working fine with one pod. while i am increasing number of pods to 2 or more then the new pods are restarting continuously due to liveliness prob failed.

Exception SystemExit: 1 in <module 'collections' from '/usr/local/lib/python2.7/collections.pyc'> ignored
/app/redash/query_runner/cass.py:9: RuntimeWarning: Cython module failed to patch module with custom type
from cassandra.cluster import Cluster
[2022-01-10 14:35:38 +0000] [71] [INFO] Booting worker with pid: 71
[2022-01-10 14:35:38 +0000] [70] [INFO] Booting worker with pid: 70
[2022-01-10 14:35:38 +0000] [72] [INFO] Booting worker with pid: 72
[2022-01-10 14:35:38 +0000] [73] [INFO] Booting worker with pid: 73
[2022-01-10 14:36:09 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:70)
[2022-01-10 14:36:09 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:71)
[2022-01-10 14:36:09 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:72)
[2022-01-10 14:36:09 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:73)

[johnson-prabu]

Team, Any update?

[homiakos]

i had same problem after upgrade

kubectl -n redash-prod logs -f redash-686b459fc7-bd4xn redash-server
Using Database: postgresql://redash:@redash-postgresql:5432/redash
Using Redis: redis://:@redash-redis-master:6379/0
[2022-02-15 21:54:28 +0000] [6] [INFO] Starting gunicorn 20.0.4
[2022-02-15 21:54:28 +0000] [6] [INFO] Listening at: http://0.0.0.0:5000 (6)
[2022-02-15 21:54:28 +0000] [6] [INFO] Using worker: sync
[2022-02-15 21:54:28 +0000] [9] [INFO] Booting worker with pid: 9
[2022-02-15 21:54:28 +0000] [10] [INFO] Booting worker with pid: 10
[2022-02-15 21:54:28 +0000] [11] [INFO] Booting worker with pid: 11
[2022-02-15 21:54:28 +0000] [12] [INFO] Booting worker with pid: 12
[2022-02-15 21:54:58 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:9)
[2022-02-15 21:54:58 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:10)
[2022-02-15 21:54:58 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:11)
[2022-02-15 21:54:58 +0000] [6] [CRITICAL] WORKER TIMEOUT (pid:12)

[azadsagar]

@homiakos @johnson-prabu did you get the solution ?

[granescb]

@azadsagar @homiakos @johnson-prabu
Guys, I had the same issue. Looks like it's a resources problem.
I just increased resources for workers (CPU especially) and they are running now.
Duplicate here: #34

[Rusiecki]

@granescb I don't have any resource requests/limits. Still no luck.
To how much did you increase those, to get it working?