"Another scheduler is already running" when using docker-compose

[ThinkTransit]

Hi,

Great project!

When running in docker/docker-compose, shutting down the container doesn't remove the lock from the redis backend. This prevents the scheduler from running.

Another scheduler is already running.

  dramatiq_crontab:
    restart: unless-stopped
    platform: linux/x86_64
    image: ghcr.io/thinktransit/myproject:$VERSION
    env_file:
      - conf/dramatiq.env
      - conf/.env
    volumes:
      - ./data/web:/vol/web
    depends_on:
      - proxy
      - cache
      - app
    command: >
      sh -c "pipenv run python manage.py crontab"

Any ideas on how to prevent this.

Thanks

[codingjoe]

Hi @ThinkTransit 👋,

Thanks for reaching out. Yes, the lock is to more than one scheduler from running at the same time. However, if the scheduler doesn't shut down gracefully, the lock isn't released. How exactly do you shut down your docker container? If you use docker stop, docker will send a SIGTERM signal, required to release the log. If you use docker kill, the machine's scheduler will just kill the process without warning and no means of releasing the lock.

You can always drop the lock manually, should you ever need to.

I believe the only way to avoid this issue, is to set a timeout to the lock and to refresh the timeout every so often. If that is something you are interested in, feel free to open a ticket and pull-request.

Cheers!
Joe

[ThinkTransit]

Hi Joe,

Thanks for your reply and explanation regarding the lock. I have tested this and yes it seems to be an incorrect shutdown that is causing the lock to remain.

On consideration I think in the situation where crontab is running in it's own container, it's probably ok to not use the redis lock, because there will only ever be one container/command running.

Patrick