health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception

[djjudas21]

Background

I've been using django-health-check for ages, with success. I'm running in Kubernetes with persistent storage on TrueNAS, via democratic-csi. The persistent volumes are ReadWriteMany and formatted as ext4.

I've recently switched my persistent storage to Rook/Ceph, reprovisioned all my volumes and redeployed the app. The Ceph volumes are otherwise working (I can create files & dirs on them) but django-health-check seems unable to create dirs.

Logs

unavailable: Unknown exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
    file_name = self.check_save(file_name, file_content)
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
    file_name = storage.save(file_name, ContentFile(content=file_content))
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
    os.makedirs(directory, exist_ok=True)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
    raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/
[pid: 8|app: 0|req: 1/1] 192.168.0.58 () {30 vars in 363 bytes} [Sun Sep  3 18:57:44 2023] GET /health/ => generated 1767 bytes in 344 msecs (HTTP/1.1 500) 7 headers in 298 bytes (1 switches on core 0)
unavailable: Unknown exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
    file_name = self.check_save(file_name, file_content)
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
    file_name = storage.save(file_name, ContentFile(content=file_content))
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
    os.makedirs(directory, exist_ok=True)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
    raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/
[pid: 7|app: 0|req: 1/2] 192.168.0.58 () {30 vars in 363 bytes} [Sun Sep  3 18:58:00 2023] GET /health/ => generated 1767 bytes in 379 msecs (HTTP/1.1 500) 7 headers in 298 bytes (1 switches on core 0)
unavailable: Unknown exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 63, in check_status
    file_name = self.check_save(file_name, file_content)
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 42, in check_save
    file_name = storage.save(file_name, ContentFile(content=file_content))
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 54, in save
    name = self._save(name, content)
  File "/usr/local/lib/python3.10/site-packages/django/core/files/storage.py", line 260, in _save
    os.makedirs(directory, exist_ok=True)
  File "/usr/local/lib/python3.10/os.py", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/camerahub/media/health_check_storage_test'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/usr/local/lib/python3.10/site-packages/health_check/storage/backends.py", line 67, in check_status
    raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Internal Server Error: /health/

Debug

My app is running as a root container, permissions on the volume look fine and I can create a dir manually. The rest of the app deployment has the same permissions as the media subdir.

root@camerahub-dev-app-5bc6645cf8-7j46z:/media# ls -la
total 8
drwxr-xr-x 2 root root 4096 May 22 00:00 .
drwxr-xr-x 1 root root 4096 Sep  3 18:57 ..
root@camerahub-dev-app-5bc6645cf8-7j46z:/media# mkdir test
root@camerahub-dev-app-5bc6645cf8-7j46z:/media# ls -la
total 12
drwxr-xr-x 1 root root 4096 Sep  3 19:49 .
drwxr-xr-x 1 root root 4096 Sep  3 18:57 ..
drwxr-xr-x 2 root root 4096 Sep  3 19:49 test

I can't see any reason why the dir can't be created by django-health-check. Any clues?

[striveforbest]

Same issue with ElasticBeanstalk.

I think it should be customizable. No reason to write into the current dir, if it's a setting, can set it to /tmp for example.

(staging-LQM1lest) [ec2-user@ip-10-251-12-77 current]$ ./manage.py health_check
ERROR:health-check:unavailable: Unknown exception
Traceback (most recent call last):
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/health_check/storage/backends.py", line 63, in check_status
    file_name = self.check_save(file_name, file_content)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/health_check/storage/backends.py", line 42, in check_save
    file_name = storage.save(file_name, ContentFile(content=file_content))
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/django/core/files/storage/base.py", line 38, in save
    name = self._save(name, content)
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/django/core/files/storage/filesystem.py", line 106, in _save
    fd = os.open(full_path, self.OS_OPEN_FLAGS, 0o666)
PermissionError: [Errno 13] Permission denied: '/var/app/current/health_check_storage_test/test-9a3e7021-0071-4cb2-8d0e-f986799faad7.txt'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/health_check/backends.py", line 30, in run_check
    self.check_status()
  File "/var/app/venv/staging-LQM1lest/lib64/python3.8/site-packages/health_check/storage/backends.py", line 67, in check_status
    raise ServiceUnavailable("Unknown exception") from e
health_check.exceptions.ServiceUnavailable: unavailable: Unknown exception
Cache backend: default   ... working
DatabaseBackend          ... working
DefaultFileStorageHealthCheck ... unavailable: Unknown exception
DiskUsage                ... working
MemoryUsage              ... working
MigrationsHealthCheck    ... working
RedisHealthCheck         ... working