Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CLOUD STORAGE not loading images #8905

Closed
2 tasks done
silvanobianchi opened this issue Jan 8, 2025 · 4 comments
Closed
2 tasks done

CLOUD STORAGE not loading images #8905

silvanobianchi opened this issue Jan 8, 2025 · 4 comments
Labels
need info Need more information to investigate the issue

Comments

@silvanobianchi
Copy link

Actions before raising this issue

  • I searched the existing issues and did not find anything similar.
  • I read/searched the docs

Steps to Reproduce

No response

Expected Behavior

No response

Possible Solution

No response

Context

Hello everyone,
the problem i have is about using cvat with a cloud storage, in my case the cloud storage is an amazon aws s3 bucket.

Premise:
I was able to map the storage through the dedicated interface and I can see the files that are inside it when I want to create a task.

The problem then arises after creating a task with the images selected from the cloud, CVAT only loads the first batch\chunk of N images (the ones that usually appear in blue in the top navigation bar between frames) while if I try to move between subsequent images CVAT keeps loading indefinitely.

I have tried:

  • adding or removing the manifest does not change the behavior
  • I tried splitting the task into N jobs using advanced configurations but what happens is that CVAT manages to load only the first job (the one with the lowest ID) and trying to open the other jobs I have the same infinite loading behavior

Did I do something wrong in the configuration? Do you know how to help me? Thanks in advance

Environment

No response
@silvanobianchi silvanobianchi added the bug Something isn't working label Jan 8, 2025
@azhavoro
Copy link
Contributor

azhavoro commented Jan 8, 2025

please attach logs from cvat_server and cvat_worker_chunks containers

@azhavoro azhavoro added need info Need more information to investigate the issue and removed bug Something isn't working labels Jan 8, 2025
@silvanobianchi
Copy link
Author

Hi, I don't have the cvat_worker_chunks container
The containers that I have are:
image

While the logs of the cvat_server container are the following (once requested the image in the not loaded chunk):

`2025-01-08 09:27:05,632 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:27:15,839 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:27:23,631 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:27:23,631] INFO cvat.apps.engine.cache: Starting to prepare chunk: key segment_20_chunk_2_0

2025-01-08 09:27:25,291 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:27:37,636 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:27:44,024 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:27:57,309 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:28:02,473 DEBG 'uvicorn-1' stdout output:
INFO: 10.130.13.62:0 - "POST /api/events?org=Vision HTTP/1.0" 201 Created

2025-01-08 09:28:10,001 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:28:13,540 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:28:13,540] WARNING django.request: Too Many Requests: /api/jobs/20/data

2025-01-08 09:28:13,540 DEBG 'uvicorn-1' stdout output:
INFO: 10.130.13.62:0 - "GET /api/jobs/20/data?org=Vision&quality=compressed&type=chunk&index=2 HTTP/1.0" 429 Too Many Requests

2025-01-08 09:28:18,657 DEBG 'uvicorn-0' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:28:23,715 DEBG 'uvicorn-0' stderr output:
[2025-01-08 09:28:23,715] INFO cvat.apps.engine.cache: Starting to prepare chunk: key segment_20_chunk_2_0

2025-01-08 09:28:28,797 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:28:37,974 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:28:49,136 DEBG 'uvicorn-0' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:29:02,560 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:29:11,733 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:29:13,625 DEBG 'uvicorn-0' stderr output:
[2025-01-08 09:29:13,625] WARNING django.request: Too Many Requests: /api/jobs/20/data

2025-01-08 09:29:13,625 DEBG 'uvicorn-0' stdout output:
INFO: 10.130.13.62:0 - "GET /api/jobs/20/data?org=Vision&quality=compressed&type=chunk&index=2 HTTP/1.0" 429 Too Many Requests

2025-01-08 09:29:23,800 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:29:23,800] INFO cvat.apps.engine.cache: Starting to prepare chunk: key segment_20_chunk_2_0

2025-01-08 09:29:24,218 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:29:33,857 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:29:45,854 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:30:00,379 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:30:06,845 DEBG 'uvicorn-0' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:30:13,709 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:30:13,709] WARNING django.request: Too Many Requests: /api/jobs/20/data

2025-01-08 09:30:13,709 DEBG 'uvicorn-1' stdout output:
INFO: 10.130.13.62:0 - "GET /api/jobs/20/data?org=Vision&quality=compressed&type=chunk&index=2 HTTP/1.0" 429 Too Many Requests

2025-01-08 09:30:19,570 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 200 OK

2025-01-08 09:30:23,875 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:30:23,875] INFO cvat.apps.engine.cache: Starting to prepare chunk: key segment_20_chunk_2_0

2025-01-08 09:30:30,349 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:30:45,229 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:30:56,854 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:31:08,263 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:31:13,797 DEBG 'uvicorn-1' stderr output:
[2025-01-08 09:31:13,796] WARNING django.request: Too Many Requests: /api/jobs/20/data

2025-01-08 09:31:13,797 DEBG 'uvicorn-1' stdout output:
INFO: 10.130.13.62:0 - "GET /api/jobs/20/data?org=Vision&quality=compressed&type=chunk&index=2 HTTP/1.0" 429 Too Many Requests

2025-01-08 09:31:19,576 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:31:27,548 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:31:36,073 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified

2025-01-08 09:31:49,982 DEBG 'uvicorn-1' stdout output:
INFO: 172.18.0.6:0 - "GET /api/auth/rules HTTP/1.0" 304 Not Modified`

@azhavoro
Copy link
Contributor

azhavoro commented Jan 8, 2025 •
edited
Loading

Hi, I don't have the cvat_worker_chunks container

This is exactly the cause of the issue - no worker processes to prepare the data.
It looks like you are using an old docker compose configuration with new images, I would suggest using latest release version instead of dev

@silvanobianchi
Copy link
Author

silvanobianchi commented Jan 9, 2025 •
edited
Loading

Switching to the newer version helped and now it's working!

Really thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
need info Need more information to investigate the issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@azhavoro @silvanobianchi