Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Nomad OASIS extremly slow since V0.9.2 #520

Open
rettigl opened this issue Jan 11, 2025 · 1 comment
Open

[Bug]: Nomad OASIS extremly slow since V0.9.2 #520

rettigl opened this issue Jan 11, 2025 · 1 comment
Labels
bug Something isn't working

Comments

@rettigl
Copy link
Collaborator

rettigl commented Jan 11, 2025

Contact Details

[email protected]

What happened?

Since v0.9.2 of pynxtools, more specifically since merging of #474, the startup of our oasis is extremly slow, i.e. the startup of the "worker" container actually times out, and the oasis does not come up.
During startup a lot of nomad.processing workers are running and leading to 100% CPU consumption (one process per core):
grafik

Timeout after 10 Minutes:

nomad@pcr840:~/nomad-prod> docker-compose up -d
Creating network "nomad_oasis_network" with the default driver
Creating nomad_oasis_elastic  ... done
Creating nomad_oasis_north    ... done
Creating nomad_oasis_mongo    ... done
Creating nomad_oasis_rabbitmq ... done
Creating nomad_oasis_app      ... done
Creating nomad_oasis_worker   ... done

ERROR: for proxy  Container "0218eb7a29a5" is unhealthy.
ERROR: Encountered errors while bringing up the project.

The image used is this one:
ghcr.io/rettigl/oasis-fhi-pc-trarpes@sha256:94354887902138bc201b8b6a0b73f4d3cb6c9c7b70fbe147991354ab8fc00d56

Build from here:
7fc8655fc4d89313f0373ed3ca0bd04f4bcd0591

Relevant log output

No response

@rettigl rettigl added the bug Something isn't working label Jan 11, 2025
@rettigl
Copy link
Collaborator Author

rettigl commented Jan 12, 2025

This gets fixed by newer dev releases of nomad. We should reflect this in the requirements, so that a combination of packages giving this error cannot be installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant