Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Confirm failed write to Redis are not ignored #657

Open
olevski opened this issue Jun 20, 2023 · 0 comments
Open

Confirm failed write to Redis are not ignored #657

olevski opened this issue Jun 20, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@olevski
Copy link
Member

olevski commented Jun 20, 2023

A recent incident on limited involved Redis being in a state where each replica of it seemed to be running independently. This lead to the ui-server and the gateway not being able to save their sessions in Redis.

But during this time we could only see failed login attempts and no keys in Redis. But there were no specific errors that indicated in the gateway logs that:

  1. The connection to redis was bad
  2. The writes to redis were bad

Therefore we should confirm that the gateway properly handles:

  • when redis is running correctly and then dies
  • when redis is not available as Renku and similar components are starting up
  • when writes to redis fail
@olevski olevski added the bug Something isn't working label Jun 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant