Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cleanup of tarball bucket to fix automated ingestion #185

Open
bedroge opened this issue Apr 1, 2024 · 2 comments
Open

cleanup of tarball bucket to fix automated ingestion #185

bedroge opened this issue Apr 1, 2024 · 2 comments

Comments

@bedroge
Copy link
Collaborator

bedroge commented Apr 1, 2024

We recently seem to have hit some kind of limit with the bucket: the automated ingestion would only pick up the first 500 tarballs it would find in the S3 bucket, meaning that new skylake tarballs would not be found (as they are picked up alphabetically, and skylake is at the bottom of the list). We can probably work around it by doing some kind of pagination, and we were considering reimplementing this workflow anyway, but for now I've done a quick workaround by setting up a new bucket (software.eessi.io-archive), syncing all files from the existing to the archive bucket, and removing them from the existing one.

We can do that more often by running the following:

export AWS_SECRET_ACCESS_KEY=XXX AWS_ACCESS_KEY_ID=YYY
aws s3 sync s3://software.eessi.io-2023.06 s3://software.eessi.io-archive
aws s3 rm  s3://software.eessi.io-2023.06 --recursive

This should only be run if there are no open PRs in the staging repo and if no tarballs were just uploaded to the bucket (i.e. no open PRs with bot: deploy label), otherwise they will be lost.

@bedroge
Copy link
Collaborator Author

bedroge commented Jun 7, 2024

As the number of tarballs was getting quite large again, I've just run this cleanup operation again.

@bedroge
Copy link
Collaborator Author

bedroge commented Sep 17, 2024

We were getting close to 1000 files in the bucket, so I've just moved all files to the archive bucket again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant