General error/missing files

[ahamelers]

We've gotten a lot of "General error" "The file is not available for download" error report emails this month.

One of these emails concerns a dataset brought to my attention by curator Evangeline:

While downloading the files for DOI: 10.5061/dryad.6hdr7sr9m (22.64 GB) I'm receiving an error in VE as per the screenshot, can you please check it and let me know.

The system is unable to find file woodBetaCCR1.rds. I also cannot find this file anywhere. It does not have a permanent storage path and I must be looking in the wrong place for the temporary path. It has no digest. How was the dataset sent through processing without creating a permanent storage path for all files, and without a digest for this file? Is the file lost permanently?

<StashEngine::DataFile:0x00007f057f385b40
 id: 3664511,
 upload_file_name: "woodBetaCCR1.rds",
 upload_content_type: "",
 upload_file_size: 22604677308,
 resource_id: 330823,
 storage_version_id: nil,
 upload_updated_at: Mon, 25 Nov 2024 21:36:29.000000000 UTC +00:00,
 created_at: Mon, 25 Nov 2024 21:36:29.000000000 UTC +00:00,
 updated_at: Mon, 25 Nov 2024 21:36:29.000000000 UTC +00:00,
 validated_at: nil,
 file_state: "created",
 url: nil,
 status_code: nil,
 timed_out: false,
 original_url: nil,
 cloud_service: nil,
 digest: nil,
 digest_type: nil,
 description: nil,
 original_filename: "woodBetaCCR1.rds",
 type: "StashEngine::DataFile",
 compressed_try: 0>

I am concerned these other emails indicate a similar problem.

[ryscher]

This particular file is large. (22GB) and has not transferred from temporary storage into the permenant store. It is still in temporary storage at https://us-west-2.console.aws.amazon.com/s3/buckets/dryad-s3-prd?region=us-west-2&bucketType=general&prefix=330823/data/&showversions=false

[ahamelers]

Okay, since the files are not lost, I will reduce the priority of this ticket. I will copy this file to permanent storage. The question to answer then is why this dataset (and potentially others) moved out of 'processing' status before the files were fully transfered, and how we can stop that from happening in the future

[ahamelers]

I've been through around 2 months of these emails and found only 5 problematic (not removed on purpose) files.

These were the expected error type, mentioned in the email, and I copied them to where they were expected to be by our system, from other (later merritt version) locations within our buckets.

[ahamelers]

We should probably consider doing something to reduce the noise caused by the withdrawn/purposefully deleted files sending tons of these emails to our system and making it difficult to notice actual issues. Maybe when things are deleted as part of the withdrawal check, we can put a flag on each file itself, and then change the UI so they are not displayed with download links but have some kind of "we deleted this!" hover text