updates to hash codes and file names #1553

silastittes · 2024-03-11T23:08:59Z

Added _v1 to annotation files on aws names as discussed in #1551.
Updated hash codes and files names in the respective annotation.py files. Tests passed locally — remote tests too, woo!

silastittes · 2024-03-11T23:11:46Z

Note, these are the new annotation files with the new hash codes, not the original ones. I have the originals but wasn't sure if we also wanted to change those file names.

codecov · 2024-03-11T23:18:11Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.86%. Comparing base (fbb94c2) to head (36e84ea).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1553   +/-   ##
=======================================
  Coverage   99.86%   99.86%           
=======================================
  Files         131      131           
  Lines        4333     4333           
  Branches      594      594           
=======================================
  Hits         4327     4327           
  Misses          3        3           
  Partials        3        3

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

petrelharp · 2024-03-12T08:32:14Z

Wow, remote tests, even - nice. Shouldn't there also be some changes to the maintenance script, though?

petrelharp · 2024-03-12T08:32:50Z

(Or if not - hm, and even if so - some documentation of how this works?)

silastittes · 2024-03-12T17:06:25Z

Yes to documentation. Should I write it in a way that applies to all files with a hash? Maybe a "how to update preexisting resources" or something?

petrelharp · 2024-03-12T19:35:31Z

Well, this needs updating at least:
https://popsim-consortium.github.io/stdpopsim-docs/latest/development.html#adding-a-genetic-map
Gee, suddenly I'm surprised we haven't encountered this problem with genetic maps?

silastittes · 2024-03-13T20:27:38Z

docs/development.rst

+be sure to give it a new version name (i.e. ``name_{id}_more_name_v1.txt``). 
+This is to ensure that the previous file versions and checksums that are hosted
+remotely remain valid when new releases are made.
+


Is this sufficient and what you had in mind @petrelharp?

Not quite. Can you add, let's see:

in what files do you need to put the new name

something more prescriptive about how to create the new name (your example is not following how you created the new names for this round - and, they are .tar.gz, not .txt, right?)

a note that in the cache the files are renamed to a standard thing

something saying why you have to do this?

Updates! Better?

oh I see what you mean about naming -- I didn't mean to add the _v1 before the exons and CDS. Would it be better to change the files names to match what I describe here? That seems a bit cleaner than having the version number in the middle.

Went ahead with updating names to match what I described in docs.

Sounds good - that made more sense to me, but I assumed you had a good reason for doing it the other way.

petrelharp · 2024-03-14T19:53:18Z

docs/development.rst

@@ -1337,7 +1337,7 @@ see `Getting set up to add a new species`_):
        long_description="FILL_ME",
        url=("https://stdpopsim.s3-us-west-2.amazonaws.com/genetic_maps/dir/filename"),
        sha256="FILL_ME",
-        file_pattern="name_{id}_more_name.txt",
+        file_pattern="name_{id}_more_name.tar.gz",


I'm confused - up above it says that {id} should be the ID of a given chromosome, but we don't .tar.gz up individual chromosomes - all the chromosomes are tar'ed up together? This seems inconsistent?

petrelharp · 2024-03-14T19:55:15Z

docs/development.rst

+when an existing resource file (such as a genetic map or annotation)
+is updated and replaces the previous version,
+be sure to update the name of the file pattern with a version number
+(i.e. ``name_{id}_more_name_v1.tar.gz``).


This seems confused - either {id} should not go here (because as above it refers to an individual chromosome, and we don't rename the individual chromosome files?) or else this is the wrong thing that needs to be changed (ie we don't change the file_pattern)?

Yeah, I think this is wrong. Note below in your PR; you are not changing file_pattern; you're changing like intervals_url. Here you should be documenting/explaining the very process you're doing in this PR.

Yes sorry, thanks for looking carefully!

petrelharp · 2024-03-14T19:57:38Z

tests/test_ensembl.py

@@ -4,4 +4,4 @@
 # Make sure we don't update the release without realising it.
 def test_version():
    release = stdpopsim.catalog.ensembl_info.release
-    assert release == 103
+    assert release == 111


Ack, this thing. See #1521

Please don't change this here.

Ok, so best to just let this test fail until the solution is worked out? I tried for follow the threads and it seems like the PR #1536 is still open?

Yes, you should definitely not change this until this is actually fixed. (However, that'd be a great thing to fix next...)

docs/development.rst

petrelharp · 2024-04-02T16:13:57Z

stdpopsim/catalog/ensembl_info.py

@@ -1,2 +1,2 @@
 # File autogenerated from Ensembl REST API. Do not edit.
-release = 103
+release = 111


please don't update this

stdpopsim/catalog/ensembl_info.py

petrelharp · 2024-04-02T16:48:45Z

I've edited the docs - please read and see if it makes sense, @silastittes .

And, I verified the only failing tests have to do with the ensembl build.

So, I think we can merge this. But: don't we need to do something to the maintenance script? I don't think that the annotation script has to automatically do the version numbers, but it should at least remind people to do this? Or something? I have not digested that side of things - I'm hoping you have the big picture here, @silastittes.

silastittes · 2024-04-02T18:14:41Z

These updates look good to me! My current understanding of the problem did not require any changes to the maintenance script. I used @andrewkern's local aws_push_annotations.py to get the old and new versions of the annotation files onto AWS with the correct names. I think as long as developers update the file names when revisions are submitted, that fixes the issue? Automatic updates in the maintenance script or elsewhere seems hard to implement and error prone, but happy to admit I may not have the whole picture either. @andrewkern, thoughts?

petrelharp · 2024-04-03T04:37:53Z

I made some adjustments to the maintenance script. Please see what you think.

silastittes · 2024-04-03T16:33:22Z

maintenance/annotation_maint.py

+                logger.info(
+                    "ALERT: need to rename the files (replace 'vX' with "
+                    "appropriate version number to not clobber existing files "
+                    "before upload."
+                )


Suggested change

logger.info(

"ALERT: need to rename the files (replace 'vX' with "

"appropriate version number to not clobber existing files "

"before upload."

)

logger.warning(

"ALERT: need to rename the files (replace 'vX' with "

"appropriate version number to not clobber existing files "

"before upload."

)

Ran locally, found it easy to ignore. Worth making a warning or some other more notable flag? Not sure what best practices dictates here.

Well, the filename itself has _vX in it, so they should notice that? Hm, maybe instead it should be _vCHANGEME?

maintenance/annotation_maint.py

petrelharp · 2024-04-04T04:00:41Z

Could you run the maint script again locally to check these changes work?

silastittes · 2024-04-04T19:01:18Z

Seems to be working to me!

ls annotations/*/*vCHANGEME.tar.gz
annotations/AraTha/araport_11_CDS_vCHANGEME.tar.gz
annotations/AraTha/araport_11_exons_vCHANGEME.tar.gz
annotations/DroMel/FlyBase_BDGP6.32.51_CDS_vCHANGEME.tar.gz
annotations/DroMel/FlyBase_BDGP6.32.51_exons_vCHANGEME.tar.gz
annotations/HomSap/ensembl_havana_104_CDS_vCHANGEME.tar.gz
annotations/HomSap/ensembl_havana_104_exons_vCHANGEME.tar.gz
annotations/PhoSin/Phocoena_sinus.mPhoSin1.pri.110_CDS_vCHANGEME.tar.gz
annotations/PhoSin/Phocoena_sinus.mPhoSin1.pri.110_exons_vCHANGEME.tar.gz

updates to hash codes and file names

771d426

note to dcs about versioning resource files.

e74ee7f

silastittes commented Mar 13, 2024

View reviewed changes

silastittes added 3 commits March 13, 2024 13:29

wtrailing hitespace

07267f9

edits to versioning note.

a3f8a3a

chaanged file names to match convention described in docs

a96f4b7

petrelharp reviewed Mar 14, 2024

View reviewed changes

docs/development.rst Outdated Show resolved Hide resolved

petrelharp reviewed Mar 14, 2024

View reviewed changes

docs/development.rst Outdated Show resolved Hide resolved

rfurther revisions to docs, reset ensemble version

5893d7f

petrelharp reviewed Apr 2, 2024

View reviewed changes

stdpopsim/catalog/ensembl_info.py Outdated Show resolved Hide resolved

more explanation and ensembl build fix

85b3093

docfix

1999063

adjust annotation maintenance script

d71b547

petrelharp mentioned this pull request Apr 3, 2024

remove custom log message editing (closes #1545) #1550

Merged

silastittes commented Apr 3, 2024

View reviewed changes

petrelharp reviewed Apr 4, 2024

View reviewed changes

maintenance/annotation_maint.py Outdated Show resolved Hide resolved

petrelharp reviewed Apr 4, 2024

View reviewed changes

maintenance/annotation_maint.py Outdated Show resolved Hide resolved

more obvious filename

327c96a

petrelharp reviewed Apr 4, 2024

View reviewed changes

maintenance/annotation_maint.py Outdated Show resolved Hide resolved

silas suggestion

36e84ea

petrelharp merged commit 6926072 into popsim-consortium:main Apr 5, 2024
11 checks passed

petrelharp mentioned this pull request Jul 2, 2024

cached files need python version information in filenames #1551

Closed

petrelharp mentioned this pull request Jan 9, 2025

Add assembly name to standard naming for annotation #1252

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

updates to hash codes and file names #1553

updates to hash codes and file names #1553

silastittes commented Mar 11, 2024 •

edited

Loading

silastittes commented Mar 11, 2024

codecov bot commented Mar 11, 2024 •

edited

Loading

petrelharp commented Mar 12, 2024

petrelharp commented Mar 12, 2024

silastittes commented Mar 12, 2024

petrelharp commented Mar 12, 2024

silastittes Mar 13, 2024

petrelharp Mar 14, 2024

silastittes Mar 14, 2024

silastittes Mar 14, 2024

silastittes Mar 14, 2024

petrelharp Mar 14, 2024

petrelharp Mar 14, 2024

petrelharp Mar 14, 2024

petrelharp Mar 14, 2024

silastittes Mar 15, 2024

petrelharp Mar 14, 2024

petrelharp Mar 14, 2024

silastittes Mar 15, 2024

petrelharp Mar 16, 2024

petrelharp Apr 2, 2024

petrelharp commented Apr 2, 2024

silastittes commented Apr 2, 2024 •

edited

Loading

petrelharp commented Apr 3, 2024

silastittes Apr 3, 2024

silastittes Apr 3, 2024

petrelharp Apr 4, 2024

petrelharp Apr 4, 2024

petrelharp commented Apr 4, 2024

silastittes commented Apr 4, 2024

updates to hash codes and file names #1553

updates to hash codes and file names #1553

Conversation

silastittes commented Mar 11, 2024 • edited Loading

silastittes commented Mar 11, 2024

codecov bot commented Mar 11, 2024 • edited Loading

Codecov Report

petrelharp commented Mar 12, 2024

petrelharp commented Mar 12, 2024

silastittes commented Mar 12, 2024

petrelharp commented Mar 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrelharp commented Apr 2, 2024

silastittes commented Apr 2, 2024 • edited Loading

petrelharp commented Apr 3, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petrelharp commented Apr 4, 2024

silastittes commented Apr 4, 2024

silastittes commented Mar 11, 2024 •

edited

Loading

codecov bot commented Mar 11, 2024 •

edited

Loading

silastittes commented Apr 2, 2024 •

edited

Loading