Anndata format output

[rob-p]

Following up on COMBINE-lab/alevin-fry#151 raised by @ilan-gold, we are planning to first add AnnData output as an option in simpleaf, and then to potentially port it "upstream" directly into alevin-fry (possibly deprecating and dropping the old output format).

This commit 20ab78e includes the first draft implementation of this idea. It creates an anndata format version of the output quant.h5ad that lives directly alongside the existing CSR matrix. Creation of this matrix is enabled by passing --anndata-out as a flag to the simpleaf quant command. I may also add an additional flag to remove the CSR matrix as well to avoid the redundancy, but for now, they live side-by-side.

There are a few dependencies that I am waiting to stabilize (e.g. ureq 3.0 and a fix for this bug in flate2), but after that, we'll be ready to cut a simpleaf release with this feature!

P.S. @an-altosian / @DongzeHE : We should decide about the conversion in the case of visium data. Since we have extra information in the row labels, we should decide if we ignore that, put it in the anndata annotation, etc.

[DongzeHE]

I think we should definitely add the aux data into the anndata obs data. Also, we should make sure we save spliced, unspliced and ambiguous count matrices into three different layers.

[ilan-gold]

Yes! You should definitely take full advantage of the data model to store all the metadata, summary statistics, and quality info you can. Happy to hop on a call to run down with you where would be the best place for this stuff if you need.

[rob-p]

Agreed! @DongzeHE : the tip of dev currently properly splits the information into spliced/unspliced and ambiguous (if alevin-fry was run in USA mode). What other auxiliary data do you have in mind?

[an-altosian]

Mainly the optional aux information in the cell barcode file and the gene_id_to_name file. Currently there are nothing else in my mind that are used in the downstream analysis.

…

On Jan 14, 2025, at 9:27 PM, Rob Patro ***@***.***> wrote: Agreed! @DongzeHE <https://github.com/DongzeHE> : the tip of dev currently properly splits the information into spliced/unspliced and ambiguous (if alevin-fry was run in USA mode). What other auxiliary data do you have in mind? — Reply to this email directly, view it on GitHub <#177 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BI7FTBUWNY43D6VSW6TCEKD2KXWTDAVCNFSM6AAAAABU65BQQKVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOBTHA4TIMI>. You are receiving this because you were mentioned.

[ilan-gold]

(Also possible this stuff is easily inferrable from the matrix output and therefore not explicitly necessary). But FWIW scanpy has the calculate_qc_metrics function so it could be cool if this stuff came with the object already!

[rob-p]

Following our discussion this morning, I've added changes in af-anndata to add the quant.json and generate_permit_list.json contents as strings (encoding the JSON) to the uns component of the anndata object. Likewise, I've added all of the information from featureDump.txt as structured columns to obs.

@an-altosian : Any other metadata you think we should add? Take a look at the af-anndata repo. If you want spatial support for this in the initial release, then we should optionally add the spot locations to the obs.

[an-altosian]

Hi @ilan-gold , thank you very much for the suggestions! It will be amazing if we can include all the QC metrics you said.

@rob-p - Still, IMO we should definitely add the spatial coordinates in the cell barcode file and the gene id to name mapping info into obs and var.

[an-altosian]

Another QC metric I can think of is # total reads, # mapped reads, and the corresponding mapping rate.

[rob-p]

All of that information (not the id to name or coordinates, but all of the info in featureDump.txt are now in the obs component of the data frame).