zarr.array from from an existing zarr.Array

[brokkoli71]

• fixes [v3] zarr.array from from an existing zarr.Array #2410

added concurrent streaming of source array into new array

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[brokkoli71]

Do we also want concurrency for different chunk sizes?

[normanrz]

Do we also want concurrency for different chunk sizes?

That would be nice, if the chunk sizes are somewhat compatible, i.e. one is a multiple of the other.

[d-v-b]

• (Is there some measure to prevent this that I am not aware of?)

if you are trying to write K input chunks into M output chunks, you can partition your K chunks into sets, where within each set elements can be written independently from all the other elements. then you write each set one after another. in the worst case scenario there will be 1 set per chunk, but you are guaranteed to avoid write collisions this way.

[d-v-b]

one question to answer here is what "auto" means for chunks if the user passes in a chunked array, but they want to use zarr-python's auto-chunking instead of the chunks that came with the array.

We might want to use a separate value that means "copy the chunks this object already has", which is distinct from "generate some chunks using the chunking heuristics". maybe something like ChunksLike: Literal['auto'] | Literal['keep'] | ShapeLike?

[brokkoli71]

one question to answer here is what "auto" means for chunks if the user passes in a chunked array, but they want to use zarr-python's auto-chunking instead of the chunks that came with the array.

Good point! I like the idea of distinguishing between keep and auto.

[codecov]

Codecov Report

Attention: Patch coverage is 96.82540% with 2 lines in your changes missing coverage. Please review.

Project coverage is 90.43%. Comparing base (eaf5d7a) to head (58f05fe).
Report is 431 commits behind head on main.

Files with missing lines	Patch %	Lines
src/zarr/core/array.py	96.42%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2622      +/-   ##
==========================================
- Coverage   99.98%   90.43%   -9.56%     
==========================================
  Files          38       59      +21     
  Lines       14718     6531    -8187     
==========================================
- Hits        14716     5906    -8810     
- Misses          2      625     +623     

Files with missing lines	Coverage Δ
src/zarr/__init__.py	100.00% <ø> (ø)
src/zarr/api/asynchronous.py	85.57% <100.00%> (ø)
src/zarr/api/synchronous.py	86.48% <100.00%> (ø)
src/zarr/core/array.py	93.82% <96.42%> (ø)

... and 93 files with indirect coverage changes

[d-v-b]

now that #2761 is in, could we use from_array inside create_array (after the data / dtype / shape validation)?

[brokkoli71]

now that #2761 is in, could we use from_array inside create_array (after the data / dtype / shape validation)?

@d-v-b Currently, in this PR from_array calls create_array. Is it redundant to have both from_array and create_array to create an array from another array? Or do you see a benefit in having both?

[d-v-b]

now that #2761 is in, could we use from_array inside create_array (after the data / dtype / shape validation)?

@d-v-b Currently, in this PR from_array calls create_array. Is it redundant to have both from_array and create_array to create an array from another array? Or do you see a benefit in having both?

create_array should call from_array if the user provided data; from_array should call the newly added init_array to persist the array metadata, and then store the array data.