Code

Append to icechunk stores#272

Code

Merged

abarciauskas-bgse merged 101 commits intomainzarr-developers/VirtualiZarr:mainfrom icechunk-appendzarr-developers/VirtualiZarr:icechunk-appendCopy head branch name to clipboardDec 5, 2024

+2,412-178

Commits

Commits on Oct 25, 2024

• Initial attempt at appending

  abarciauskas-bgse

  committed
  d3a4048

  Copy full SHA for d3a4048

• Working on tests for generate chunk key function

  abarciauskas-bgse

  committed
  5d5f9e2

  Copy full SHA for 5d5f9e2

Commits on Oct 26, 2024

• Linting

  abarciauskas-bgse

  committed
  360ea14

  Copy full SHA for 360ea14

• Refactor gen virtual dataset method

  abarciauskas-bgse

  committed
  d3c2851

  Copy full SHA for d3c2851

Commits on Oct 27, 2024

• Fix spelling

  abarciauskas-bgse

  committed
  a7a1e50

  Copy full SHA for a7a1e50

Commits on Oct 28, 2024

• Linting

  abarciauskas-bgse

  committed
  0365a45

  Copy full SHA for 0365a45

• Linting

  abarciauskas-bgse

  committed
  5846d7e

  Copy full SHA for 5846d7e

Commits on Oct 30, 2024

• Linting

  abarciauskas-bgse

  committed
  66bbd6e

  Copy full SHA for 66bbd6e

Commits on Nov 1, 2024

• Passing compression test

  abarciauskas-bgse

  committed
  000c68f

  Copy full SHA for 000c68f

Commits on Nov 5, 2024

• Merge branch 'main' into icechunk-append

  TomNicholas

  authored
  3131167

  Copy full SHA for 3131167

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 5906687
  pre-commit-ci[bot]

  committed
  5906687

  Copy full SHA for 5906687

Commits on Nov 10, 2024

• merge main and linting

  abarciauskas-bgse

  committed
  80e4dcb

  Copy full SHA for 80e4dcb

• linting

  abarciauskas-bgse

  committed
  672e5e1

  Copy full SHA for 672e5e1

• Fix test failing due to incorrect dtype

  abarciauskas-bgse

  committed
  0fce71f

  Copy full SHA for 0fce71f

• Fix conflict in conftest

  abarciauskas-bgse

  committed
  41f80f8

  Copy full SHA for 41f80f8

• linting

  abarciauskas-bgse

  committed
  e60437f

  Copy full SHA for e60437f

• Linting

  abarciauskas-bgse

  committed
  de2f135

  Copy full SHA for de2f135

• Remove obsolete test file for appending

  abarciauskas-bgse

  committed
  c8a46a6

  Copy full SHA for c8a46a6

Commits on Nov 12, 2024

• Create netcdf4 files factor in conftest

  abarciauskas-bgse

  committed
  7663ad7

  Copy full SHA for 7663ad7

• Linting

  abarciauskas-bgse

  committed
  1d704ff

  Copy full SHA for 1d704ff

• Refactor to use combineable zarr arrays

  abarciauskas-bgse

  committed
  af5f57d

  Copy full SHA for af5f57d

• linting

  abarciauskas-bgse

  committed
  6f4cfd9

  Copy full SHA for 6f4cfd9

• Implement no append dim test

  abarciauskas-bgse

  committed
  98c7052

  Copy full SHA for 98c7052

• Add test for when append dim is not in dims

  abarciauskas-bgse

  committed
  f36adf2

  Copy full SHA for f36adf2

Commits on Nov 15, 2024

• Fix mypy errors

  abarciauskas-bgse

  committed
  e922ccd

  Copy full SHA for e922ccd

• type ignore import untyped zarr

  abarciauskas-bgse

  committed
  ca80cb2

  Copy full SHA for ca80cb2

• Use Union type for check_combineable_zarr_arrays arg

  abarciauskas-bgse

  committed
  f186d4f

  Copy full SHA for f186d4f

• Fix import

  abarciauskas-bgse

  committed
  2a90d9c

  Copy full SHA for 2a90d9c

• Fix imports for get_codecs

  abarciauskas-bgse

  committed
  0253a8a

  Copy full SHA for 0253a8a

• use new factory in test

  abarciauskas-bgse

  committed
  7369fcf

  Copy full SHA for 7369fcf

• Remove need for dask in fixture

  abarciauskas-bgse

  committed
  2949493

  Copy full SHA for 2949493

• Fix for when zarr is not installed

  abarciauskas-bgse

  committed
  c305dad

  Copy full SHA for c305dad

Commits on Nov 16, 2024

• Address test failures

  abarciauskas-bgse

  committed
  aaede73

  Copy full SHA for aaede73

• Add get_codecs file

  abarciauskas-bgse

  committed
  5d685c6

  Copy full SHA for 5d685c6

• Add dask to upstream

  abarciauskas-bgse

  committed
  d704de2

  Copy full SHA for d704de2

• Merge branch 'main' into icechunk-append

  abarciauskas-bgse

  authored
  29ca87c

  Copy full SHA for 29ca87c

Commits on Nov 17, 2024

• Remove dependency on dask and h5netcdf engine

  abarciauskas-bgse

  committed
  d84c58c

  Copy full SHA for d84c58c

• Merge branch 'icechunk-append' of github.com:zarr-developers/virtualizarr into icechunk-append

  abarciauskas-bgse

  committed
  f19e3d1

  Copy full SHA for f19e3d1

Commits on Nov 18, 2024

• Remove obsolete comment

  abarciauskas-bgse

  committed
  2505d9e

  Copy full SHA for 2505d9e

• Remove duplicate zarr array type check

  abarciauskas-bgse

  committed
  aaa7f01

  Copy full SHA for aaa7f01

Commits on Nov 19, 2024

• Move codecs module and type output

  abarciauskas-bgse

  committed
  5071ed7

  Copy full SHA for 5071ed7

• Actually add codecs file

  abarciauskas-bgse

  committed
  7067b43

  Copy full SHA for 7067b43

• Merge branch 'main' into icechunk-append

  abarciauskas-bgse

  authored
  f867e14

  Copy full SHA for f867e14

• Fix merge mistake

  abarciauskas-bgse

  committed
  0ec5084

  Copy full SHA for 0ec5084

• Ignore import untyped

  abarciauskas-bgse

  committed
  5630b34

  Copy full SHA for 5630b34

Commits on Nov 20, 2024

• Merge branch 'main' into icechunk-append

  abarciauskas-bgse

  authored
  2fd7fea

  Copy full SHA for 2fd7fea

• Add tests for codecs

  abarciauskas-bgse

  committed
  d93f2ce

  Copy full SHA for d93f2ce

• Merge branch 'icechunk-append' of github.com:zarr-developers/virtualizarr into icechunk-append

  abarciauskas-bgse

  committed
  b21daca

  Copy full SHA for b21daca

• Resolve mypy errors

  abarciauskas-bgse

  committed
  a6c2ccb

  Copy full SHA for a6c2ccb

• Fix test

  abarciauskas-bgse

  committed
  98a676d

  Copy full SHA for 98a676d

• Import zarr in function

  abarciauskas-bgse

  committed
  db3313f

  Copy full SHA for db3313f

• Use existing importorskip function

  abarciauskas-bgse

  committed
  145ed0e

  Copy full SHA for 145ed0e

• Modify comments

  abarciauskas-bgse

  committed
  80cd358

  Copy full SHA for 80cd358

• Comment updates and spelling of combinable

  abarciauskas-bgse

  committed
  3035f05

  Copy full SHA for 3035f05

• Revert change to check compatible encoding

  abarciauskas-bgse

  committed
  28e05db

  Copy full SHA for 28e05db

• Ignore zarr untyped import errors

  abarciauskas-bgse

  committed
  2c07cdf

  Copy full SHA for 2c07cdf

Commits on Nov 21, 2024

• Implement a manifest.utils module

  abarciauskas-bgse

  committed
  19837b2

  Copy full SHA for 19837b2

• pass the array into resize_array

  Show description for 532ff38
  

  abarciauskas-bgse

  and

  TomNicholas

  authored
  532ff38

  Copy full SHA for 532ff38

• Refactor resize_array

  abarciauskas-bgse

  committed
  24f7274

  Copy full SHA for 24f7274

• Remove unnecessary zarr imports

  abarciauskas-bgse

  committed
  113cd2c

  Copy full SHA for 113cd2c

• Add pinned version of icechunk as an optional dependency

  abarciauskas-bgse

  committed
  61ce01a

  Copy full SHA for 61ce01a

Commits on Nov 22, 2024

• Add append_dim in docstring

  abarciauskas-bgse

  committed
  defe7d9

  Copy full SHA for defe7d9

• Kludgy solution to v2 v3 codecs difference

  abarciauskas-bgse

  committed
  cb82d40

  Copy full SHA for cb82d40

• Add normalize to v3 parameter

  abarciauskas-bgse

  committed
  2f6cbc2

  Copy full SHA for 2f6cbc2

• Add more info to docstring

  abarciauskas-bgse

  committed
  a442fa4

  Copy full SHA for a442fa4

• Fix typing issues

  abarciauskas-bgse

  committed
  7d1bb36

  Copy full SHA for 7d1bb36

• Add decorator for zarr python v3 test

  abarciauskas-bgse

  committed
  39677e8

  Copy full SHA for 39677e8

• Fix mypy and ruff errors

  abarciauskas-bgse

  committed
  5fa7177

  Copy full SHA for 5fa7177

Commits on Nov 23, 2024

• Only append if append_dim in dims

  abarciauskas-bgse

  committed
  e109c0d

  Copy full SHA for e109c0d

Commits on Nov 25, 2024

• Add example notebook

  abarciauskas-bgse

  committed
  eb0e8f2

  Copy full SHA for eb0e8f2

• Add a runtime

  abarciauskas-bgse

  committed
  fd2df4e

  Copy full SHA for fd2df4e

• Add failing test

  abarciauskas-bgse

  committed
  1659d21

  Copy full SHA for 1659d21

• Fix multiple appends

  abarciauskas-bgse

  committed
  f5976d1

  Copy full SHA for f5976d1

• Fix test error message

  abarciauskas-bgse

  committed
  f903291

  Copy full SHA for f903291

Commits on Nov 26, 2024

• Add new cell to notebook to display original time chunk

  abarciauskas-bgse

  committed
  c109626

  Copy full SHA for c109626

• Upgrade icechunk to 1.0.0a5

  abarciauskas-bgse

  committed
  dd9c381

  Copy full SHA for dd9c381

• Upgrade icechunk in upstream.yml

  abarciauskas-bgse

  committed
  7c1fcfa

  Copy full SHA for 7c1fcfa

• Updated notebook with kechunk comment an upgraded icechunk version

  abarciauskas-bgse

  committed
  1bb2ad0

  Copy full SHA for 1bb2ad0

Commits on Dec 4, 2024

• Modify test so it fails without updated icechunk

  abarciauskas-bgse

  committed
  64f2478

  Copy full SHA for 64f2478

• Update icechunk dependency

  abarciauskas-bgse

  committed
  13de8d3

  Copy full SHA for 13de8d3

• Fix mypy errors

  abarciauskas-bgse

  committed
  4a65e5a

  Copy full SHA for 4a65e5a

• update icechunk version in pyproject

  abarciauskas-bgse

  committed
  64e5277

  Copy full SHA for 64e5277

• Merge branch 'main' into icechunk-append

  abarciauskas-bgse

  authored
  d6ef97f

  Copy full SHA for d6ef97f

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 9d2f7f8
  pre-commit-ci[bot]

  committed
  9d2f7f8

  Copy full SHA for 9d2f7f8

• Remove obsolete comment

  abarciauskas-bgse

  committed
  53045c3

  Copy full SHA for 53045c3

Commits on Dec 5, 2024

• Use icechunk 0.1.0a7

  abarciauskas-bgse

  committed
  b21a0e5

  Copy full SHA for b21a0e5

• Updated notebook

  abarciauskas-bgse

  committed
  208e83e

  Copy full SHA for 208e83e

• Updated notebook

  abarciauskas-bgse

  committed
  79e0c1b

  Copy full SHA for 79e0c1b

• print store

  abarciauskas-bgse

  committed
  e38823c

  Copy full SHA for e38823c

• Update notebook (#327)

  Show description for ad17b83
  

  mpiannucci

  and

  abarciauskas-bgse

  authored
  ad17b83

  Copy full SHA for ad17b83

• Add append to examples

  abarciauskas-bgse

  committed
  8b9a830

  Copy full SHA for 8b9a830

• Add to releases.rst

  abarciauskas-bgse

  committed
  3f9f58c

  Copy full SHA for 3f9f58c

• Revert change to .gitignore

  abarciauskas-bgse

  committed
  8496359

  Copy full SHA for 8496359

• Merge branch 'main' into icechunk-append

  TomNicholas

  authored
  7dc9186

  Copy full SHA for 7dc9186

• Update ci/upstream.yml

  Show description for 491b701
  

  abarciauskas-bgse

  and

  TomNicholas

  authored
  491b701

  Copy full SHA for 491b701

• Update pyproject.toml

  Show description for 94ef469
  

  abarciauskas-bgse

  and

  TomNicholas

  authored
  94ef469

  Copy full SHA for 94ef469

• Update virtualizarr/tests/test_writers/test_icechunk.py

  Show description for fad188b
  

  abarciauskas-bgse

  and

  TomNicholas

  authored
  fad188b

  Copy full SHA for fad188b

• [pre-commit.ci] auto fixes from pre-commit.com hooks

  Show description for 8df67e9
  pre-commit-ci[bot]

  committed
  8df67e9

  Copy full SHA for 8df67e9

• Update virtualizarr/accessor.py

  Show description for 258c92f
  

  abarciauskas-bgse

  and

  TomNicholas

  authored
  258c92f

  Copy full SHA for 258c92f

• Separate out multiple arrays test

  abarciauskas-bgse

  committed
  84a4d01

  Copy full SHA for 84a4d01

• Merge branch 'icechunk-append' of github.com:zarr-developers/virtualizarr into icechunk-append

  abarciauskas-bgse

  committed
  299f580

  Copy full SHA for 299f580