Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

iterate over chunks on dataset initialization #99

Closed
wants to merge 8 commits into from
Closed

iterate over chunks on dataset initialization #99

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
b9dc300
iterate over chunks on dataset initialization
jananzhu Jan 11, 2021
e22a368
Merge remote-tracking branch 'upstream/jreadey-work' into init-datase…
jananzhu Apr 4, 2021
8f9f1eb
Merge tag 'v0.9.2' of ssh://github.com/HDFGroup/h5pyd into init-datas…
jananzhu Jan 28, 2022
7d2fde8
Merge branch 'fancyindx' of ssh://github.com/HDFGroup/h5pyd into init…
jananzhu Feb 9, 2022
c48e5d4
Merge branch 'fancyindx' of ssh://github.com/HDFGroup/h5pyd into init…
jananzhu Feb 23, 2022
7d7e657
define dataset shape and dtype when data is passed
jananzhu Mar 9, 2022
263ccf0
update version
jananzhu Dec 20, 2022
5d81592
pin google-auth-oauthlib dependency
jananzhu Dec 20, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions h5pyd/_hl/group.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
from .table import Table
from .datatype import Datatype
from . import h5type
from h5pyd._apps.chunkiter import ChunkIterator


class Group(HLObject, MutableMappingHDF5):
Expand Down Expand Up @@ -358,14 +359,17 @@ def create_dataset(self, name, shape=None, dtype=None, data=None, **kwds):
if data is not None and not isinstance(data, Empty) and (numpy.product(shape) != numpy.product(data.shape)):
raise ValueError("Shape tuple is incompatible with data")
"""
dsid = dataset.make_new_dset(self, shape=shape, dtype=dtype, data=data, **kwds)
if data is not None and (shape is None or dtype is None):
shape = data.shape
dtype = data.dtype
dsid = dataset.make_new_dset(self, shape=shape, dtype=dtype, **kwds)
dset = dataset.Dataset(dsid)

"""
if data is not None:
self.log.info("initialize data")
dset[...] = data
"""
it = ChunkIterator(dset)
for chunk in it:
dset[chunk] = data[chunk]

if name is not None:
items = name.split('/')
Expand Down
6 changes: 3 additions & 3 deletions setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@
with open(path.join(here, 'README.rst'), encoding='utf-8') as f:
long_description = f.read()

setup(name='h5pyd',
version='0.10.0',
setup(name='h5pyd-redesign',
version='0.10.3',
description='h5py compatible client lib for HDF REST API',
long_description=long_description,
url='http://github.com/HDFGroup/h5pyd',
Expand All @@ -37,7 +37,7 @@
'msrestazure',
'cryptography',
'google-api-python-client',
'google-auth-oauthlib',
'google-auth-oauthlib<=0.5.3',
'google-auth<2.0dev',
'adal'],
setup_requires=['pkgconfig'],
Expand Down