Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

slight data corruption in week: 11 Feb to 17 Feb? #5

Open
yestiseye opened this issue Feb 23, 2019 · 0 comments
Open

slight data corruption in week: 11 Feb to 17 Feb? #5

yestiseye opened this issue Feb 23, 2019 · 0 comments

Comments

@yestiseye
Copy link

Using the web_api.load_data() function I'm getting a crash as follows:

BIOMASS 2019-02-11 00:00:00 2019-02-18 00:00:00
index: 2016
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 4249, in create_block_manager_from_blocks
    mgr = BlockManager(blocks, axes)
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 2761, in __init__
    self._verify_integrity()
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 2971, in _verify_integrity
    construction_error(tot_items, block.shape[1:], self.axes)
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 4233, in construction_error
    passed, implied))
ValueError: Shape of passed values is (1, 2015), indices imply (1, 2016)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./getit.py", line 7, in <module>
    df_5, df_30 = web_api.load_data(d1 = datetime.datetime(2019,2,14), d2 = datetime.datetime(2019,2,15), region='qld1')
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 28, in load_data
    frame_list = [series for series in historical_week(d1=d1,d2=d2,region=region)]
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 28, in <listcomp>
    frame_list = [series for series in historical_week(d1=d1,d2=d2,region=region)]
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 23, in historical_week
    yield load_week(dt_i, region=region)
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 61, in load_week
    df = download_week(dt_start, region=region)
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 70, in download_week
    return 	merge_series(r)
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 74, in merge_series
    df = pd.concat([json_to_pd(data_set) for data_set in data_sets(r)], axis=1)	
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 74, in <listcomp>
    df = pd.concat([json_to_pd(data_set) for data_set in data_sets(r)], axis=1)	
  File "/home/dozu/whtr/data/opennempy/web_api.py", line 147, in json_to_pd
    return pd.DataFrame(index=index, data = series['data'], columns=[name])
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 321, in __init__
    copy=copy)
  File "/usr/lib/python3/dist-packages/pandas/core/frame.py", line 474, in _init_ndarray
    return create_block_manager_from_blocks([values], [columns, index])
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 4256, in create_block_manager_from_blocks
    construction_error(tot_items, blocks[0].shape[1:], axes, e)
  File "/usr/lib/python3/dist-packages/pandas/core/internals.py", line 4233, in construction_error
    passed, implied))
ValueError: Shape of passed values is (1, 2015), indices imply (1, 2016)

So potentially the data is short a single column? Similar issues with NSW, SA, and Tasmanian data.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant