Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CURVAS dataset #75

Merged
merged 10 commits into from
Oct 4, 2024
Merged

CURVAS dataset #75

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
aa359d1
added CURVAS dataset interface
Oct 3, 2024
85ce9ea
simplified
Oct 3, 2024
5a28c0d
Merge branch 'master' into curvas
Oct 3, 2024
2c1674e
reformatted
Oct 3, 2024
5e7da76
final touch
Oct 3, 2024
7047767
docs
Oct 3, 2024
64c3db3
removed populate from docs
Oct 3, 2024
d89a258
space-aware datatypes
Oct 4, 2024
bfa11f5
oi innit bruv
Oct 4, 2024
0fffbf0
order
Oct 4, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ print(entry.split, entry.patient)
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.cl_detection.CLDetection2023">CLDetection2023</a> | 400 | Head | X-ray |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.crlm.CRLM">CRLM</a> | 197 | Abdomen | CT, SEG |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.ct_ich.CT_ICH">CT_ICH</a> | 75 | Head | CT |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.curvas.CURVAS">CURVAS</a> | 90 | Abdomen | CT |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.crossmoda.CrossMoDA">CrossMoDA</a> | 484 | Head | MRI T1c, MRI T2hr |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.deeplesion.DeepLesion">DeepLesion</a> | 20094 | Abdomen, Thorax | CT |
| <a href="https://neuro-ml.github.io/amid/latest/datasets-api/#amid.egd.EGD">EGD</a> | 3096 | Head | FLAIR, MRI T1, MRI T1GD, MRI T2 |
Expand Down
106 changes: 106 additions & 0 deletions amid/curvas.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
import gzip
import zipfile
from typing import Dict
from zipfile import ZipFile

import nibabel
import numpy as np

from .internals import Dataset, field, licenses, register


@register(
body_region='Abdomen',
license=licenses.CC_BY_40,
link='https://zenodo.org/records/13767408',
modality='CT',
prep_data_size='30G',
raw_data_size='30G',
task='Abdominal organ pathologies segmentation',
)
class CURVAS(Dataset):
"""
Pancreas, liver and kidney cysts segmentation from multi-rater annotated data.

The dataset was used at the MICCAI 2024 CURVAS challenge.

Parameters
----------
root : str, Path, optional
path to the folder containing the raw downloaded archives.
If not provided, the cache is assumed to be already populated.

Notes
-----
Download link: https://zenodo.org/records/13767408

The `root` folder should contain the three downloaded .zip archives, namely:
`training_set.zip`, `validation_set.zip` and `testing_set.zip`.

Examples
--------
>>> # Place the downloaded folders in any folder and pass the path to the constructor:
>>> ds = CURVAS(root='/path/to/downloaded/data/folder/')
>>> print(len(ds.ids))
# 90
>>> print(ds.image(ds.ids[5]).shape)
# (512, 512, 1045)
>>> print(ds.mask(ds.ids[35]).shape)
# (512, 512, 992)

"""

@property
def ids(self):
def _extract(split):
archive = self.root / f'{split}_set.zip'
with ZipFile(archive) as zf:
namelist = [x for x in zf.namelist() if len(x.rstrip('/').split('/')) == 2]
ids = [f'{x.split("/")[1]}-{split}' for x in namelist]
return ids

return sorted(
[
*_extract('training'), # 20 Training cases
*_extract('validation'), # 5 Validation cases
*_extract('testing'), # 65 Testing cases
]
)

def _file(self, i, obj):
uid, split = i.split('-')

archive = self.root / f'{split}_set.zip'
file = f'{split}_set/{uid}/{obj}.nii.gz'

return zipfile.Path(archive, file)

@field
def image(self, i) -> np.ndarray:
with self._file(i, 'image').open('rb') as opened:
with gzip.GzipFile(fileobj=opened) as nii:
nii = nibabel.FileHolder(fileobj=nii)
image = nibabel.Nifti1Image.from_file_map({'header': nii, 'image': nii})
return np.asarray(image.dataobj).astype(np.int16)

@field
def affine(self, i) -> np.ndarray:
"""The 4x4 matrix that gives the image's spatial orientation"""
with self._file(i, 'image').open('rb') as opened:
with gzip.GzipFile(fileobj=opened) as nii:
nii = nibabel.FileHolder(fileobj=nii)
image = nibabel.Nifti1Image.from_file_map({'header': nii, 'image': nii})
return image.affine

@field
def masks(self, i) -> Dict[str, np.ndarray]:
masks = {}
for x in range(1, 4):
with self._file(i, f'annotation_{x}').open('rb') as opened:
with gzip.GzipFile(fileobj=opened) as nii:
nii = nibabel.FileHolder(fileobj=nii)
image = nibabel.Nifti1Image.from_file_map({'header': nii, 'image': nii})

masks[f'annotation_{x}'] = np.asarray(image.dataobj).astype(np.uint8)

return masks
20 changes: 2 additions & 18 deletions docs/CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -101,25 +101,9 @@ id_ = dataset.ids[0]
print(dataset.image(id_).shape)
```

6\. Populate the dataset:

```shell
amid populate LiTS /shared/data/LiTS
```

!!! tip
Use the option `--n-jobs` to speed up the process.

!!! tip
Use the option `--help` for a more detailed information on this command.

7\. If there is no error, the file `amid/data/lits.hash` will appear (the name depends on `short_name` given to `normalize`).

8\. Check the codestyle using the `lint.sh` script in the repository's root and make changes if flake8 is not happy:
6\. Check the codestyle using the `lint.sh` script in the repository's root and make changes if flake8 is not happy:

```shell
pip install -r lint-requirements.txt # only for the first time
./lint.sh
```

9\. Commit all the files you added, including the `*.hash` one.
```
2 changes: 2 additions & 0 deletions docs/datasets-api.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@

::: amid.ct_ich.CT_ICH

::: amid.curvas.CURVAS

::: amid.crossmoda.CrossMoDA

::: amid.deeplesion.DeepLesion
Expand Down
1 change: 1 addition & 0 deletions docs/datasets.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.cl_detection.CLDetection2023">CLDetection2023</a> | 400 | Head | <a href="https://creativecommons.org/licenses/by-nc/4.0/">CC BY-NC 4.0</a> | X-ray | 1.8G | 1.5G | Keypoint detection | <a href="https://github.com/cwwang1979/CL-detection2023/">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.crlm.CRLM">CRLM</a> | 197 | Abdomen | <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a> | CT, SEG | 11G | 11G | Segmentation, Classification | <a href="https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=89096268#89096268412b832037484784bd78caf58e052641">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.ct_ich.CT_ICH">CT_ICH</a> | 75 | Head | <a href="https://www.physionet.org/about/licenses/physionet-restricted-health-data-license-150/">PhysioNet Restricted Health Data License 1.5.0</a> | CT | 661M | 2,8G | Intracranial hemorrhage segmentation | <a href="https://physionet.org/content/ct-ich/1.3.1/">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.curvas.CURVAS">CURVAS</a> | 90 | Abdomen | <a href="https://creativecommons.org/licenses/by/4.0/">CC BY 4.0</a> | CT | 30G | 30G | Abdominal organ pathologies segmentation | <a href="https://zenodo.org/records/13767408">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.crossmoda.CrossMoDA">CrossMoDA</a> | 484 | Head | <a href="https://creativecommons.org/licenses/by-nc-sa/4.0/">CC BY-NC-SA 4.0</a> | MRI T1c, MRI T2hr | 8,96G | 17G | Segmentation, Classification, Domain Adaptation | <a href="https://zenodo.org/record/6504722#.YsgwnNJByV4">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.deeplesion.DeepLesion">DeepLesion</a> | 20094 | Abdomen, Thorax | | CT | 259G | 259G | Localisation, Detection, Classification | <a href="https://nihcc.app.box.com/v/DeepLesion">Source</a> |
| <a href="https://neuro-ml.github.io/amid/0.14.0/datasets-api/#amid.egd.EGD">EGD</a> | 3096 | Head | EGD data license | FLAIR, MRI T1, MRI T1GD, MRI T2 | 107,49G | 40G | Segmentation | <a href="https://xnat.bmia.nl/data/archive/projects/egd">Source</a> |
Expand Down
Loading