Merging changes from dev_graph

.github/workflows/tests-conda.yml

@@ -39,7 +39,7 @@ jobs:
       with:
         auto-activate-base: false
         activate-environment: pathml
-        environment-file: environment_test.yml
+        environment-file: requirements/environment_test.yml
         # mamba-version: "*"
         miniforge-version: latest
         use-mamba: true
@@ -48,15 +48,15 @@ jobs:
     - name: Debugging
       run: |
         echo "Printing the environment.yml file..."
-        cat environment_test.yml
+        cat requirements/environment_test.yml
         echo "Checking the status of mamba..."
         mamba --version
         echo "Checking the available disk space..."
         df -h
         
     - name: Install dependencies with mamba
       shell: bash -l {0}
-      run: mamba env update --file environment_test.yml --name pathml
+      run: mamba env update --file requirements/environment_test.yml --name pathml
     - name: Conda info
       shell: bash -l {0}
       run: |
@@ -71,10 +71,6 @@ jobs:
     - name: Install PathML
       shell: bash -l {0}      
       run: pip install -e .
-
-  # - name: Install python-spams
-  #   shell: bash -l {0}      
-  #   run: pip install spams
 
     - name: Install torchvision
       shell: bash -l {0}

CONTRIBUTING.rst

@@ -78,7 +78,7 @@ Building documentation locally
 .. code-block::
 
     cd docs                                     # enter docs directory
-    pip install -r readthedocs-requirements     # install packages to build docs
+    pip install -r readthedocs-requirements.txt     # install packages to build docs
     make html                                   # build docs in html format
 
 Then use your favorite web browser to open ``pathml/docs/build/html/index.html``

README.md

@@ -28,25 +28,19 @@ Imaging datasets in cancer research are growing exponentially in both quantity a
 :construction: the `dev` branch is under active development, with experimental features, bug fixes, and refactors that may happen at any time! 
 Stable versions are available as tagged releases on GitHub, or as versioned releases on PyPI
 
-
 # Installation
 
-There are several ways to install `PathML`:
+`PathML` is an advanced tool for pathology image analysis. Below are simplified instructions to help you install PathML on your system. Whether you're a user or a developer, follow these steps to get started.
 
-1. `pip install` from PyPI (**recommended for users**)
-2. Clone repo to local machine and install from source (recommended for developers/contributors)
-3. Use the PathML Docker container
+## 1. Prerequisites
 
-Options (1) and (2) require that you first install all external dependencies:
-* OpenSlide
-* JDK 17
+We recommend using [Conda](https://conda.io/projects/conda/en/latest/user-guide/getting-started.html#) for managing your environments. 
 
-We recommend using conda for environment management. 
-Download Miniconda [here](https://docs.conda.io/en/latest/miniconda.html)
+#### Installing Conda
 
-*Note: these instructions are for Linux. Commands may be different for other platforms.*
+If you don't have Conda installed, you can download Miniconda [here]. (https://docs.conda.io/en/latest/miniconda.html)
 
-### Updating Conda and Using libmamba
+#### Updating Conda and Using libmamba
 
 Recent versions of Conda have integrated `libmamba`, a faster dependency solver. To benefit from this improvement, first ensure your Conda is updated:
 
@@ -61,84 +55,56 @@ conda install -n base conda-libmamba-solver
 conda config --set solver libmamba
 ````
 
-Please refer to [this Anaconda blog post](https://www.anaconda.com/blog/a-faster-conda-for-a-growing-community).
+*Note: these instructions are for Linux. Commands may be different for other platforms.*
 
-## Installation option 1: pip install
+## 2. PathML Installation Methods
 
-### Common Steps
+### 2.1 Install with pip (Recommended for Users)
 
-Create conda environment:
+#### Common Steps
+
+Create and Activate Conda Environment:
 ````
 conda create --name pathml python=3.9
 conda activate pathml
 ````
 
-### Platform-Specific External Dependencies
+#### Platform-Specific External Dependencies
 
-#### Install external dependencies (Linux) with [Apt](https://ubuntu.com/server/docs/package-management):
+* Linux: Install external dependencies with [Apt](https://ubuntu.com/server/docs/package-management):
 ````
 sudo apt-get install openslide-tools g++ gcc libblas-dev liblapack-dev
 ````
 
-#### Install external dependencies (MacOS) with [Brew](www.brew.sh):
+* MacOS: Install external dependencies with [Brew](www.brew.sh):
 ````
 brew install openslide
 ````
 
-#### Windows
+* Windows:
 
-##### Option A: Install with [vcpkg](https://vcpkg.io/en/):
+ 1. Option A: Install with [vcpkg](https://vcpkg.io/en/):
 ````
 vcpkg install openslide
 ````
 
-##### Option B: Using Pre-built OpenSlide Binaries (Alternative)
+ 2. Option B: Using Pre-built OpenSlide Binaries (Alternative)
 For Windows users, an alternative to using `vcpkg` is to download and use pre-built OpenSlide binaries. This method is recommended if you prefer a quicker setup.
 
-1. Download the OpenSlide Windows binaries from the [OpenSlide Downloads](https://openslide.org/download/) page.
-2. Extract the archive to your desired location, e.g., `C:\OpenSlide\`.
-
-**Importing PathML in Windows:**
-
-Insert the following code snippet at the beginning of your Python script or Jupyter notebook before importing PathML. This code sets up the DLL directory for OpenSlide, ensuring that the library is properly loaded:
-
-```python
+  - Download the OpenSlide Windows binaries from the [OpenSlide Downloads](https://openslide.org/download/) page.
+  - Extract the archive to your desired location, e.g., `C:\OpenSlide\`.
 
-# The path can also be read from a config file, etc.
-OPENSLIDE_PATH = r'c:\path\to\openslide-win64\bin'
-
-import os
-if hasattr(os, 'add_dll_directory'):
-    # Windows-specific setup
-    with os.add_dll_directory(OPENSLIDE_PATH):
-        import openslide
-else:
-    # For other OSes, this step is not needed
-    import openslide
-
-# Now you can proceed with using PathML
-import pathml
-
-```
-This code snippet ensures that the OpenSlide DLLs are correctly found by Python on Windows systems. Replace c:\path\to\openslide-win64\bin with the actual path where you extracted the OpenSlide binaries.
-
-If you encounter any DLL load failures, verify that the OpenSlide `bin` directory is correctly added to your `PATH`.
-
-### Install OpenJDK 17
+#### Install OpenJDK 17
 ````
 conda install -c conda-forge 'openjdk<=18.0'
 ````
 
-### Optionally install CUDA
-
-[Follow instructions here](#CUDA)
-
-### Install `PathML` from PyPI
+#### Install `PathML` from PyPI
 ````
 pip install pathml
 ````
 
-## Installation option 2: clone repo and install from source
+### 2.2 Install from Source (Recommended for Developers)
 
 Clone repo:
 ````
@@ -147,19 +113,19 @@ cd pathml
 ````
 
 Create conda environment:
+
 ````
 conda env create -f environment.yml
 conda activate pathml
 ````
-
-Optionally install CUDA (instructions [here](#CUDA))
+To use GPU acceleration for model training or other tasks, you must install CUDA. The default CUDA version in our environment file is 11.6. To install a different CUDA version, refer to the instructions [here](##CUDA)). 
 
 Install `PathML` from source: 
 ````
 pip install -e .
 ````
 
-## Installation option 3: Docker
+### 2.3 Use Docker Container
 
 First, download or build the PathML Docker container:
 
@@ -196,7 +162,7 @@ Note that these instructions assume that there are no other processes using port
 Please refer to the `Docker run` [documentation](https://docs.docker.com/engine/reference/run/) for further instructions
 on accessing the container, e.g. for mounting volumes to access files on a local machine from within the container.
 
-## Option 4: Google Colab
+### 2.4 Use Google Colab
 
 To get PathML running in a Colab environment:
 
@@ -214,6 +180,38 @@ os.environ["JAVA_HOME"] = "/usr/lib/jvm/java-17-openjdk-amd64"
 *Thanks to all of our open-source collaborators for helping maintain these installation instructions!*  
 *Please open an issue for any bugs or other problems during installation process.*
 
+## 3. Import PathML
+
+After you have installed all necessary dependencies and PathML itself, import it using the following command:
+
+````
+import pathml
+````
+
+For Windows users, insert the following code snippet at the beginning of your Python script or Jupyter notebook before importing PathML. This code sets up the DLL directory for OpenSlide, ensuring that the library is properly loaded:
+
+```python
+
+# The path can also be read from a config file, etc.
+OPENSLIDE_PATH = r'c:\path\to\openslide-win64\bin'
+
+import os
+if hasattr(os, 'add_dll_directory'):
+    # Windows-specific setup
+    with os.add_dll_directory(OPENSLIDE_PATH):
+        import openslide
+else:
+    # For other OSes, this step is not needed
+    import openslide
+
+# Now you can proceed with using PathML
+import pathml
+```
+This code snippet ensures that the OpenSlide DLLs are correctly found by Python on Windows systems. Replace c:\path\to\openslide-win64\bin with the actual path where you extracted the OpenSlide binaries.
+
+If you encounter any DLL load failures, verify that the OpenSlide `bin` directory is correctly added to your `PATH`.
+
+
 ## CUDA
 
 To use GPU acceleration for model training or other tasks, you must install CUDA. 
@@ -224,22 +222,25 @@ Check the version of CUDA:
 nvidia-smi
 ````
 
-Install correct version of `cudatoolkit`:
+Replace both instances of 'cu116' in `requirements/requirements_torch.txt` with the CUDA version you see. For example, for CUDA 11.7, 'cu116' becomes 'cu117'. 
+
+Then create the environment:
+
 ````
-# update this command with your CUDA version number
-conda install cudatoolkit=11.0
+conda env create -f environment.yml
+conda activate pathml
 ````
 
 After installing PyTorch, optionally verify successful PyTorch installation with CUDA support: 
 ````
 python -c "import torch; print(torch.cuda.is_available())"
 ````
 
-# Using with Jupyter
+## Using with Jupyter
 
 Jupyter notebooks are a convenient way to work interactively. To use `PathML` in Jupyter notebooks: 
 
-## Set JAVA_HOME environment variable
+### Set JAVA_HOME environment variable
 
 PathML relies on Java to enable support for reading a wide range of file formats.
 Before using `PathML` in Jupyter, you may need to manually set the `JAVA_HOME` environment variable 
@@ -252,7 +253,7 @@ specifying the path to Java. To do so:
     os.environ["JAVA_HOME"] = "/opt/conda/envs/pathml" # change path as needed
     ````
 
-## Register environment as an IPython kernel
+### Register environment as an IPython kernel
 ````
 conda activate pathml
 conda install ipykernel

docs/source/api_datasets_reference.rst

@@ -1,13 +1,14 @@
 Datasets API
 ============
 
-PanNuke
--------
+Downloadable Datasets
+----------------------
 
 .. autoapiclass:: pathml.datasets.PanNukeDataModule
+.. autoapiclass:: pathml.datasets.DeepFocusDataModule
 
+ML Dataset classes
+-------------------
 
-DeepFocus
----------
-
-.. autoapiclass:: pathml.datasets.DeepFocusDataModule
+.. autoapiclass:: pathml.datasets.TileDataset
+.. autoapiclass:: pathml.datasets.EntityDataset

docs/source/api_graph_reference.rst

@@ -0,0 +1,24 @@
+Graph API
+=============
+
+Graph Building
+--------------------
+
+.. autoapiclass:: pathml.graph.preprocessing.BaseGraphBuilder
+.. autoapiclass:: pathml.graph.preprocessing.KNNGraphBuilder
+.. autoapiclass:: pathml.graph.preprocessing.RAGGraphBuilder
+.. autoapiclass:: pathml.graph.preprocessing.MSTGraphBuilder
+
+Tissue Extraction
+------------------
+
+.. autoapiclass:: pathml.graph.preprocessing.SuperpixelExtractor
+.. autoapiclass:: pathml.graph.preprocessing.SLICSuperpixelExtractor
+.. autoapiclass:: pathml.graph.preprocessing.MergedSuperpixelExtractor
+.. autoapiclass:: pathml.graph.preprocessing.ColorMergedSuperpixelExtractor
+
+Graph Feature Extraction
+-------------------------
+
+.. autoapiclass:: pathml.graph.preprocessing.GraphFeatureExtractor
+    :members:

docs/source/api_ml_reference.rst

@@ -1,25 +1,30 @@
 ML API
 ======
 
-h5path Dataset
---------------
+Models
+---------
+
+.. autoapiclass:: pathml.ml.models.hovernet.HoVerNet
+    :members:
 
-.. autoapiclass:: pathml.ml.TileDataset
+.. autoapiclass:: pathml.ml.models.hactnet.HACTNet
     :members:
 
-HoVer-Net
+Layers
 ---------
 
-.. autoapiclass:: pathml.ml.HoVerNet
+.. autoapiclass:: pathml.ml.layers.GNNLayer
     :members:
 
 Helper functions
 ^^^^^^^^^^^^^^^^
 
-.. autoapifunction:: pathml.ml.hovernet.compute_hv_map
+.. autoapifunction:: pathml.ml.models.hovernet.compute_hv_map
+
+.. autoapifunction:: pathml.ml.models.hovernet.loss_hovernet
+
+.. autoapifunction:: pathml.ml.models.hovernet.remove_small_objs
 
-.. autoapifunction:: pathml.ml.hovernet.loss_hovernet
+.. autoapifunction:: pathml.ml.models.hovernet.post_process_batch_hovernet
 
-.. autoapifunction:: pathml.ml.hovernet.remove_small_objs
 
-.. autoapifunction:: pathml.ml.hovernet.post_process_batch_hovernet