From c7f7f3ea679efce78af13c7871a5fe931cd96214 Mon Sep 17 00:00:00 2001 From: Nicolas Fernandez <cornhundred@users.noreply.github.com> Date: Thu, 2 Jan 2025 18:01:16 -0500 Subject: [PATCH] Deployed 42ea883 with MkDocs version: 1.6.0 --- search/search_index.json | 2 +- technologies/index.html | 16 +++++++++------- technologies/index.md | 13 ++++++++----- 3 files changed, 18 insertions(+), 13 deletions(-) diff --git a/search/search_index.json b/search/search_index.json index 7ecc8868..f08e26f5 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome to Celldega's Documentation","text":"Celldega Landscape visualization of a human skin cancer Xenium dataset obtained from 10X Genomics. <p>Celldega is a spatial analysis and visualization library that is being developed by the Spatial Technology Platform at the Broad Institute of MIT and Harvard. This project enables researchers to easily visualize large ST datasets (e.g., datasets with >100M transcripts) alongside single-cell and spatial analysis notebook workflows (e.g., sverse tools and novel spatial analysis approaches).</p> <ul> <li>Getting Started</li> <li>Installation</li> <li>Usage</li> </ul>"},{"location":"#about","title":"About","text":"<p>Celldega is named after a bodega, a small shop with all the essentials that is part of the fabric of a neighborhood.</p>"},{"location":"examples/","title":"Jupyter Notebook Examples","text":"<p>Landscape View Xenium</p>"},{"location":"examples/short_notebooks/Landscape_View_Xenium/","title":"Landscape View Xenium","text":"In\u00a0[2]: Copied! <pre># %load_ext autoreload\n# %autoreload 2\n# %env ANYWIDGET_HMR=1\n</pre> # %load_ext autoreload # %autoreload 2 # %env ANYWIDGET_HMR=1 In\u00a0[3]: Copied! <pre>import celldega as dega\ndega.__version__\n</pre> import celldega as dega dega.__version__ Out[3]: <pre>'0.0.0'</pre> In\u00a0[6]: Copied! <pre>from observable_jupyter import embed\n</pre> from observable_jupyter import embed In\u00a0[11]: Copied! <pre>base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Mouse_Brain_Coronal_FF_outs/main/Xenium_Prime_Mouse_Brain_Coronal_FF_outs'\n</pre> base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Mouse_Brain_Coronal_FF_outs/main/Xenium_Prime_Mouse_Brain_Coronal_FF_outs' In\u00a0[14]: Copied! <pre>embed('@cornhundred/celldega-landscape-ist', inputs={'base_url': base_url}, cells=['landscape_container'], display_logo=False)\n</pre> embed('@cornhundred/celldega-landscape-ist', inputs={'base_url': base_url}, cells=['landscape_container'], display_logo=False) In\u00a0[13]: Copied! <pre># base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Human_Skin_FFPE_outs/main/Xenium_Prime_Human_Skin_FFPE_outs'\n\n# landscape_ist = dega.viz.Landscape(\n# technology='Xenium',\n# ini_zoom = -4.5,\n# ini_x=6000,\n# ini_y=8000,\n# base_url = base_url,\n\n# )\n\n# landscape_ist\n</pre> # base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Human_Skin_FFPE_outs/main/Xenium_Prime_Human_Skin_FFPE_outs' # landscape_ist = dega.viz.Landscape( # technology='Xenium', # ini_zoom = -4.5, # ini_x=6000, # ini_y=8000, # base_url = base_url, # ) # landscape_ist In\u00a0[\u00a0]: Copied! <pre>\n</pre>"},{"location":"examples/short_notebooks/Landscape_View_Xenium/#landscape-view-xenium","title":"Landscape View Xenium\u00b6","text":""},{"location":"gallery/","title":"Celldega Gallery","text":"<p>This page includes links to visualizations that are made with the stand-alone Celldega JavaScript library.</p>"},{"location":"gallery/#imaging-spatial-transcriptomics","title":"Imaging Spatial Transcriptomics","text":""},{"location":"gallery/#xenium","title":"Xenium","text":"<ul> <li>Xenium Mouse Brain </li> <li>Xenium Human Skin Cancer </li> </ul>"},{"location":"gallery/#sequencing-spatial-transcriptomics","title":"Sequencing Spatial Transcriptomics","text":""},{"location":"gallery/#visium-hd","title":"Visium HD","text":""},{"location":"gallery/gallery_xenium/","title":"Celldega Xenium Gallery","text":""},{"location":"gallery/gallery_xenium/#xenium-prime-mouse-brain-coronal-ff","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium/#xenium-prime-human-skin-ffpe-outs","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"gallery/gallery_xenium/#xenium-human-pancreas-ffpe","title":"Xenium Human Pancreas FFPE","text":""},{"location":"gallery/gallery_xenium/#bone-marrow","title":"Bone Marrow","text":""},{"location":"gallery/gallery_xenium_mouse_brain/","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium_multi/","title":"Xenium Multi Dataset","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-prime-mouse-brain-coronal-ff","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-prime-human-skin-ffpe-outs","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-human-pancreas-ffpe","title":"Xenium Human Pancreas FFPE","text":""},{"location":"gallery/gallery_xenium_multi/#bone-marrow","title":"Bone Marrow","text":""},{"location":"gallery/gallery_xenium_skin_cancer/","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"javascript/","title":"JavaScript API Overview","text":"<p>Celldega's visualization methods can be used as a stand-alone JavaScript library outside the context of a Jupyter notebook. This can be used to create showcase visualizations with publicly hosted data.</p>"},{"location":"javascript/api/","title":"Celldega JavaScript API Documentation","text":"<p>The JavaScript component of Celldega is used within the Jupyter Widgets framework to provide interactive visualization in the context of a Jupyter notebook but can also be used as a standalone JavaScript library.</p>"},{"location":"javascript/api/#landscape_ist-api-documentation","title":"<code>landscape_ist</code> API Documentation","text":"<p>The <code>landscape_ist</code> function initializes and renders an interactive spatial transcriptomics (IST) landscape visualization. This API is designed to work with Deck.gl and includes customizable visualization options, dynamic data updates, and UI interactions.</p>"},{"location":"javascript/api/#parameters","title":"Parameters","text":"<ul> <li><code>el</code> (<code>HTMLElement</code>): The root DOM element where the visualization is rendered.</li> <li><code>ini_model</code> (<code>Object</code>): The initial data model containing configuration and state.</li> <li><code>token</code> (<code>string</code>): Authentication token for accessing data.</li> <li><code>ini_x</code>, <code>ini_y</code>, <code>ini_z</code> (<code>number</code>): Initial spatial coordinates for the view.</li> <li><code>ini_zoom</code> (<code>number</code>): Initial zoom level for the visualization.</li> <li><code>base_url</code> (<code>string</code>): Base URL for accessing data files.</li> <li><code>dataset_name</code> (<code>string</code>, optional): Name of the dataset being visualized.</li> <li><code>trx_radius</code> (<code>number</code>, optional): Initial radius for transcript points. Default: <code>0.25</code>.</li> <li><code>width</code> (<code>number|string</code>, optional): Width of the visualization. Default: <code>100%</code>.</li> <li><code>height</code> (<code>number</code>, optional): Height of the visualization. Default: <code>800</code>.</li> <li><code>view_change_custom_callback</code> (<code>Function</code>, optional): Custom callback triggered on view changes.</li> </ul>"},{"location":"javascript/api/#public-api","title":"Public API","text":"<p>The <code>landscape_ist</code> function returns an object (<code>landscape</code>) with several methods for interacting with the visualization.</p>"},{"location":"javascript/api/#update_matrix_gene","title":"<code>update_matrix_gene</code>","text":"<p>Updates the visualization to highlight data for a specific gene.</p>"},{"location":"javascript/api/#parameters_1","title":"Parameters","text":"<ul> <li><code>inst_gene</code> (<code>string</code>): The gene to highlight.</li> </ul>"},{"location":"javascript/api/#behavior","title":"Behavior","text":"<ul> <li>Updates the transcript layer to show data for the specified gene.</li> <li>Scrolls the bar graph to bring the selected gene into view.</li> <li>Toggles visibility of image layers and controls based on the selected gene.</li> </ul>"},{"location":"javascript/api/#update_matrix_col","title":"<code>update_matrix_col</code>","text":"<p>Updates the visualization to highlight data for a specific column (e.g., cluster).</p>"},{"location":"javascript/api/#parameters_2","title":"Parameters","text":"<ul> <li><code>inst_col</code> (<code>string</code>): The column to highlight.</li> </ul>"},{"location":"javascript/api/#behavior_1","title":"Behavior","text":"<ul> <li>Highlights the bar graph corresponding to the selected column.</li> <li>Updates cell and path layers to reflect the selected column.</li> <li>Toggles visibility of layers based on the column selection.</li> </ul>"},{"location":"javascript/api/#update_matrix_dendro_col","title":"<code>update_matrix_dendro_col</code>","text":"<p>Updates the visualization based on a dendrogram selection of columns.</p>"},{"location":"javascript/api/#parameters_3","title":"Parameters","text":"<ul> <li><code>selected_cols</code> (<code>Array<string></code>): The list of selected column names.</li> </ul>"},{"location":"javascript/api/#behavior_2","title":"Behavior","text":"<ul> <li>Highlights the selected columns in the bar graph.</li> <li>Updates layers to reflect the selection.</li> </ul>"},{"location":"javascript/api/#update_view_state","title":"<code>update_view_state</code>","text":"<p>Updates the view state of the Deck.gl visualization.</p>"},{"location":"javascript/api/#parameters_4","title":"Parameters","text":"<ul> <li><code>new_view_state</code> (<code>Object</code>): The new view state configuration.</li> <li><code>close_up</code> (<code>boolean</code>): Whether the view should zoom in closely.</li> <li><code>trx_layer</code> (<code>Object</code>): The transcript layer to update.</li> </ul>"},{"location":"javascript/api/#behavior_3","title":"Behavior","text":"<ul> <li>Adjusts the viewport and reconfigures layers based on the new view state.</li> </ul>"},{"location":"javascript/api/#update_layers","title":"<code>update_layers</code>","text":"<p>Updates all visualization layers.</p>"},{"location":"javascript/api/#behavior_4","title":"Behavior","text":"<ul> <li>Refreshes the Deck.gl layers with the current visualization state.</li> </ul>"},{"location":"javascript/api/#finalize","title":"<code>finalize</code>","text":"<p>Finalizes the Deck.gl instance and cleans up resources.</p>"},{"location":"javascript/api/#behavior_5","title":"Behavior","text":"<ul> <li>Disposes of all Deck.gl resources and event listeners to prevent memory leaks.</li> </ul>"},{"location":"javascript/api/#usage-example","title":"Usage Example","text":"<pre><code>\njavascript\nimport { landscape_ist } from 'path/to/landscape_ist';\n\nconst rootElement = document.getElementById('visualization-container');\nconst model = { /* Model containing visualization data */ };\n\nconst visualization = await landscape_ist(\n rootElement,\n model,\n 'example-token',\n 100,\n 200,\n 0,\n -5,\n 'https://example.com/data',\n 'Example Dataset'\n);\n\n// Update the visualization with a specific gene.\nvisualization.update_matrix_gene('TP53');\n\n// Update the visualization with a specific column.\nvisualization.update_matrix_col('Cluster 1');\n\n// Finalize the visualization when done.\nvisualization.finalize();\n\n</code></pre>"},{"location":"javascript/api/#matrix_viz-api-documentation","title":"<code>matrix_viz</code> API Documentation","text":"<p>The <code>matrix_viz</code> function initializes and renders a matrix visualization. This API is built using approaches and code adaptations from the Clustergrammer-GL library, and it integrates tightly with Deck.gl to provide interactive and dynamic visualizations.</p>"},{"location":"javascript/api/#parameters_5","title":"Parameters","text":"<ul> <li><code>model</code> (<code>Object</code>): The model object containing configuration data for the visualization.</li> <li><code>el</code> (<code>HTMLElement</code>): The root DOM element where the visualization is rendered.</li> <li><code>network</code> (<code>Object</code>): The network object containing the matrix data to visualize.</li> <li><code>width</code> (<code>string|number</code>, optional): The width of the visualization. Default: <code>'800'</code>.</li> <li><code>height</code> (<code>string|number</code>, optional): The height of the visualization. Default: <code>'800'</code>.</li> <li><code>row_label_callback</code> (<code>Function</code>, optional): A callback function triggered on row label interactions.</li> <li><code>col_label_callback</code> (<code>Function</code>, optional): A callback function triggered on column label interactions.</li> <li><code>col_dendro_callback</code> (<code>Function</code>, optional): A callback function triggered on dendrogram column interactions.</li> </ul>"},{"location":"javascript/api/#internal-behavior","title":"Internal Behavior","text":"<p>The function performs the following setup: 1. Deck.gl Integration: - Initializes a Deck.gl instance for the matrix visualization. - Sets properties for interactivity, including tooltips, view state changes, and layer filtering.</p> <ol> <li>Matrix Data Setup:</li> <li>Parses and structures the matrix data from the <code>network</code> object.</li> <li> <p>Configures labels, categories, and dendrograms for both rows and columns.</p> </li> <li> <p>Layer Initialization:</p> </li> <li>Creates layers for:<ul> <li>Matrix cells.</li> <li>Row and column labels.</li> <li>Row and column categories.</li> <li>Row and column dendrograms.</li> </ul> </li> <li> <p>Attaches interactions (e.g., click events) to these layers.</p> </li> <li> <p>UI Setup:</p> </li> <li>Creates a container for the visualization and appends it to the root DOM element.</li> </ol>"},{"location":"javascript/api/#example-usage","title":"Example Usage","text":"<pre><code>import { matrix_viz } from 'path/to/matrix_viz';\n\nconst rootElement = document.getElementById('matrix-container');\nconst model = { /* Model containing visualization data */ };\nconst network = { /* Network object representing the matrix data */ };\n\n// Callback functions\nconst rowLabelCallback = (row) => {\n console.log('Row label clicked:', row);\n};\n\nconst colLabelCallback = (col) => {\n console.log('Column label clicked:', col);\n};\n\nconst colDendroCallback = (dendro) => {\n console.log('Column dendrogram clicked:', dendro);\n};\n\n// Initialize the matrix visualization\nawait matrix_viz(\n model,\n rootElement,\n network,\n 800,\n 800,\n rowLabelCallback,\n colLabelCallback,\n colDendroCallback\n);\n</code></pre>"},{"location":"overview/","title":"Overview","text":"<p>The Celldega library is being developed to help researchers easily visualize and analyze high-dimensional spatial-omics data in the context of a notebook workflow. Initial development has been focused on spatial transcriptomics visualization.</p> <p>Celldega can be used as a Python library in a Jupyter notebook environment or as a stand-alone JavaScript library for creating visualizations.</p> <ul> <li>Getting Started</li> <li>Installation</li> <li>Usage</li> </ul>"},{"location":"overview/file_formats/","title":"File Formats","text":""},{"location":"overview/file_formats/#landscapefiles","title":"LandscapeFiles","text":"<p>This format is small enough that we can store an entire Xenium dataset on GitHub as a repo for the purposes of building a public gallery.</p>"},{"location":"overview/getting_started/","title":"Getting Started","text":"<p>Celldega is a spatial analysis and visualization library that is being developed by the Spatial Technology Platform at the Broad Institute of MIT and Harvard. Celldega can be used as a Jupyter Widget in Python as well as a stand-alone JavaScript library.</p> <p>Please see examples notebooks below to try out Celldega in a Jupyter notebook or ObservableHQ JavaScript notebook:</p> <ul> <li>Celldega_Xenium_Landscape_Visualizations_Colab.ipynb</li> <li>Celldega Landscape Xenium ObservableHQ</li> </ul>"},{"location":"overview/installation/","title":"Installation","text":""},{"location":"overview/installation/#python","title":"Python","text":"<p>The Celldega library can be installed using pip</p> <p>Celldega can be installed using pip:</p> <pre><code>pip install celldega\n</code></pre> <p>Celldega can also be installed with the optional pre-processing requirements (e.g., vips for image pre-processing) using:</p> <pre><code>pip install celldega[pre]\n</code></pre>"},{"location":"overview/installation/#javascript","title":"JavaScript","text":"<p>Celldega can be used in a JavaScript environment such as ObservableHQ by importing it as a module from content delivery networks like esm.sh:</p> <pre><code>celldega = await import('https://esm.sh/celldega@latest')\n</code></pre> <p>Or by importing it from a local file</p> <pre><code>import celldega from 'js/widget.js'\n</code></pre>"},{"location":"overview/usage/","title":"Celldega Usage","text":""},{"location":"overview/usage/#terrabio","title":"Terra.bio","text":"<p>** Coming soon **</p>"},{"location":"python/","title":"Python API Overview","text":""},{"location":"python/#pre-module-overview","title":"Pre Module Overview","text":"<p>The <code>pre</code> module contains methods for pre-processing LandscapeFiles.</p>"},{"location":"python/#viz-module-overview","title":"Viz Module Overview","text":"<p>The <code>viz</code> module contains functions and classes for data visualization.</p>"},{"location":"python/api/","title":"Python API Reference","text":"<p>Module for pre-processing to generate LandscapeFiles from ST data.</p> <p>Module for visualization</p>"},{"location":"python/api/#celldega.pre.convert_long_id_to_short","title":"<code>convert_long_id_to_short(df)</code>","text":"<p>Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>The DataFrame containing the EntityID.</p> required <p>Returns: pd.DataFrame: The original DataFrame with an additional column named <code>cell_id</code> containing the shortened cell IDs.</p> <p>The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates it to create a shorter identifier that is added as a new column to the DataFrame.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_long_id_to_short(df):\n \"\"\"\n Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.\n\n Args:\n df (pd.DataFrame): The DataFrame containing the EntityID.\n Returns:\n pd.DataFrame: The original DataFrame with an additional column named `cell_id`\n containing the shortened cell IDs.\n\n The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates\n it to create a shorter identifier that is added as a new column to the DataFrame.\n \"\"\"\n # Function to hash and encode the cell ID\n def hash_and_shorten_id(cell_id):\n # Create a hash of the cell ID\n cell_id_bytes = str(cell_id).encode('utf-8')\n hash_object = hashlib.sha256(cell_id_bytes)\n hash_digest = hash_object.digest()\n\n # Encode the hash to a base64 string to mix letters and numbers, truncate to 9 characters\n short_id = base64.urlsafe_b64encode(hash_digest).decode('utf-8')[:9]\n return short_id\n\n # Apply the hash_and_shorten_id function to each cell ID in the specified column\n df['cell_id'] = df['EntityID'].apply(hash_and_shorten_id)\n\n return df\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_jpeg","title":"<code>convert_to_jpeg(image_path, quality=80)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/api/#celldega.pre.convert_to_jpeg--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/api/#celldega.pre.convert_to_jpeg--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_jpeg(image_path, quality=80):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".jpeg\")\n image.jpegsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_png","title":"<code>convert_to_png(image_path)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/api/#celldega.pre.convert_to_png--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/api/#celldega.pre.convert_to_png--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_png(image_path):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".png\")\n image.pngsave(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_webp","title":"<code>convert_to_webp(image_path, quality=100)</code>","text":"<p>Convert a TIFF image to a WEBP image with a specified quality score.</p>"},{"location":"python/api/#celldega.pre.convert_to_webp--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=100) Quality score for the WEBP image (higher is better quality)</p>"},{"location":"python/api/#celldega.pre.convert_to_webp--returns","title":"Returns","text":"<p>new_image_path : str Path to the WEBP image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_webp(image_path, quality=100):\n \"\"\"\n Convert a TIFF image to a WEBP image with a specified quality score.\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=100)\n Quality score for the WEBP image (higher is better quality)\n\n Returns\n -------\n new_image_path : str\n Path to the WEBP image file\n \"\"\"\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a WEBP with specified quality\n new_image_path = image_path.replace(\".tif\", \".webp\")\n image.webpsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.get_max_zoom_level","title":"<code>get_max_zoom_level(path_image_pyramid)</code>","text":"<p>Returns the maximum zoom level based on the highest-numbered directory in the specified path_image_pyramid.</p> <p>Parameters:</p> Name Type Description Default <code>path_image_pyramid</code> <code>str</code> <p>The path to the directory containing zoom level directories.</p> required <p>Returns:</p> Name Type Description <code>max_pyramid_zoom</code> <code>int</code> <p>The maximum zoom level.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def get_max_zoom_level(path_image_pyramid):\n \"\"\"\n Returns the maximum zoom level based on the highest-numbered directory\n in the specified path_image_pyramid.\n\n Parameters:\n path_image_pyramid (str): The path to the directory containing zoom level directories.\n\n Returns:\n max_pyramid_zoom (int): The maximum zoom level.\n \"\"\"\n # List all entries in the path_image_pyramid that are directories and can be converted to integers\n zoom_levels = [\n entry\n for entry in os.listdir(path_image_pyramid)\n if os.path.isdir(os.path.join(path_image_pyramid, entry)) and entry.isdigit()\n ]\n\n # Convert to integer and find the maximum value\n max_pyramid_zoom = max(map(int, zoom_levels)) if zoom_levels else None\n\n return max_pyramid_zoom\n</code></pre>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles","title":"<code>make_cell_boundary_tiles(technology, path_cell_boundaries, path_meta_cell_micron, path_transformation_matrix, path_output, coarse_tile_factor=20, tile_size=250, tile_bounds=None, image_scale=1, max_workers=8)</code>","text":"<p>Processes cell boundary data and divides it into spatial tiles based on the provided technology. Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles. The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.</p>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\". path_cell_boundaries : str Path to the file containing the cell boundaries (Parquet format). path_meta_cell_micron : str Path to the file containing cell metadata (CSV format). path_transformation_matrix : str Path to the file containing the transformation matrix (CSV format). path_output : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional, default=20. scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional, default=500 Size of each fine-grain tile in microns. tile_bounds : dict, optional Dictionary containing the minimum and maximum bounds for x and y coordinates. image_scale : float, optional, default=1 Scale factor to apply to the geometry data. max_workers : int, optional, default=8 Maximum number of parallel workers for processing tiles.</p>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/boundary_tile.py</code> <pre><code>def make_cell_boundary_tiles(\n technology,\n path_cell_boundaries,\n path_meta_cell_micron,\n path_transformation_matrix,\n path_output,\n coarse_tile_factor=20,\n tile_size=250,\n tile_bounds=None,\n image_scale=1,\n max_workers=8\n):\n\n\n \"\"\"\n Processes cell boundary data and divides it into spatial tiles based on the provided technology.\n Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles.\n The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.\n\n Parameters\n ----------\n technology : str\n The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\".\n path_cell_boundaries : str\n Path to the file containing the cell boundaries (Parquet format).\n path_meta_cell_micron : str\n Path to the file containing cell metadata (CSV format).\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV format).\n path_output : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional, default=20.\n scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional, default=500\n Size of each fine-grain tile in microns.\n tile_bounds : dict, optional\n Dictionary containing the minimum and maximum bounds for x and y coordinates.\n image_scale : float, optional, default=1\n Scale factor to apply to the geometry data.\n max_workers : int, optional, default=8\n Maximum number of parallel workers for processing tiles.\n\n Returns\n -------\n None\n \"\"\"\n\n def numpy_affine_transform(coords, matrix):\n \"\"\"Apply affine transformation to numpy coordinates.\"\"\"\n # Homogeneous coordinates for affine transformation\n coords = np.hstack([coords, np.ones((coords.shape[0], 1))])\n transformed_coords = coords @ matrix.T\n return transformed_coords[:, :2] # Drop the homogeneous coordinate\n\n def batch_transform_geometries(geometries, transformation_matrix, scale):\n \"\"\"\n Batch transform geometries using numpy for optimized performance.\n \"\"\"\n # Extract affine transformation parameters into a 3x3 matrix for numpy\n affine_matrix = np.array([\n [transformation_matrix[0, 0], transformation_matrix[0, 1], transformation_matrix[0, 2]],\n [transformation_matrix[1, 0], transformation_matrix[1, 1], transformation_matrix[1, 2]],\n [0, 0, 1]\n ])\n\n transformed_geometries = []\n\n for polygon in geometries:\n # Extract coordinates and transform them\n if isinstance(polygon, MultiPolygon):\n polygon = next(polygon.geoms) # Use the first geometry\n\n # Transform the exterior of the polygon\n exterior_coords = np.array(polygon.exterior.coords)\n\n # Apply the affine transformation and scale\n transformed_coords = numpy_affine_transform(exterior_coords, affine_matrix) / scale\n\n # Append the result to the transformed_geometries list\n transformed_geometries.append([transformed_coords.tolist()])\n\n return transformed_geometries\n\n\n def filter_and_save_fine_boundary(coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output):\n cell_ids = coarse_tile.index.values\n\n tile_filter = (\n (coarse_tile[\"center_x\"] >= fine_tile_x_min) & (coarse_tile[\"center_x\"] < fine_tile_x_max) &\n (coarse_tile[\"center_y\"] >= fine_tile_y_min) & (coarse_tile[\"center_y\"] < fine_tile_y_max)\n )\n filtered_indices = np.where(tile_filter)[0]\n\n keep_cells = cell_ids[filtered_indices]\n fine_tile_cells = coarse_tile.loc[keep_cells, [\"GEOMETRY\"]]\n fine_tile_cells = fine_tile_cells.assign(name=fine_tile_cells.index)\n\n if not fine_tile_cells.empty:\n filename = f\"{path_output}/cell_tile_{fine_i}_{fine_j}.parquet\"\n fine_tile_cells.to_parquet(filename)\n\n def process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y):\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n futures.append(executor.submit(\n filter_and_save_fine_boundary, coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output\n ))\n\n for future in futures:\n future.result()\n\n tile_size_x = tile_size\n tile_size_y = tile_size\n\n transformation_matrix = pd.read_csv(path_transformation_matrix, header=None, sep=\" \").values\n\n # Load cell boundary data based on the technology\n if technology == \"MERSCOPE\":\n df_meta = pd.read_parquet(f\"{path_output.replace('cell_segmentation','cell_metadata.parquet')}\")\n entity_to_cell_id_dict = pd.Series(df_meta.index.values, index=df_meta.EntityID).to_dict()\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n cells_orig['cell_id'] = cells_orig['EntityID'].map(entity_to_cell_id_dict)\n cells_orig = cells_orig[cells_orig[\"ZIndex\"] == 1]\n\n # Correct cell_id issues with meta_cell\n meta_cell = pd.read_csv(path_meta_cell_micron)\n meta_cell['cell_id'] = meta_cell['EntityID'].map(entity_to_cell_id_dict)\n cells_orig.index = meta_cell[meta_cell[\"cell_id\"].isin(cells_orig['cell_id'])].index\n\n # Correct 'MultiPolygon' to 'Polygon'\n cells_orig[\"geometry\"] = cells_orig[\"Geometry\"].apply(\n lambda x: list(x.geoms)[0] if isinstance(x, MultiPolygon) else x\n )\n\n cells_orig.set_index('cell_id', inplace=True)\n\n elif technology == \"Xenium\":\n xenium_cells = pd.read_parquet(path_cell_boundaries)\n grouped = xenium_cells.groupby(\"cell_id\")[[\"vertex_x\", \"vertex_y\"]].agg(lambda x: x.tolist())\n grouped[\"geometry\"] = grouped.apply(lambda row: Polygon(zip(row[\"vertex_x\"], row[\"vertex_y\"])), axis=1)\n cells_orig = gpd.GeoDataFrame(grouped, geometry=\"geometry\")[[\"geometry\"]]\n\n elif technology == \"custom\":\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n\n # Transform geometries\n cells_orig[\"GEOMETRY\"] = batch_transform_geometries(cells_orig[\"geometry\"], transformation_matrix, image_scale)\n\n # Convert transformed geometries to polygons and calculate centroids\n cells_orig[\"polygon\"] = cells_orig[\"GEOMETRY\"].apply(lambda x: Polygon(x[0]))\n gdf_cells = gpd.GeoDataFrame(geometry=cells_orig[\"polygon\"])\n gdf_cells[\"center_x\"] = gdf_cells.geometry.centroid.x\n gdf_cells[\"center_y\"] = gdf_cells.geometry.centroid.y\n gdf_cells[\"GEOMETRY\"] = cells_orig[\"GEOMETRY\"]\n\n # Ensure the output directory exists\n if not os.path.exists(path_output):\n os.makedirs(path_output)\n\n # Calculate tile bounds and fine/coarse tiles\n x_min, x_max = tile_bounds[\"x_min\"], tile_bounds[\"x_max\"]\n y_min, y_max = tile_bounds[\"y_min\"], tile_bounds[\"y_max\"]\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Process coarse tiles in parallel\n for i in tqdm(range(n_coarse_tiles_x), desc=\"Processing coarse tiles\"):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n coarse_tile = gdf_cells[\n (gdf_cells[\"center_x\"] >= coarse_tile_x_min) & (gdf_cells[\"center_x\"] < coarse_tile_x_max) &\n (gdf_cells[\"center_y\"] >= coarse_tile_y_min) & (gdf_cells[\"center_y\"] < coarse_tile_y_max)\n ]\n if not coarse_tile.empty:\n process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid","title":"<code>make_deepzoom_pyramid(image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix='.jpeg')</code>","text":"<p>Create a DeepZoom image pyramid from a JPEG image</p>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid--parameters","title":"Parameters","text":"<p>image_path : str Path to the JPEG image file tile_size : int (default=512) Tile size for the DeepZoom pyramid overlap : int (default=0) Overlap size for the DeepZoom pyramid suffix : str (default='jpeg') Suffix for the DeepZoom pyramid tiles</p>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_deepzoom_pyramid(\n image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix=\".jpeg\"\n):\n \"\"\"\n Create a DeepZoom image pyramid from a JPEG image\n\n Parameters\n ----------\n image_path : str\n Path to the JPEG image file\n tile_size : int (default=512)\n Tile size for the DeepZoom pyramid\n overlap : int (default=0)\n Overlap size for the DeepZoom pyramid\n suffix : str (default='jpeg')\n Suffix for the DeepZoom pyramid tiles\n\n Returns\n -------\n None\n\n \"\"\"\n\n # Define the output path\n output_path = Path(output_path)\n\n # Load the JPEG image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # check if the output path exists and create it if it does not\n output_path.mkdir(parents=True, exist_ok=True)\n\n # append the pyramid name to the output path\n output_path = output_path / pyramid_name\n\n # Save the image as a DeepZoom image pyramid\n image.dzsave(output_path, tile_size=tile_size, overlap=overlap, suffix=suffix)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord","title":"<code>make_meta_cell_image_coord(technology, path_transformation_matrix, path_meta_cell_micron, path_meta_cell_image, image_scale)</code>","text":"<p>Apply an affine transformation to the cell coordinates in microns and save the transformed coordinates in pixels</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_transformation_matrix : str Path to the transformation matrix file path_meta_cell_micron : str Path to the meta cell file with coordinates in microns path_meta_cell_image : str Path to save the meta cell file with coordinates in pixels</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--examples","title":"Examples","text":"<p>make_meta_cell_image_coord( ... technology='Xenium', ... path_transformation_matrix='data/transformation_matrix.txt', ... path_meta_cell_micron='data/meta_cell_micron.csv', ... path_meta_cell_image='data/meta_cell_image.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_cell_image_coord(\n technology,\n path_transformation_matrix,\n path_meta_cell_micron,\n path_meta_cell_image,\n image_scale\n):\n \"\"\"\n Apply an affine transformation to the cell coordinates in microns and save\n the transformed coordinates in pixels\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_transformation_matrix : str\n Path to the transformation matrix file\n path_meta_cell_micron : str\n Path to the meta cell file with coordinates in microns\n path_meta_cell_image : str\n Path to save the meta cell file with coordinates in pixels\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_cell_image_coord(\n ... technology='Xenium',\n ... path_transformation_matrix='data/transformation_matrix.txt',\n ... path_meta_cell_micron='data/meta_cell_micron.csv',\n ... path_meta_cell_image='data/meta_cell_image.parquet'\n ... )\n\n \"\"\"\n\n transformation_matrix = pd.read_csv(\n path_transformation_matrix, header=None, sep=\" \"\n ).values\n\n if technology == \"MERSCOPE\":\n meta_cell = pd.read_csv(path_meta_cell_micron, usecols=[\"EntityID\", \"center_x\", \"center_y\"])\n meta_cell = convert_long_id_to_short(meta_cell)\n meta_cell[\"name\"] = meta_cell[\"cell_id\"]\n meta_cell = meta_cell.set_index('cell_id')\n elif technology == \"Xenium\":\n usecols = [\"cell_id\", \"x_centroid\", \"y_centroid\"]\n meta_cell = pd.read_csv(path_meta_cell_micron, index_col=0, usecols=usecols)\n meta_cell.columns = [\"center_x\", \"center_y\"]\n meta_cell[\"name\"] = pd.Series(meta_cell.index, index=meta_cell.index)\n\n # Adding a ones column to accommodate for affine transformation\n meta_cell[\"ones\"] = 1\n\n # Preparing the data for matrix multiplication\n points = meta_cell[[\"center_x\", \"center_y\", \"ones\"]].values\n\n # Applying the transformation matrix\n transformed_points = np.dot(transformation_matrix, points.T).T\n\n # Updating the DataFrame with transformed coordinates\n meta_cell[\"center_x\"] = transformed_points[:, 0]\n meta_cell[\"center_y\"] = transformed_points[:, 1]\n\n # Dropping the ones column as it's no longer needed\n meta_cell.drop(columns=[\"ones\"], inplace=True)\n\n meta_cell[\"center_x\"] = meta_cell[\"center_x\"] / image_scale\n meta_cell[\"center_y\"] = meta_cell[\"center_y\"] / image_scale\n\n meta_cell[\"geometry\"] = meta_cell.apply(\n lambda row: [row[\"center_x\"], row[\"center_y\"]], axis=1\n )\n\n if technology == \"MERSCOPE\":\n meta_cell = meta_cell[[\"name\", \"geometry\", \"EntityID\"]]\n else:\n meta_cell = meta_cell[[\"name\", \"geometry\"]]\n\n\n meta_cell.to_parquet(path_meta_cell_image)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_meta_gene","title":"<code>make_meta_gene(technology, path_cbg, path_output)</code>","text":"<p>Create a DataFrame with genes and their assigned colors</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_cbg : str Path to the cell-by-gene matrix data (the data format can vary based on technology) path_output : str Path to save the meta gene file</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--examples","title":"Examples","text":"<p>make_meta_gene( ... technology='Xenium', ... path_cbg='data/', ... path_output='data/meta_gene.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_gene(technology, path_cbg, path_output):\n \"\"\"\n Create a DataFrame with genes and their assigned colors\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_cbg : str\n Path to the cell-by-gene matrix data (the data format can vary based on technology)\n path_output : str\n Path to save the meta gene file\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_gene(\n ... technology='Xenium',\n ... path_cbg='data/',\n ... path_output='data/meta_gene.parquet'\n ... )\n \"\"\"\n\n if technology == \"MERSCOPE\":\n cbg = pd.read_csv(path_cbg, index_col=0)\n genes = cbg.columns.tolist()\n elif technology == \"Xenium\":\n # genes = pd.read_csv(path_cbg + 'features.tsv.gz', sep='\\t', header=None)[1].values.tolist()\n cbg = read_cbg_mtx(path_cbg)\n genes = cbg.columns.tolist()\n\n # Get all categorical color palettes from Matplotlib and flatten them into a single list of colors\n palettes = [plt.get_cmap(name).colors for name in plt.colormaps() if \"tab\" in name]\n flat_colors = [color for palette in palettes for color in palette]\n\n # Convert RGB tuples to hex codes\n flat_colors_hex = [to_hex(color) for color in flat_colors]\n\n # Use modular arithmetic to assign a color to each gene, white for genes with \"Blank\"\n colors = [\n flat_colors_hex[i % len(flat_colors_hex)] if \"Blank\" not in gene else \"#FFFFFF\"\n for i, gene in enumerate(genes)\n ]\n\n # Create a DataFrame with genes and their assigned colors\n ser_color = pd.Series(colors, index=genes)\n\n # calculate gene expression metadata\n meta_gene = calc_meta_gene_data(cbg)\n meta_gene['color'] = ser_color\n\n # Identify sparse columns\n sparse_cols = [col for col in meta_gene.columns if pd.api.types.is_sparse(meta_gene[col])]\n\n # Convert sparse columns to dense\n for col in sparse_cols:\n meta_gene[col] = meta_gene[col].sparse.to_dense()\n\n meta_gene.to_parquet(path_output)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_trx_tiles","title":"<code>make_trx_tiles(technology, path_trx, path_transformation_matrix, path_trx_tiles, coarse_tile_factor=10, tile_size=250, chunk_size=1000000, verbose=False, image_scale=1, max_workers=8)</code>","text":"<p>Processes transcript data by dividing it into coarse-grain and fine-grain tiles, applying transformations, and saving the results in a parallelized manner.</p>"},{"location":"python/api/#celldega.pre.make_trx_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\"). path_trx : str Path to the file containing the transcript data. path_transformation_matrix : str Path to the file containing the transformation matrix (CSV file). path_trx_tiles : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional Scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional Size of each fine-grain tile in microns (default is 250). chunk_size : int, optional Number of rows to process per chunk for memory efficiency (default is 1000000). verbose : bool, optional Flag to enable verbose output (default is False). image_scale : float, optional Scale factor to apply to the transcript coordinates (default is 0.5). max_workers : int, optional Maximum number of parallel workers for processing tiles (default is 8).</p>"},{"location":"python/api/#celldega.pre.make_trx_tiles--returns","title":"Returns","text":"<p>dict A dictionary containing the bounds of the processed data in both x and y directions.</p> Source code in <code>src/celldega/pre/trx_tile.py</code> <pre><code>def make_trx_tiles(\n technology,\n path_trx,\n path_transformation_matrix,\n path_trx_tiles,\n coarse_tile_factor=10,\n tile_size=250,\n chunk_size=1000000,\n verbose=False,\n image_scale=1,\n max_workers=8\n):\n \"\"\"\n Processes transcript data by dividing it into coarse-grain and fine-grain tiles,\n applying transformations, and saving the results in a parallelized manner.\n\n Parameters\n ----------\n technology : str\n The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\").\n path_trx : str\n Path to the file containing the transcript data.\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV file).\n path_trx_tiles : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional\n Scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional\n Size of each fine-grain tile in microns (default is 250).\n chunk_size : int, optional\n Number of rows to process per chunk for memory efficiency (default is 1000000).\n verbose : bool, optional\n Flag to enable verbose output (default is False).\n image_scale : float, optional\n Scale factor to apply to the transcript coordinates (default is 0.5).\n max_workers : int, optional\n Maximum number of parallel workers for processing tiles (default is 8).\n\n Returns\n -------\n dict\n A dictionary containing the bounds of the processed data in both x and y directions.\n \"\"\"\n\n def process_coarse_tile(trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers):\n # Filter the entire dataset for the current coarse tile\n coarse_tile = trx.filter(\n (pl.col(\"transformed_x\") >= coarse_tile_x_min) & (pl.col(\"transformed_x\") < coarse_tile_x_max) &\n (pl.col(\"transformed_y\") >= coarse_tile_y_min) & (pl.col(\"transformed_y\") < coarse_tile_y_max)\n )\n\n if not coarse_tile.is_empty():\n # Now process fine tiles using global fine tile indices\n process_fine_tiles(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers) \n\n\n def process_fine_tiles(coarse_tile, coarse_i, coarse_j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers=8):\n\n # Use ThreadPoolExecutor for parallel processing of fine-grain tiles within the coarse tile\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n\n # Iterate over fine-grain tiles within the global bounds\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n # Submit the task for each fine tile to process in parallel\n futures.append(executor.submit(\n filter_and_save_fine_tile, coarse_tile, coarse_i, coarse_j, fine_i, fine_j, \n fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles\n ))\n\n # Wait for all futures to complete\n for future in concurrent.futures.as_completed(futures):\n future.result() # Raise exceptions if any occurred during execution\n\n\n def filter_and_save_fine_tile(coarse_tile, coarse_i, coarse_j, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles):\n\n # Filter the coarse tile for the current fine tile's boundaries\n fine_tile_trx = coarse_tile.filter(\n (pl.col(\"transformed_x\") >= fine_tile_x_min) & (pl.col(\"transformed_x\") < fine_tile_x_max) &\n (pl.col(\"transformed_y\") >= fine_tile_y_min) & (pl.col(\"transformed_y\") < fine_tile_y_max)\n )\n\n if not fine_tile_trx.is_empty():\n # Add geometry column as a list of [x, y] pairs\n fine_tile_trx = fine_tile_trx.with_columns(\n pl.concat_list([pl.col(\"transformed_x\"), pl.col(\"transformed_y\")]).alias(\"geometry\")\n ).drop(['transformed_x', 'transformed_y'])\n\n # Define the filename based on fine tile coordinates\n filename = f\"{path_trx_tiles}/transcripts_tile_{fine_i}_{fine_j}.parquet\"\n\n # Save the filtered DataFrame to a Parquet file\n fine_tile_trx.to_pandas().to_parquet(filename)\n\n\n # Load transformation matrix\n transformation_matrix = np.loadtxt(path_transformation_matrix)\n\n # Load the transcript data based on the technology using Polars\n if technology == \"MERSCOPE\":\n trx_ini = pl.read_csv(path_trx, columns=[\"gene\", \"global_x\", \"global_y\"])\n trx_ini = trx_ini.with_columns([\n pl.col(\"global_x\").alias(\"x\"),\n pl.col(\"global_y\").alias(\"y\"),\n pl.col(\"gene\").alias(\"name\")\n ]).select([\"name\", \"x\", \"y\"])\n\n elif technology == \"Xenium\":\n trx_ini = pl.read_parquet(path_trx).select([\n pl.col(\"feature_name\").alias(\"name\"),\n pl.col(\"x_location\").alias(\"x\"),\n pl.col(\"y_location\").alias(\"y\")\n ])\n\n # Process the data in chunks and apply transformations\n all_chunks = []\n\n for start_row in tqdm(range(0, trx_ini.height, chunk_size), desc=\"Processing chunks\"):\n chunk = trx_ini.slice(start_row, chunk_size)\n\n # Apply transformation matrix to the coordinates\n points = np.hstack([chunk.select([\"x\", \"y\"]).to_numpy(), np.ones((chunk.height, 1))])\n transformed_points = np.dot(points, transformation_matrix.T)[:, :2]\n\n # Create new transformed columns and drop original x, y columns\n transformed_chunk = chunk.with_columns([\n (pl.Series(transformed_points[:, 0]) * image_scale).round(2).alias(\"transformed_x\"),\n (pl.Series(transformed_points[:, 1]) * image_scale).round(2).alias(\"transformed_y\")\n ]).drop([\"x\", \"y\"])\n all_chunks.append(transformed_chunk)\n\n # Concatenate all chunks after processing\n trx = pl.concat(all_chunks)\n\n # Ensure the output directory exists\n if not os.path.exists(path_trx_tiles):\n os.makedirs(path_trx_tiles)\n\n # Get min and max x, y values\n x_min, x_max = trx.select([\n pl.col(\"transformed_x\").min().alias(\"x_min\"),\n pl.col(\"transformed_x\").max().alias(\"x_max\")\n ]).row(0)\n\n y_min, y_max = trx.select([\n pl.col(\"transformed_y\").min().alias(\"y_min\"),\n pl.col(\"transformed_y\").max().alias(\"y_max\")\n ]).row(0)\n\n # Calculate the number of fine-grain tiles globally\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n\n # Calculate the number of coarse-grain tiles\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Use ThreadPoolExecutor for parallel processing of coarse-grain tiles\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for i in range(n_coarse_tiles_x):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n # Submit each coarse tile for parallel processing\n futures.append(executor.submit(\n process_coarse_tile, trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers\n ))\n\n # Wait for all coarse tiles to complete\n for future in tqdm(concurrent.futures.as_completed(futures), desc=\"Processing coarse tiles\", unit=\"tile\"):\n future.result() # Raise exceptions if any occurred during execution\n\n # Return the tile bounds\n tile_bounds = {\n \"x_min\": x_min,\n \"x_max\": x_max,\n \"y_min\": y_min,\n \"y_max\": y_max,\n }\n\n return tile_bounds\n</code></pre>"},{"location":"python/api/#celldega.pre.reduce_image_size","title":"<code>reduce_image_size(image_path, scale_image=0.5, path_landscape_files='')</code>","text":""},{"location":"python/api/#celldega.pre.reduce_image_size--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file scale_image : float (default=0.5) Scale factor for the image resize</p>"},{"location":"python/api/#celldega.pre.reduce_image_size--returns","title":"Returns","text":"<p>new_image_path : str Path to the resized image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def reduce_image_size(image_path, scale_image=0.5, path_landscape_files=\"\"):\n \"\"\"\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n scale_image : float (default=0.5)\n Scale factor for the image resize\n\n Returns\n -------\n new_image_path : str\n Path to the resized image file\n \"\"\"\n\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n resized_image = image.resize(scale_image)\n\n new_image_name = image_path.split(\"/\")[-1].replace(\".tif\", \"_downsize.tif\")\n new_image_path = f\"{path_landscape_files}/{new_image_name}\"\n resized_image.write_to_file(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.save_landscape_parameters","title":"<code>save_landscape_parameters(technology, path_landscape_files, image_name='dapi_files', tile_size=1000, image_info={}, image_format='.webp')</code>","text":"<p>Save the landscape parameters to a JSON file.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def save_landscape_parameters(\n technology, path_landscape_files, image_name=\"dapi_files\", tile_size=1000, image_info={}, image_format='.webp'\n):\n \"\"\"\n Save the landscape parameters to a JSON file.\n \"\"\"\n\n path_image_pyramid = f\"{path_landscape_files}/pyramid_images/{image_name}\"\n\n print(path_image_pyramid)\n\n max_pyramid_zoom = get_max_zoom_level(path_image_pyramid)\n\n landscape_parameters = {\n \"technology\": technology,\n \"max_pyramid_zoom\": max_pyramid_zoom,\n \"tile_size\": tile_size,\n \"image_info\": image_info,\n \"image_format\": image_format\n }\n\n path_landscape_parameters = f\"{path_landscape_files}/landscape_parameters.json\"\n\n with open(path_landscape_parameters, \"w\") as file:\n json.dump(landscape_parameters, file, indent=4)\n</code></pre>"},{"location":"python/api/#celldega.viz.Landscape","title":"<code>Landscape</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of spatial omics data. This widget currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)</p> <p>Parameters:</p> Name Type Description Default <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> required <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> required <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> required <code>token</code> <code>str</code> <p>The token traitlet.</p> required <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> required <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize. This will show up in the user interface bar.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>technology</code> <code>str</code> <p>The technology used.</p> <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> <code>token</code> <code>str</code> <p>The token traitlet.</p> <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> <code>ini_z</code> <code>float</code> <p>The initial z-coordinate of the view.</p> <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize.</p> <code>update_trigger</code> <code>dict</code> <p>The dictionary to trigger updates.</p> <code>cell_clusters</code> <code>dict</code> <p>The dictionary containing cell cluster information.</p> <p>Returns:</p> Name Type Description <code>Landscape</code> <p>A widget for visualizing a 'landscape' view of spatial omics data.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Landscape(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of spatial omics data. This widget\n currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)\n\n Args:\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n token (str): The token traitlet.\n base_url (str): The base URL for the widget.\n dataset_name (str, optional): The name of the dataset to visualize. This will show up in the user interface bar.\n\n Attributes:\n component (str): The name of the component.\n technology (str): The technology used.\n base_url (str): The base URL for the widget.\n token (str): The token traitlet.\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_z (float): The initial z-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n dataset_name (str): The name of the dataset to visualize.\n update_trigger (dict): The dictionary to trigger updates.\n cell_clusters (dict): The dictionary containing cell cluster information.\n\n Returns:\n Landscape: A widget for visualizing a 'landscape' view of spatial omics data.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n component = traitlets.Unicode(\"Landscape\").tag(sync=True)\n\n technology = traitlets.Unicode(\"sst\").tag(sync=True)\n base_url = traitlets.Unicode(\"\").tag(sync=True)\n token = traitlets.Unicode(\"\").tag(sync=True)\n ini_x = traitlets.Float(1000).tag(sync=True)\n ini_y = traitlets.Float(1000).tag(sync=True)\n ini_z = traitlets.Float(0).tag(sync=True)\n ini_zoom = traitlets.Float(0).tag(sync=True)\n square_tile_size = traitlets.Float(1.4).tag(sync=True)\n dataset_name = traitlets.Unicode(\"\").tag(sync=True)\n region = traitlets.Dict({}).tag(sync=True)\n\n update_trigger = traitlets.Dict().tag(sync=True)\n cell_clusters = traitlets.Dict().tag(sync=True)\n\n width = traitlets.Int(0).tag(sync=True)\n height = traitlets.Int(800).tag(sync=True)\n\n def trigger_update(self, new_value):\n # This method updates the update_trigger traitlet with a new value\n # You can pass any information necessary for the update, or just a timestamp\n self.update_trigger = new_value\n\n def update_cell_clusters(self, new_clusters):\n # Convert the new_clusters to a JSON serializable format if necessary\n self.cell_clusters = new_clusters\n</code></pre>"},{"location":"python/api/#celldega.viz.Matrix","title":"<code>Matrix</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of a hierarchically clustered matrix.</p> <p>Parameters:</p> Name Type Description Default <code>value</code> <code>int</code> <p>The value traitlet.</p> required <code>component</code> <code>str</code> <p>The component traitlet.</p> required <code>network</code> <code>dict</code> <p>The network traitlet.</p> required <code>click_info</code> <code>dict</code> <p>The click_info traitlet.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>network</code> <code>dict</code> <p>The network dictionary.</p> <code>click_info</code> <code>dict</code> <p>The click_info dictionary.</p> <p>Returns:</p> Name Type Description <code>Matrix</code> <p>A widget for visualizing a hierarchically clustered matrix.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Matrix(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of a hierarchically clustered matrix.\n\n Args:\n value (int): The value traitlet.\n component (str): The component traitlet.\n network (dict): The network traitlet.\n click_info (dict): The click_info traitlet.\n\n Attributes:\n component (str): The name of the component.\n network (dict): The network dictionary.\n click_info (dict): The click_info dictionary.\n\n Returns:\n Matrix: A widget for visualizing a hierarchically clustered matrix.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n value = traitlets.Int(0).tag(sync=True)\n component = traitlets.Unicode(\"Matrix\").tag(sync=True)\n\n network = traitlets.Dict({}).tag(sync=True)\n click_info = traitlets.Dict({}).tag(sync=True)\n</code></pre>"},{"location":"python/pre/api/","title":"Pre Module API Reference","text":"<p>Module for pre-processing to generate LandscapeFiles from ST data.</p>"},{"location":"python/pre/api/#celldega.pre.convert_long_id_to_short","title":"<code>convert_long_id_to_short(df)</code>","text":"<p>Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>The DataFrame containing the EntityID.</p> required <p>Returns: pd.DataFrame: The original DataFrame with an additional column named <code>cell_id</code> containing the shortened cell IDs.</p> <p>The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates it to create a shorter identifier that is added as a new column to the DataFrame.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_long_id_to_short(df):\n \"\"\"\n Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.\n\n Args:\n df (pd.DataFrame): The DataFrame containing the EntityID.\n Returns:\n pd.DataFrame: The original DataFrame with an additional column named `cell_id`\n containing the shortened cell IDs.\n\n The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates\n it to create a shorter identifier that is added as a new column to the DataFrame.\n \"\"\"\n # Function to hash and encode the cell ID\n def hash_and_shorten_id(cell_id):\n # Create a hash of the cell ID\n cell_id_bytes = str(cell_id).encode('utf-8')\n hash_object = hashlib.sha256(cell_id_bytes)\n hash_digest = hash_object.digest()\n\n # Encode the hash to a base64 string to mix letters and numbers, truncate to 9 characters\n short_id = base64.urlsafe_b64encode(hash_digest).decode('utf-8')[:9]\n return short_id\n\n # Apply the hash_and_shorten_id function to each cell ID in the specified column\n df['cell_id'] = df['EntityID'].apply(hash_and_shorten_id)\n\n return df\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg","title":"<code>convert_to_jpeg(image_path, quality=80)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_jpeg(image_path, quality=80):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".jpeg\")\n image.jpegsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_png","title":"<code>convert_to_png(image_path)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_png--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_png--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_png(image_path):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".png\")\n image.pngsave(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp","title":"<code>convert_to_webp(image_path, quality=100)</code>","text":"<p>Convert a TIFF image to a WEBP image with a specified quality score.</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=100) Quality score for the WEBP image (higher is better quality)</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp--returns","title":"Returns","text":"<p>new_image_path : str Path to the WEBP image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_webp(image_path, quality=100):\n \"\"\"\n Convert a TIFF image to a WEBP image with a specified quality score.\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=100)\n Quality score for the WEBP image (higher is better quality)\n\n Returns\n -------\n new_image_path : str\n Path to the WEBP image file\n \"\"\"\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a WEBP with specified quality\n new_image_path = image_path.replace(\".tif\", \".webp\")\n image.webpsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.get_max_zoom_level","title":"<code>get_max_zoom_level(path_image_pyramid)</code>","text":"<p>Returns the maximum zoom level based on the highest-numbered directory in the specified path_image_pyramid.</p> <p>Parameters:</p> Name Type Description Default <code>path_image_pyramid</code> <code>str</code> <p>The path to the directory containing zoom level directories.</p> required <p>Returns:</p> Name Type Description <code>max_pyramid_zoom</code> <code>int</code> <p>The maximum zoom level.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def get_max_zoom_level(path_image_pyramid):\n \"\"\"\n Returns the maximum zoom level based on the highest-numbered directory\n in the specified path_image_pyramid.\n\n Parameters:\n path_image_pyramid (str): The path to the directory containing zoom level directories.\n\n Returns:\n max_pyramid_zoom (int): The maximum zoom level.\n \"\"\"\n # List all entries in the path_image_pyramid that are directories and can be converted to integers\n zoom_levels = [\n entry\n for entry in os.listdir(path_image_pyramid)\n if os.path.isdir(os.path.join(path_image_pyramid, entry)) and entry.isdigit()\n ]\n\n # Convert to integer and find the maximum value\n max_pyramid_zoom = max(map(int, zoom_levels)) if zoom_levels else None\n\n return max_pyramid_zoom\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles","title":"<code>make_cell_boundary_tiles(technology, path_cell_boundaries, path_meta_cell_micron, path_transformation_matrix, path_output, coarse_tile_factor=20, tile_size=250, tile_bounds=None, image_scale=1, max_workers=8)</code>","text":"<p>Processes cell boundary data and divides it into spatial tiles based on the provided technology. Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles. The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.</p>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\". path_cell_boundaries : str Path to the file containing the cell boundaries (Parquet format). path_meta_cell_micron : str Path to the file containing cell metadata (CSV format). path_transformation_matrix : str Path to the file containing the transformation matrix (CSV format). path_output : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional, default=20. scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional, default=500 Size of each fine-grain tile in microns. tile_bounds : dict, optional Dictionary containing the minimum and maximum bounds for x and y coordinates. image_scale : float, optional, default=1 Scale factor to apply to the geometry data. max_workers : int, optional, default=8 Maximum number of parallel workers for processing tiles.</p>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/boundary_tile.py</code> <pre><code>def make_cell_boundary_tiles(\n technology,\n path_cell_boundaries,\n path_meta_cell_micron,\n path_transformation_matrix,\n path_output,\n coarse_tile_factor=20,\n tile_size=250,\n tile_bounds=None,\n image_scale=1,\n max_workers=8\n):\n\n\n \"\"\"\n Processes cell boundary data and divides it into spatial tiles based on the provided technology.\n Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles.\n The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.\n\n Parameters\n ----------\n technology : str\n The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\".\n path_cell_boundaries : str\n Path to the file containing the cell boundaries (Parquet format).\n path_meta_cell_micron : str\n Path to the file containing cell metadata (CSV format).\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV format).\n path_output : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional, default=20.\n scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional, default=500\n Size of each fine-grain tile in microns.\n tile_bounds : dict, optional\n Dictionary containing the minimum and maximum bounds for x and y coordinates.\n image_scale : float, optional, default=1\n Scale factor to apply to the geometry data.\n max_workers : int, optional, default=8\n Maximum number of parallel workers for processing tiles.\n\n Returns\n -------\n None\n \"\"\"\n\n def numpy_affine_transform(coords, matrix):\n \"\"\"Apply affine transformation to numpy coordinates.\"\"\"\n # Homogeneous coordinates for affine transformation\n coords = np.hstack([coords, np.ones((coords.shape[0], 1))])\n transformed_coords = coords @ matrix.T\n return transformed_coords[:, :2] # Drop the homogeneous coordinate\n\n def batch_transform_geometries(geometries, transformation_matrix, scale):\n \"\"\"\n Batch transform geometries using numpy for optimized performance.\n \"\"\"\n # Extract affine transformation parameters into a 3x3 matrix for numpy\n affine_matrix = np.array([\n [transformation_matrix[0, 0], transformation_matrix[0, 1], transformation_matrix[0, 2]],\n [transformation_matrix[1, 0], transformation_matrix[1, 1], transformation_matrix[1, 2]],\n [0, 0, 1]\n ])\n\n transformed_geometries = []\n\n for polygon in geometries:\n # Extract coordinates and transform them\n if isinstance(polygon, MultiPolygon):\n polygon = next(polygon.geoms) # Use the first geometry\n\n # Transform the exterior of the polygon\n exterior_coords = np.array(polygon.exterior.coords)\n\n # Apply the affine transformation and scale\n transformed_coords = numpy_affine_transform(exterior_coords, affine_matrix) / scale\n\n # Append the result to the transformed_geometries list\n transformed_geometries.append([transformed_coords.tolist()])\n\n return transformed_geometries\n\n\n def filter_and_save_fine_boundary(coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output):\n cell_ids = coarse_tile.index.values\n\n tile_filter = (\n (coarse_tile[\"center_x\"] >= fine_tile_x_min) & (coarse_tile[\"center_x\"] < fine_tile_x_max) &\n (coarse_tile[\"center_y\"] >= fine_tile_y_min) & (coarse_tile[\"center_y\"] < fine_tile_y_max)\n )\n filtered_indices = np.where(tile_filter)[0]\n\n keep_cells = cell_ids[filtered_indices]\n fine_tile_cells = coarse_tile.loc[keep_cells, [\"GEOMETRY\"]]\n fine_tile_cells = fine_tile_cells.assign(name=fine_tile_cells.index)\n\n if not fine_tile_cells.empty:\n filename = f\"{path_output}/cell_tile_{fine_i}_{fine_j}.parquet\"\n fine_tile_cells.to_parquet(filename)\n\n def process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y):\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n futures.append(executor.submit(\n filter_and_save_fine_boundary, coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output\n ))\n\n for future in futures:\n future.result()\n\n tile_size_x = tile_size\n tile_size_y = tile_size\n\n transformation_matrix = pd.read_csv(path_transformation_matrix, header=None, sep=\" \").values\n\n # Load cell boundary data based on the technology\n if technology == \"MERSCOPE\":\n df_meta = pd.read_parquet(f\"{path_output.replace('cell_segmentation','cell_metadata.parquet')}\")\n entity_to_cell_id_dict = pd.Series(df_meta.index.values, index=df_meta.EntityID).to_dict()\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n cells_orig['cell_id'] = cells_orig['EntityID'].map(entity_to_cell_id_dict)\n cells_orig = cells_orig[cells_orig[\"ZIndex\"] == 1]\n\n # Correct cell_id issues with meta_cell\n meta_cell = pd.read_csv(path_meta_cell_micron)\n meta_cell['cell_id'] = meta_cell['EntityID'].map(entity_to_cell_id_dict)\n cells_orig.index = meta_cell[meta_cell[\"cell_id\"].isin(cells_orig['cell_id'])].index\n\n # Correct 'MultiPolygon' to 'Polygon'\n cells_orig[\"geometry\"] = cells_orig[\"Geometry\"].apply(\n lambda x: list(x.geoms)[0] if isinstance(x, MultiPolygon) else x\n )\n\n cells_orig.set_index('cell_id', inplace=True)\n\n elif technology == \"Xenium\":\n xenium_cells = pd.read_parquet(path_cell_boundaries)\n grouped = xenium_cells.groupby(\"cell_id\")[[\"vertex_x\", \"vertex_y\"]].agg(lambda x: x.tolist())\n grouped[\"geometry\"] = grouped.apply(lambda row: Polygon(zip(row[\"vertex_x\"], row[\"vertex_y\"])), axis=1)\n cells_orig = gpd.GeoDataFrame(grouped, geometry=\"geometry\")[[\"geometry\"]]\n\n elif technology == \"custom\":\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n\n # Transform geometries\n cells_orig[\"GEOMETRY\"] = batch_transform_geometries(cells_orig[\"geometry\"], transformation_matrix, image_scale)\n\n # Convert transformed geometries to polygons and calculate centroids\n cells_orig[\"polygon\"] = cells_orig[\"GEOMETRY\"].apply(lambda x: Polygon(x[0]))\n gdf_cells = gpd.GeoDataFrame(geometry=cells_orig[\"polygon\"])\n gdf_cells[\"center_x\"] = gdf_cells.geometry.centroid.x\n gdf_cells[\"center_y\"] = gdf_cells.geometry.centroid.y\n gdf_cells[\"GEOMETRY\"] = cells_orig[\"GEOMETRY\"]\n\n # Ensure the output directory exists\n if not os.path.exists(path_output):\n os.makedirs(path_output)\n\n # Calculate tile bounds and fine/coarse tiles\n x_min, x_max = tile_bounds[\"x_min\"], tile_bounds[\"x_max\"]\n y_min, y_max = tile_bounds[\"y_min\"], tile_bounds[\"y_max\"]\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Process coarse tiles in parallel\n for i in tqdm(range(n_coarse_tiles_x), desc=\"Processing coarse tiles\"):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n coarse_tile = gdf_cells[\n (gdf_cells[\"center_x\"] >= coarse_tile_x_min) & (gdf_cells[\"center_x\"] < coarse_tile_x_max) &\n (gdf_cells[\"center_y\"] >= coarse_tile_y_min) & (gdf_cells[\"center_y\"] < coarse_tile_y_max)\n ]\n if not coarse_tile.empty:\n process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid","title":"<code>make_deepzoom_pyramid(image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix='.jpeg')</code>","text":"<p>Create a DeepZoom image pyramid from a JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid--parameters","title":"Parameters","text":"<p>image_path : str Path to the JPEG image file tile_size : int (default=512) Tile size for the DeepZoom pyramid overlap : int (default=0) Overlap size for the DeepZoom pyramid suffix : str (default='jpeg') Suffix for the DeepZoom pyramid tiles</p>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_deepzoom_pyramid(\n image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix=\".jpeg\"\n):\n \"\"\"\n Create a DeepZoom image pyramid from a JPEG image\n\n Parameters\n ----------\n image_path : str\n Path to the JPEG image file\n tile_size : int (default=512)\n Tile size for the DeepZoom pyramid\n overlap : int (default=0)\n Overlap size for the DeepZoom pyramid\n suffix : str (default='jpeg')\n Suffix for the DeepZoom pyramid tiles\n\n Returns\n -------\n None\n\n \"\"\"\n\n # Define the output path\n output_path = Path(output_path)\n\n # Load the JPEG image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # check if the output path exists and create it if it does not\n output_path.mkdir(parents=True, exist_ok=True)\n\n # append the pyramid name to the output path\n output_path = output_path / pyramid_name\n\n # Save the image as a DeepZoom image pyramid\n image.dzsave(output_path, tile_size=tile_size, overlap=overlap, suffix=suffix)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord","title":"<code>make_meta_cell_image_coord(technology, path_transformation_matrix, path_meta_cell_micron, path_meta_cell_image, image_scale)</code>","text":"<p>Apply an affine transformation to the cell coordinates in microns and save the transformed coordinates in pixels</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_transformation_matrix : str Path to the transformation matrix file path_meta_cell_micron : str Path to the meta cell file with coordinates in microns path_meta_cell_image : str Path to save the meta cell file with coordinates in pixels</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--examples","title":"Examples","text":"<p>make_meta_cell_image_coord( ... technology='Xenium', ... path_transformation_matrix='data/transformation_matrix.txt', ... path_meta_cell_micron='data/meta_cell_micron.csv', ... path_meta_cell_image='data/meta_cell_image.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_cell_image_coord(\n technology,\n path_transformation_matrix,\n path_meta_cell_micron,\n path_meta_cell_image,\n image_scale\n):\n \"\"\"\n Apply an affine transformation to the cell coordinates in microns and save\n the transformed coordinates in pixels\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_transformation_matrix : str\n Path to the transformation matrix file\n path_meta_cell_micron : str\n Path to the meta cell file with coordinates in microns\n path_meta_cell_image : str\n Path to save the meta cell file with coordinates in pixels\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_cell_image_coord(\n ... technology='Xenium',\n ... path_transformation_matrix='data/transformation_matrix.txt',\n ... path_meta_cell_micron='data/meta_cell_micron.csv',\n ... path_meta_cell_image='data/meta_cell_image.parquet'\n ... )\n\n \"\"\"\n\n transformation_matrix = pd.read_csv(\n path_transformation_matrix, header=None, sep=\" \"\n ).values\n\n if technology == \"MERSCOPE\":\n meta_cell = pd.read_csv(path_meta_cell_micron, usecols=[\"EntityID\", \"center_x\", \"center_y\"])\n meta_cell = convert_long_id_to_short(meta_cell)\n meta_cell[\"name\"] = meta_cell[\"cell_id\"]\n meta_cell = meta_cell.set_index('cell_id')\n elif technology == \"Xenium\":\n usecols = [\"cell_id\", \"x_centroid\", \"y_centroid\"]\n meta_cell = pd.read_csv(path_meta_cell_micron, index_col=0, usecols=usecols)\n meta_cell.columns = [\"center_x\", \"center_y\"]\n meta_cell[\"name\"] = pd.Series(meta_cell.index, index=meta_cell.index)\n\n # Adding a ones column to accommodate for affine transformation\n meta_cell[\"ones\"] = 1\n\n # Preparing the data for matrix multiplication\n points = meta_cell[[\"center_x\", \"center_y\", \"ones\"]].values\n\n # Applying the transformation matrix\n transformed_points = np.dot(transformation_matrix, points.T).T\n\n # Updating the DataFrame with transformed coordinates\n meta_cell[\"center_x\"] = transformed_points[:, 0]\n meta_cell[\"center_y\"] = transformed_points[:, 1]\n\n # Dropping the ones column as it's no longer needed\n meta_cell.drop(columns=[\"ones\"], inplace=True)\n\n meta_cell[\"center_x\"] = meta_cell[\"center_x\"] / image_scale\n meta_cell[\"center_y\"] = meta_cell[\"center_y\"] / image_scale\n\n meta_cell[\"geometry\"] = meta_cell.apply(\n lambda row: [row[\"center_x\"], row[\"center_y\"]], axis=1\n )\n\n if technology == \"MERSCOPE\":\n meta_cell = meta_cell[[\"name\", \"geometry\", \"EntityID\"]]\n else:\n meta_cell = meta_cell[[\"name\", \"geometry\"]]\n\n\n meta_cell.to_parquet(path_meta_cell_image)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene","title":"<code>make_meta_gene(technology, path_cbg, path_output)</code>","text":"<p>Create a DataFrame with genes and their assigned colors</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_cbg : str Path to the cell-by-gene matrix data (the data format can vary based on technology) path_output : str Path to save the meta gene file</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--examples","title":"Examples","text":"<p>make_meta_gene( ... technology='Xenium', ... path_cbg='data/', ... path_output='data/meta_gene.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_gene(technology, path_cbg, path_output):\n \"\"\"\n Create a DataFrame with genes and their assigned colors\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_cbg : str\n Path to the cell-by-gene matrix data (the data format can vary based on technology)\n path_output : str\n Path to save the meta gene file\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_gene(\n ... technology='Xenium',\n ... path_cbg='data/',\n ... path_output='data/meta_gene.parquet'\n ... )\n \"\"\"\n\n if technology == \"MERSCOPE\":\n cbg = pd.read_csv(path_cbg, index_col=0)\n genes = cbg.columns.tolist()\n elif technology == \"Xenium\":\n # genes = pd.read_csv(path_cbg + 'features.tsv.gz', sep='\\t', header=None)[1].values.tolist()\n cbg = read_cbg_mtx(path_cbg)\n genes = cbg.columns.tolist()\n\n # Get all categorical color palettes from Matplotlib and flatten them into a single list of colors\n palettes = [plt.get_cmap(name).colors for name in plt.colormaps() if \"tab\" in name]\n flat_colors = [color for palette in palettes for color in palette]\n\n # Convert RGB tuples to hex codes\n flat_colors_hex = [to_hex(color) for color in flat_colors]\n\n # Use modular arithmetic to assign a color to each gene, white for genes with \"Blank\"\n colors = [\n flat_colors_hex[i % len(flat_colors_hex)] if \"Blank\" not in gene else \"#FFFFFF\"\n for i, gene in enumerate(genes)\n ]\n\n # Create a DataFrame with genes and their assigned colors\n ser_color = pd.Series(colors, index=genes)\n\n # calculate gene expression metadata\n meta_gene = calc_meta_gene_data(cbg)\n meta_gene['color'] = ser_color\n\n # Identify sparse columns\n sparse_cols = [col for col in meta_gene.columns if pd.api.types.is_sparse(meta_gene[col])]\n\n # Convert sparse columns to dense\n for col in sparse_cols:\n meta_gene[col] = meta_gene[col].sparse.to_dense()\n\n meta_gene.to_parquet(path_output)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles","title":"<code>make_trx_tiles(technology, path_trx, path_transformation_matrix, path_trx_tiles, coarse_tile_factor=10, tile_size=250, chunk_size=1000000, verbose=False, image_scale=1, max_workers=8)</code>","text":"<p>Processes transcript data by dividing it into coarse-grain and fine-grain tiles, applying transformations, and saving the results in a parallelized manner.</p>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\"). path_trx : str Path to the file containing the transcript data. path_transformation_matrix : str Path to the file containing the transformation matrix (CSV file). path_trx_tiles : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional Scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional Size of each fine-grain tile in microns (default is 250). chunk_size : int, optional Number of rows to process per chunk for memory efficiency (default is 1000000). verbose : bool, optional Flag to enable verbose output (default is False). image_scale : float, optional Scale factor to apply to the transcript coordinates (default is 0.5). max_workers : int, optional Maximum number of parallel workers for processing tiles (default is 8).</p>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles--returns","title":"Returns","text":"<p>dict A dictionary containing the bounds of the processed data in both x and y directions.</p> Source code in <code>src/celldega/pre/trx_tile.py</code> <pre><code>def make_trx_tiles(\n technology,\n path_trx,\n path_transformation_matrix,\n path_trx_tiles,\n coarse_tile_factor=10,\n tile_size=250,\n chunk_size=1000000,\n verbose=False,\n image_scale=1,\n max_workers=8\n):\n \"\"\"\n Processes transcript data by dividing it into coarse-grain and fine-grain tiles,\n applying transformations, and saving the results in a parallelized manner.\n\n Parameters\n ----------\n technology : str\n The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\").\n path_trx : str\n Path to the file containing the transcript data.\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV file).\n path_trx_tiles : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional\n Scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional\n Size of each fine-grain tile in microns (default is 250).\n chunk_size : int, optional\n Number of rows to process per chunk for memory efficiency (default is 1000000).\n verbose : bool, optional\n Flag to enable verbose output (default is False).\n image_scale : float, optional\n Scale factor to apply to the transcript coordinates (default is 0.5).\n max_workers : int, optional\n Maximum number of parallel workers for processing tiles (default is 8).\n\n Returns\n -------\n dict\n A dictionary containing the bounds of the processed data in both x and y directions.\n \"\"\"\n\n def process_coarse_tile(trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers):\n # Filter the entire dataset for the current coarse tile\n coarse_tile = trx.filter(\n (pl.col(\"transformed_x\") >= coarse_tile_x_min) & (pl.col(\"transformed_x\") < coarse_tile_x_max) &\n (pl.col(\"transformed_y\") >= coarse_tile_y_min) & (pl.col(\"transformed_y\") < coarse_tile_y_max)\n )\n\n if not coarse_tile.is_empty():\n # Now process fine tiles using global fine tile indices\n process_fine_tiles(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers) \n\n\n def process_fine_tiles(coarse_tile, coarse_i, coarse_j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers=8):\n\n # Use ThreadPoolExecutor for parallel processing of fine-grain tiles within the coarse tile\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n\n # Iterate over fine-grain tiles within the global bounds\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n # Submit the task for each fine tile to process in parallel\n futures.append(executor.submit(\n filter_and_save_fine_tile, coarse_tile, coarse_i, coarse_j, fine_i, fine_j, \n fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles\n ))\n\n # Wait for all futures to complete\n for future in concurrent.futures.as_completed(futures):\n future.result() # Raise exceptions if any occurred during execution\n\n\n def filter_and_save_fine_tile(coarse_tile, coarse_i, coarse_j, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles):\n\n # Filter the coarse tile for the current fine tile's boundaries\n fine_tile_trx = coarse_tile.filter(\n (pl.col(\"transformed_x\") >= fine_tile_x_min) & (pl.col(\"transformed_x\") < fine_tile_x_max) &\n (pl.col(\"transformed_y\") >= fine_tile_y_min) & (pl.col(\"transformed_y\") < fine_tile_y_max)\n )\n\n if not fine_tile_trx.is_empty():\n # Add geometry column as a list of [x, y] pairs\n fine_tile_trx = fine_tile_trx.with_columns(\n pl.concat_list([pl.col(\"transformed_x\"), pl.col(\"transformed_y\")]).alias(\"geometry\")\n ).drop(['transformed_x', 'transformed_y'])\n\n # Define the filename based on fine tile coordinates\n filename = f\"{path_trx_tiles}/transcripts_tile_{fine_i}_{fine_j}.parquet\"\n\n # Save the filtered DataFrame to a Parquet file\n fine_tile_trx.to_pandas().to_parquet(filename)\n\n\n # Load transformation matrix\n transformation_matrix = np.loadtxt(path_transformation_matrix)\n\n # Load the transcript data based on the technology using Polars\n if technology == \"MERSCOPE\":\n trx_ini = pl.read_csv(path_trx, columns=[\"gene\", \"global_x\", \"global_y\"])\n trx_ini = trx_ini.with_columns([\n pl.col(\"global_x\").alias(\"x\"),\n pl.col(\"global_y\").alias(\"y\"),\n pl.col(\"gene\").alias(\"name\")\n ]).select([\"name\", \"x\", \"y\"])\n\n elif technology == \"Xenium\":\n trx_ini = pl.read_parquet(path_trx).select([\n pl.col(\"feature_name\").alias(\"name\"),\n pl.col(\"x_location\").alias(\"x\"),\n pl.col(\"y_location\").alias(\"y\")\n ])\n\n # Process the data in chunks and apply transformations\n all_chunks = []\n\n for start_row in tqdm(range(0, trx_ini.height, chunk_size), desc=\"Processing chunks\"):\n chunk = trx_ini.slice(start_row, chunk_size)\n\n # Apply transformation matrix to the coordinates\n points = np.hstack([chunk.select([\"x\", \"y\"]).to_numpy(), np.ones((chunk.height, 1))])\n transformed_points = np.dot(points, transformation_matrix.T)[:, :2]\n\n # Create new transformed columns and drop original x, y columns\n transformed_chunk = chunk.with_columns([\n (pl.Series(transformed_points[:, 0]) * image_scale).round(2).alias(\"transformed_x\"),\n (pl.Series(transformed_points[:, 1]) * image_scale).round(2).alias(\"transformed_y\")\n ]).drop([\"x\", \"y\"])\n all_chunks.append(transformed_chunk)\n\n # Concatenate all chunks after processing\n trx = pl.concat(all_chunks)\n\n # Ensure the output directory exists\n if not os.path.exists(path_trx_tiles):\n os.makedirs(path_trx_tiles)\n\n # Get min and max x, y values\n x_min, x_max = trx.select([\n pl.col(\"transformed_x\").min().alias(\"x_min\"),\n pl.col(\"transformed_x\").max().alias(\"x_max\")\n ]).row(0)\n\n y_min, y_max = trx.select([\n pl.col(\"transformed_y\").min().alias(\"y_min\"),\n pl.col(\"transformed_y\").max().alias(\"y_max\")\n ]).row(0)\n\n # Calculate the number of fine-grain tiles globally\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n\n # Calculate the number of coarse-grain tiles\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Use ThreadPoolExecutor for parallel processing of coarse-grain tiles\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for i in range(n_coarse_tiles_x):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n # Submit each coarse tile for parallel processing\n futures.append(executor.submit(\n process_coarse_tile, trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers\n ))\n\n # Wait for all coarse tiles to complete\n for future in tqdm(concurrent.futures.as_completed(futures), desc=\"Processing coarse tiles\", unit=\"tile\"):\n future.result() # Raise exceptions if any occurred during execution\n\n # Return the tile bounds\n tile_bounds = {\n \"x_min\": x_min,\n \"x_max\": x_max,\n \"y_min\": y_min,\n \"y_max\": y_max,\n }\n\n return tile_bounds\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.reduce_image_size","title":"<code>reduce_image_size(image_path, scale_image=0.5, path_landscape_files='')</code>","text":""},{"location":"python/pre/api/#celldega.pre.reduce_image_size--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file scale_image : float (default=0.5) Scale factor for the image resize</p>"},{"location":"python/pre/api/#celldega.pre.reduce_image_size--returns","title":"Returns","text":"<p>new_image_path : str Path to the resized image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def reduce_image_size(image_path, scale_image=0.5, path_landscape_files=\"\"):\n \"\"\"\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n scale_image : float (default=0.5)\n Scale factor for the image resize\n\n Returns\n -------\n new_image_path : str\n Path to the resized image file\n \"\"\"\n\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n resized_image = image.resize(scale_image)\n\n new_image_name = image_path.split(\"/\")[-1].replace(\".tif\", \"_downsize.tif\")\n new_image_path = f\"{path_landscape_files}/{new_image_name}\"\n resized_image.write_to_file(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.save_landscape_parameters","title":"<code>save_landscape_parameters(technology, path_landscape_files, image_name='dapi_files', tile_size=1000, image_info={}, image_format='.webp')</code>","text":"<p>Save the landscape parameters to a JSON file.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def save_landscape_parameters(\n technology, path_landscape_files, image_name=\"dapi_files\", tile_size=1000, image_info={}, image_format='.webp'\n):\n \"\"\"\n Save the landscape parameters to a JSON file.\n \"\"\"\n\n path_image_pyramid = f\"{path_landscape_files}/pyramid_images/{image_name}\"\n\n print(path_image_pyramid)\n\n max_pyramid_zoom = get_max_zoom_level(path_image_pyramid)\n\n landscape_parameters = {\n \"technology\": technology,\n \"max_pyramid_zoom\": max_pyramid_zoom,\n \"tile_size\": tile_size,\n \"image_info\": image_info,\n \"image_format\": image_format\n }\n\n path_landscape_parameters = f\"{path_landscape_files}/landscape_parameters.json\"\n\n with open(path_landscape_parameters, \"w\") as file:\n json.dump(landscape_parameters, file, indent=4)\n</code></pre>"},{"location":"python/viz/api/","title":"Viz Module API Reference","text":""},{"location":"python/viz/api/#widget-classes","title":"Widget Classes","text":""},{"location":"python/viz/api/#celldega.viz.widget.Landscape","title":"<code>Landscape</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of spatial omics data. This widget currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)</p> <p>Parameters:</p> Name Type Description Default <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> required <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> required <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> required <code>token</code> <code>str</code> <p>The token traitlet.</p> required <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> required <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize. This will show up in the user interface bar.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>technology</code> <code>str</code> <p>The technology used.</p> <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> <code>token</code> <code>str</code> <p>The token traitlet.</p> <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> <code>ini_z</code> <code>float</code> <p>The initial z-coordinate of the view.</p> <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize.</p> <code>update_trigger</code> <code>dict</code> <p>The dictionary to trigger updates.</p> <code>cell_clusters</code> <code>dict</code> <p>The dictionary containing cell cluster information.</p> <p>Returns:</p> Name Type Description <code>Landscape</code> <p>A widget for visualizing a 'landscape' view of spatial omics data.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Landscape(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of spatial omics data. This widget\n currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)\n\n Args:\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n token (str): The token traitlet.\n base_url (str): The base URL for the widget.\n dataset_name (str, optional): The name of the dataset to visualize. This will show up in the user interface bar.\n\n Attributes:\n component (str): The name of the component.\n technology (str): The technology used.\n base_url (str): The base URL for the widget.\n token (str): The token traitlet.\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_z (float): The initial z-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n dataset_name (str): The name of the dataset to visualize.\n update_trigger (dict): The dictionary to trigger updates.\n cell_clusters (dict): The dictionary containing cell cluster information.\n\n Returns:\n Landscape: A widget for visualizing a 'landscape' view of spatial omics data.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n component = traitlets.Unicode(\"Landscape\").tag(sync=True)\n\n technology = traitlets.Unicode(\"sst\").tag(sync=True)\n base_url = traitlets.Unicode(\"\").tag(sync=True)\n token = traitlets.Unicode(\"\").tag(sync=True)\n ini_x = traitlets.Float(1000).tag(sync=True)\n ini_y = traitlets.Float(1000).tag(sync=True)\n ini_z = traitlets.Float(0).tag(sync=True)\n ini_zoom = traitlets.Float(0).tag(sync=True)\n square_tile_size = traitlets.Float(1.4).tag(sync=True)\n dataset_name = traitlets.Unicode(\"\").tag(sync=True)\n region = traitlets.Dict({}).tag(sync=True)\n\n update_trigger = traitlets.Dict().tag(sync=True)\n cell_clusters = traitlets.Dict().tag(sync=True)\n\n width = traitlets.Int(0).tag(sync=True)\n height = traitlets.Int(800).tag(sync=True)\n\n def trigger_update(self, new_value):\n # This method updates the update_trigger traitlet with a new value\n # You can pass any information necessary for the update, or just a timestamp\n self.update_trigger = new_value\n\n def update_cell_clusters(self, new_clusters):\n # Convert the new_clusters to a JSON serializable format if necessary\n self.cell_clusters = new_clusters\n</code></pre>"},{"location":"python/viz/api/#celldega.viz.widget.Matrix","title":"<code>Matrix</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of a hierarchically clustered matrix.</p> <p>Parameters:</p> Name Type Description Default <code>value</code> <code>int</code> <p>The value traitlet.</p> required <code>component</code> <code>str</code> <p>The component traitlet.</p> required <code>network</code> <code>dict</code> <p>The network traitlet.</p> required <code>click_info</code> <code>dict</code> <p>The click_info traitlet.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>network</code> <code>dict</code> <p>The network dictionary.</p> <code>click_info</code> <code>dict</code> <p>The click_info dictionary.</p> <p>Returns:</p> Name Type Description <code>Matrix</code> <p>A widget for visualizing a hierarchically clustered matrix.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Matrix(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of a hierarchically clustered matrix.\n\n Args:\n value (int): The value traitlet.\n component (str): The component traitlet.\n network (dict): The network traitlet.\n click_info (dict): The click_info traitlet.\n\n Attributes:\n component (str): The name of the component.\n network (dict): The network dictionary.\n click_info (dict): The click_info dictionary.\n\n Returns:\n Matrix: A widget for visualizing a hierarchically clustered matrix.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n value = traitlets.Int(0).tag(sync=True)\n component = traitlets.Unicode(\"Matrix\").tag(sync=True)\n\n network = traitlets.Dict({}).tag(sync=True)\n click_info = traitlets.Dict({}).tag(sync=True)\n</code></pre>"},{"location":"technologies/","title":"Technologies","text":"<p>Celldega is utilizing a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit.</p>"},{"location":"technologies/#visualization-technologies","title":"Visualization Technologies","text":"<p>Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is easily linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega <code>viz</code> module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio).</p> <p>The Celldega Landscape visualization method (see Gallery) utilizes novel vector tiling approaches to enable interactive visualization of large ST datasets in a notebook environment or as a stand-alone webpage. This approach allows Celldega to visualize larger datasets than currently available open-source tools (e.g., datasets with hundreds of millions of transcripts). We also utilize modern web image data formats (WebP) to reduce the data storage burden for interactive visualization. The resulting LandscapeFiles data format serves as a compact and highly performant visualization-specific data format.</p>"},{"location":"technologies/#terrabio","title":"Terra.bio","text":"<p>Terra.bio is a cloud-based compute and data storage platform that is being developed by the Broad Institute of MIT and Harvard. We are utilizing Terra.bio to help Spatial Technology Platform clients access, analyze, and visualize their ST data.</p>"},{"location":"technologies/#jupyter-widget","title":"Jupyter Widget","text":"<p>We utilize the Jupyter Widget ecosystem to build interactive spatial and data visualizations that enable users to perform two way communication between JavaScript (front-end) and Python (back-end). We are utilizing the AnyWidget implementation to build our custom widgets.</p>"},{"location":"technologies/#deckgl","title":"Deck.gl","text":"<p>Celldega uses the GPU-powered data visualization library deck.gl to create high-performance spatial- and data-visualizations.</p>"},{"location":"technologies/#apache-parquet","title":"Apache Parquet","text":"<p>Celldega uses the Apache Parquet file format for storing spatial data. This file format in combination with the JavaScript library ParquetWASM and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see GeoArrow and GeoParquet in deck.gl).</p>"},{"location":"technologies/#parquetwasm-and-apache-arrow","title":"ParquetWASM and Apache Arrow","text":"<p>ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a complementary columnar in-memory format for storing data from Apache Parquet files. See GeoArrow and GeoParquet in deck.gl for more information.</p>"},{"location":"technologies/#webp","title":"WebP","text":""},{"location":"technologies/#deepzoom","title":"DeepZoom","text":""},{"location":"technologies/#clustergrammer-visualization","title":"Clustergrammer Visualization","text":""},{"location":"technologies/#data-analysis-technologies","title":"Data Analysis Technologies","text":""},{"location":"technologies/#geopandas","title":"GeoPandas","text":""},{"location":"technologies/#libpysal-python-spatial-analysis-library-core","title":"LibPySal: Python Spatial Analysis Library Core","text":""},{"location":"technologies/#clustergrammer-data-analysis","title":"Clustergrammer Data Analysis","text":""}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Welcome to Celldega's Documentation","text":"Celldega Landscape visualization of a human skin cancer Xenium dataset obtained from 10X Genomics. <p>Celldega is a spatial analysis and visualization library that is being developed by the Spatial Technology Platform at the Broad Institute of MIT and Harvard. This project enables researchers to easily visualize large ST datasets (e.g., datasets with >100M transcripts) alongside single-cell and spatial analysis notebook workflows (e.g., sverse tools and novel spatial analysis approaches).</p> <ul> <li>Getting Started</li> <li>Installation</li> <li>Usage</li> </ul>"},{"location":"#about","title":"About","text":"<p>Celldega is named after a bodega, a small shop with all the essentials that is part of the fabric of a neighborhood.</p>"},{"location":"examples/","title":"Jupyter Notebook Examples","text":"<p>Landscape View Xenium</p>"},{"location":"examples/short_notebooks/Landscape_View_Xenium/","title":"Landscape View Xenium","text":"In\u00a0[2]: Copied! <pre># %load_ext autoreload\n# %autoreload 2\n# %env ANYWIDGET_HMR=1\n</pre> # %load_ext autoreload # %autoreload 2 # %env ANYWIDGET_HMR=1 In\u00a0[3]: Copied! <pre>import celldega as dega\ndega.__version__\n</pre> import celldega as dega dega.__version__ Out[3]: <pre>'0.0.0'</pre> In\u00a0[6]: Copied! <pre>from observable_jupyter import embed\n</pre> from observable_jupyter import embed In\u00a0[11]: Copied! <pre>base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Mouse_Brain_Coronal_FF_outs/main/Xenium_Prime_Mouse_Brain_Coronal_FF_outs'\n</pre> base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Mouse_Brain_Coronal_FF_outs/main/Xenium_Prime_Mouse_Brain_Coronal_FF_outs' In\u00a0[14]: Copied! <pre>embed('@cornhundred/celldega-landscape-ist', inputs={'base_url': base_url}, cells=['landscape_container'], display_logo=False)\n</pre> embed('@cornhundred/celldega-landscape-ist', inputs={'base_url': base_url}, cells=['landscape_container'], display_logo=False) In\u00a0[13]: Copied! <pre># base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Human_Skin_FFPE_outs/main/Xenium_Prime_Human_Skin_FFPE_outs'\n\n# landscape_ist = dega.viz.Landscape(\n# technology='Xenium',\n# ini_zoom = -4.5,\n# ini_x=6000,\n# ini_y=8000,\n# base_url = base_url,\n\n# )\n\n# landscape_ist\n</pre> # base_url = 'https://raw.githubusercontent.com/broadinstitute/celldega_Xenium_Prime_Human_Skin_FFPE_outs/main/Xenium_Prime_Human_Skin_FFPE_outs' # landscape_ist = dega.viz.Landscape( # technology='Xenium', # ini_zoom = -4.5, # ini_x=6000, # ini_y=8000, # base_url = base_url, # ) # landscape_ist In\u00a0[\u00a0]: Copied! <pre>\n</pre>"},{"location":"examples/short_notebooks/Landscape_View_Xenium/#landscape-view-xenium","title":"Landscape View Xenium\u00b6","text":""},{"location":"gallery/","title":"Celldega Gallery","text":"<p>This page includes links to visualizations that are made with the stand-alone Celldega JavaScript library.</p>"},{"location":"gallery/#imaging-spatial-transcriptomics","title":"Imaging Spatial Transcriptomics","text":""},{"location":"gallery/#xenium","title":"Xenium","text":"<ul> <li>Xenium Mouse Brain </li> <li>Xenium Human Skin Cancer </li> </ul>"},{"location":"gallery/#sequencing-spatial-transcriptomics","title":"Sequencing Spatial Transcriptomics","text":""},{"location":"gallery/#visium-hd","title":"Visium HD","text":""},{"location":"gallery/gallery_xenium/","title":"Celldega Xenium Gallery","text":""},{"location":"gallery/gallery_xenium/#xenium-prime-mouse-brain-coronal-ff","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium/#xenium-prime-human-skin-ffpe-outs","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"gallery/gallery_xenium/#xenium-human-pancreas-ffpe","title":"Xenium Human Pancreas FFPE","text":""},{"location":"gallery/gallery_xenium/#bone-marrow","title":"Bone Marrow","text":""},{"location":"gallery/gallery_xenium_mouse_brain/","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium_multi/","title":"Xenium Multi Dataset","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-prime-mouse-brain-coronal-ff","title":"Xenium Prime Mouse Brain Coronal FF","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-prime-human-skin-ffpe-outs","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"gallery/gallery_xenium_multi/#xenium-human-pancreas-ffpe","title":"Xenium Human Pancreas FFPE","text":""},{"location":"gallery/gallery_xenium_multi/#bone-marrow","title":"Bone Marrow","text":""},{"location":"gallery/gallery_xenium_skin_cancer/","title":"Xenium Prime Human Skin FFPE outs","text":""},{"location":"javascript/","title":"JavaScript API Overview","text":"<p>Celldega's visualization methods can be used as a stand-alone JavaScript library outside the context of a Jupyter notebook. This can be used to create showcase visualizations with publicly hosted data.</p>"},{"location":"javascript/api/","title":"Celldega JavaScript API Documentation","text":"<p>The JavaScript component of Celldega is used within the Jupyter Widgets framework to provide interactive visualization in the context of a Jupyter notebook but can also be used as a standalone JavaScript library.</p>"},{"location":"javascript/api/#landscape_ist-api-documentation","title":"<code>landscape_ist</code> API Documentation","text":"<p>The <code>landscape_ist</code> function initializes and renders an interactive spatial transcriptomics (IST) landscape visualization. This API is designed to work with Deck.gl and includes customizable visualization options, dynamic data updates, and UI interactions.</p>"},{"location":"javascript/api/#parameters","title":"Parameters","text":"<ul> <li><code>el</code> (<code>HTMLElement</code>): The root DOM element where the visualization is rendered.</li> <li><code>ini_model</code> (<code>Object</code>): The initial data model containing configuration and state.</li> <li><code>token</code> (<code>string</code>): Authentication token for accessing data.</li> <li><code>ini_x</code>, <code>ini_y</code>, <code>ini_z</code> (<code>number</code>): Initial spatial coordinates for the view.</li> <li><code>ini_zoom</code> (<code>number</code>): Initial zoom level for the visualization.</li> <li><code>base_url</code> (<code>string</code>): Base URL for accessing data files.</li> <li><code>dataset_name</code> (<code>string</code>, optional): Name of the dataset being visualized.</li> <li><code>trx_radius</code> (<code>number</code>, optional): Initial radius for transcript points. Default: <code>0.25</code>.</li> <li><code>width</code> (<code>number|string</code>, optional): Width of the visualization. Default: <code>100%</code>.</li> <li><code>height</code> (<code>number</code>, optional): Height of the visualization. Default: <code>800</code>.</li> <li><code>view_change_custom_callback</code> (<code>Function</code>, optional): Custom callback triggered on view changes.</li> </ul>"},{"location":"javascript/api/#public-api","title":"Public API","text":"<p>The <code>landscape_ist</code> function returns an object (<code>landscape</code>) with several methods for interacting with the visualization.</p>"},{"location":"javascript/api/#update_matrix_gene","title":"<code>update_matrix_gene</code>","text":"<p>Updates the visualization to highlight data for a specific gene.</p>"},{"location":"javascript/api/#parameters_1","title":"Parameters","text":"<ul> <li><code>inst_gene</code> (<code>string</code>): The gene to highlight.</li> </ul>"},{"location":"javascript/api/#behavior","title":"Behavior","text":"<ul> <li>Updates the transcript layer to show data for the specified gene.</li> <li>Scrolls the bar graph to bring the selected gene into view.</li> <li>Toggles visibility of image layers and controls based on the selected gene.</li> </ul>"},{"location":"javascript/api/#update_matrix_col","title":"<code>update_matrix_col</code>","text":"<p>Updates the visualization to highlight data for a specific column (e.g., cluster).</p>"},{"location":"javascript/api/#parameters_2","title":"Parameters","text":"<ul> <li><code>inst_col</code> (<code>string</code>): The column to highlight.</li> </ul>"},{"location":"javascript/api/#behavior_1","title":"Behavior","text":"<ul> <li>Highlights the bar graph corresponding to the selected column.</li> <li>Updates cell and path layers to reflect the selected column.</li> <li>Toggles visibility of layers based on the column selection.</li> </ul>"},{"location":"javascript/api/#update_matrix_dendro_col","title":"<code>update_matrix_dendro_col</code>","text":"<p>Updates the visualization based on a dendrogram selection of columns.</p>"},{"location":"javascript/api/#parameters_3","title":"Parameters","text":"<ul> <li><code>selected_cols</code> (<code>Array<string></code>): The list of selected column names.</li> </ul>"},{"location":"javascript/api/#behavior_2","title":"Behavior","text":"<ul> <li>Highlights the selected columns in the bar graph.</li> <li>Updates layers to reflect the selection.</li> </ul>"},{"location":"javascript/api/#update_view_state","title":"<code>update_view_state</code>","text":"<p>Updates the view state of the Deck.gl visualization.</p>"},{"location":"javascript/api/#parameters_4","title":"Parameters","text":"<ul> <li><code>new_view_state</code> (<code>Object</code>): The new view state configuration.</li> <li><code>close_up</code> (<code>boolean</code>): Whether the view should zoom in closely.</li> <li><code>trx_layer</code> (<code>Object</code>): The transcript layer to update.</li> </ul>"},{"location":"javascript/api/#behavior_3","title":"Behavior","text":"<ul> <li>Adjusts the viewport and reconfigures layers based on the new view state.</li> </ul>"},{"location":"javascript/api/#update_layers","title":"<code>update_layers</code>","text":"<p>Updates all visualization layers.</p>"},{"location":"javascript/api/#behavior_4","title":"Behavior","text":"<ul> <li>Refreshes the Deck.gl layers with the current visualization state.</li> </ul>"},{"location":"javascript/api/#finalize","title":"<code>finalize</code>","text":"<p>Finalizes the Deck.gl instance and cleans up resources.</p>"},{"location":"javascript/api/#behavior_5","title":"Behavior","text":"<ul> <li>Disposes of all Deck.gl resources and event listeners to prevent memory leaks.</li> </ul>"},{"location":"javascript/api/#usage-example","title":"Usage Example","text":"<pre><code>\njavascript\nimport { landscape_ist } from 'path/to/landscape_ist';\n\nconst rootElement = document.getElementById('visualization-container');\nconst model = { /* Model containing visualization data */ };\n\nconst visualization = await landscape_ist(\n rootElement,\n model,\n 'example-token',\n 100,\n 200,\n 0,\n -5,\n 'https://example.com/data',\n 'Example Dataset'\n);\n\n// Update the visualization with a specific gene.\nvisualization.update_matrix_gene('TP53');\n\n// Update the visualization with a specific column.\nvisualization.update_matrix_col('Cluster 1');\n\n// Finalize the visualization when done.\nvisualization.finalize();\n\n</code></pre>"},{"location":"javascript/api/#matrix_viz-api-documentation","title":"<code>matrix_viz</code> API Documentation","text":"<p>The <code>matrix_viz</code> function initializes and renders a matrix visualization. This API is built using approaches and code adaptations from the Clustergrammer-GL library, and it integrates tightly with Deck.gl to provide interactive and dynamic visualizations.</p>"},{"location":"javascript/api/#parameters_5","title":"Parameters","text":"<ul> <li><code>model</code> (<code>Object</code>): The model object containing configuration data for the visualization.</li> <li><code>el</code> (<code>HTMLElement</code>): The root DOM element where the visualization is rendered.</li> <li><code>network</code> (<code>Object</code>): The network object containing the matrix data to visualize.</li> <li><code>width</code> (<code>string|number</code>, optional): The width of the visualization. Default: <code>'800'</code>.</li> <li><code>height</code> (<code>string|number</code>, optional): The height of the visualization. Default: <code>'800'</code>.</li> <li><code>row_label_callback</code> (<code>Function</code>, optional): A callback function triggered on row label interactions.</li> <li><code>col_label_callback</code> (<code>Function</code>, optional): A callback function triggered on column label interactions.</li> <li><code>col_dendro_callback</code> (<code>Function</code>, optional): A callback function triggered on dendrogram column interactions.</li> </ul>"},{"location":"javascript/api/#internal-behavior","title":"Internal Behavior","text":"<p>The function performs the following setup: 1. Deck.gl Integration: - Initializes a Deck.gl instance for the matrix visualization. - Sets properties for interactivity, including tooltips, view state changes, and layer filtering.</p> <ol> <li>Matrix Data Setup:</li> <li>Parses and structures the matrix data from the <code>network</code> object.</li> <li> <p>Configures labels, categories, and dendrograms for both rows and columns.</p> </li> <li> <p>Layer Initialization:</p> </li> <li>Creates layers for:<ul> <li>Matrix cells.</li> <li>Row and column labels.</li> <li>Row and column categories.</li> <li>Row and column dendrograms.</li> </ul> </li> <li> <p>Attaches interactions (e.g., click events) to these layers.</p> </li> <li> <p>UI Setup:</p> </li> <li>Creates a container for the visualization and appends it to the root DOM element.</li> </ol>"},{"location":"javascript/api/#example-usage","title":"Example Usage","text":"<pre><code>import { matrix_viz } from 'path/to/matrix_viz';\n\nconst rootElement = document.getElementById('matrix-container');\nconst model = { /* Model containing visualization data */ };\nconst network = { /* Network object representing the matrix data */ };\n\n// Callback functions\nconst rowLabelCallback = (row) => {\n console.log('Row label clicked:', row);\n};\n\nconst colLabelCallback = (col) => {\n console.log('Column label clicked:', col);\n};\n\nconst colDendroCallback = (dendro) => {\n console.log('Column dendrogram clicked:', dendro);\n};\n\n// Initialize the matrix visualization\nawait matrix_viz(\n model,\n rootElement,\n network,\n 800,\n 800,\n rowLabelCallback,\n colLabelCallback,\n colDendroCallback\n);\n</code></pre>"},{"location":"overview/","title":"Overview","text":"<p>The Celldega library is being developed to help researchers easily visualize and analyze high-dimensional spatial-omics data in the context of a notebook workflow. Initial development has been focused on spatial transcriptomics visualization.</p> <p>Celldega can be used as a Python library in a Jupyter notebook environment or as a stand-alone JavaScript library for creating visualizations.</p> <ul> <li>Getting Started</li> <li>Installation</li> <li>Usage</li> </ul>"},{"location":"overview/file_formats/","title":"File Formats","text":""},{"location":"overview/file_formats/#landscapefiles","title":"LandscapeFiles","text":"<p>This format is small enough that we can store an entire Xenium dataset on GitHub as a repo for the purposes of building a public gallery.</p>"},{"location":"overview/getting_started/","title":"Getting Started","text":"<p>Celldega is a spatial analysis and visualization library that is being developed by the Spatial Technology Platform at the Broad Institute of MIT and Harvard. Celldega can be used as a Jupyter Widget in Python as well as a stand-alone JavaScript library.</p> <p>Please see examples notebooks below to try out Celldega in a Jupyter notebook or ObservableHQ JavaScript notebook:</p> <ul> <li>Celldega_Xenium_Landscape_Visualizations_Colab.ipynb</li> <li>Celldega Landscape Xenium ObservableHQ</li> </ul>"},{"location":"overview/installation/","title":"Installation","text":""},{"location":"overview/installation/#python","title":"Python","text":"<p>The Celldega library can be installed using pip</p> <p>Celldega can be installed using pip:</p> <pre><code>pip install celldega\n</code></pre> <p>Celldega can also be installed with the optional pre-processing requirements (e.g., vips for image pre-processing) using:</p> <pre><code>pip install celldega[pre]\n</code></pre>"},{"location":"overview/installation/#javascript","title":"JavaScript","text":"<p>Celldega can be used in a JavaScript environment such as ObservableHQ by importing it as a module from content delivery networks like esm.sh:</p> <pre><code>celldega = await import('https://esm.sh/celldega@latest')\n</code></pre> <p>Or by importing it from a local file</p> <pre><code>import celldega from 'js/widget.js'\n</code></pre>"},{"location":"overview/usage/","title":"Celldega Usage","text":""},{"location":"overview/usage/#terrabio","title":"Terra.bio","text":"<p>** Coming soon **</p>"},{"location":"python/","title":"Python API Overview","text":""},{"location":"python/#pre-module-overview","title":"Pre Module Overview","text":"<p>The <code>pre</code> module contains methods for pre-processing LandscapeFiles.</p>"},{"location":"python/#viz-module-overview","title":"Viz Module Overview","text":"<p>The <code>viz</code> module contains functions and classes for data visualization.</p>"},{"location":"python/api/","title":"Python API Reference","text":"<p>Module for pre-processing to generate LandscapeFiles from ST data.</p> <p>Module for visualization</p>"},{"location":"python/api/#celldega.pre.convert_long_id_to_short","title":"<code>convert_long_id_to_short(df)</code>","text":"<p>Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>The DataFrame containing the EntityID.</p> required <p>Returns: pd.DataFrame: The original DataFrame with an additional column named <code>cell_id</code> containing the shortened cell IDs.</p> <p>The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates it to create a shorter identifier that is added as a new column to the DataFrame.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_long_id_to_short(df):\n \"\"\"\n Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.\n\n Args:\n df (pd.DataFrame): The DataFrame containing the EntityID.\n Returns:\n pd.DataFrame: The original DataFrame with an additional column named `cell_id`\n containing the shortened cell IDs.\n\n The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates\n it to create a shorter identifier that is added as a new column to the DataFrame.\n \"\"\"\n # Function to hash and encode the cell ID\n def hash_and_shorten_id(cell_id):\n # Create a hash of the cell ID\n cell_id_bytes = str(cell_id).encode('utf-8')\n hash_object = hashlib.sha256(cell_id_bytes)\n hash_digest = hash_object.digest()\n\n # Encode the hash to a base64 string to mix letters and numbers, truncate to 9 characters\n short_id = base64.urlsafe_b64encode(hash_digest).decode('utf-8')[:9]\n return short_id\n\n # Apply the hash_and_shorten_id function to each cell ID in the specified column\n df['cell_id'] = df['EntityID'].apply(hash_and_shorten_id)\n\n return df\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_jpeg","title":"<code>convert_to_jpeg(image_path, quality=80)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/api/#celldega.pre.convert_to_jpeg--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/api/#celldega.pre.convert_to_jpeg--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_jpeg(image_path, quality=80):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".jpeg\")\n image.jpegsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_png","title":"<code>convert_to_png(image_path)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/api/#celldega.pre.convert_to_png--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/api/#celldega.pre.convert_to_png--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_png(image_path):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".png\")\n image.pngsave(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.convert_to_webp","title":"<code>convert_to_webp(image_path, quality=100)</code>","text":"<p>Convert a TIFF image to a WEBP image with a specified quality score.</p>"},{"location":"python/api/#celldega.pre.convert_to_webp--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=100) Quality score for the WEBP image (higher is better quality)</p>"},{"location":"python/api/#celldega.pre.convert_to_webp--returns","title":"Returns","text":"<p>new_image_path : str Path to the WEBP image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_webp(image_path, quality=100):\n \"\"\"\n Convert a TIFF image to a WEBP image with a specified quality score.\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=100)\n Quality score for the WEBP image (higher is better quality)\n\n Returns\n -------\n new_image_path : str\n Path to the WEBP image file\n \"\"\"\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a WEBP with specified quality\n new_image_path = image_path.replace(\".tif\", \".webp\")\n image.webpsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.get_max_zoom_level","title":"<code>get_max_zoom_level(path_image_pyramid)</code>","text":"<p>Returns the maximum zoom level based on the highest-numbered directory in the specified path_image_pyramid.</p> <p>Parameters:</p> Name Type Description Default <code>path_image_pyramid</code> <code>str</code> <p>The path to the directory containing zoom level directories.</p> required <p>Returns:</p> Name Type Description <code>max_pyramid_zoom</code> <code>int</code> <p>The maximum zoom level.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def get_max_zoom_level(path_image_pyramid):\n \"\"\"\n Returns the maximum zoom level based on the highest-numbered directory\n in the specified path_image_pyramid.\n\n Parameters:\n path_image_pyramid (str): The path to the directory containing zoom level directories.\n\n Returns:\n max_pyramid_zoom (int): The maximum zoom level.\n \"\"\"\n # List all entries in the path_image_pyramid that are directories and can be converted to integers\n zoom_levels = [\n entry\n for entry in os.listdir(path_image_pyramid)\n if os.path.isdir(os.path.join(path_image_pyramid, entry)) and entry.isdigit()\n ]\n\n # Convert to integer and find the maximum value\n max_pyramid_zoom = max(map(int, zoom_levels)) if zoom_levels else None\n\n return max_pyramid_zoom\n</code></pre>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles","title":"<code>make_cell_boundary_tiles(technology, path_cell_boundaries, path_meta_cell_micron, path_transformation_matrix, path_output, coarse_tile_factor=20, tile_size=250, tile_bounds=None, image_scale=1, max_workers=8)</code>","text":"<p>Processes cell boundary data and divides it into spatial tiles based on the provided technology. Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles. The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.</p>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\". path_cell_boundaries : str Path to the file containing the cell boundaries (Parquet format). path_meta_cell_micron : str Path to the file containing cell metadata (CSV format). path_transformation_matrix : str Path to the file containing the transformation matrix (CSV format). path_output : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional, default=20. scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional, default=500 Size of each fine-grain tile in microns. tile_bounds : dict, optional Dictionary containing the minimum and maximum bounds for x and y coordinates. image_scale : float, optional, default=1 Scale factor to apply to the geometry data. max_workers : int, optional, default=8 Maximum number of parallel workers for processing tiles.</p>"},{"location":"python/api/#celldega.pre.make_cell_boundary_tiles--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/boundary_tile.py</code> <pre><code>def make_cell_boundary_tiles(\n technology,\n path_cell_boundaries,\n path_meta_cell_micron,\n path_transformation_matrix,\n path_output,\n coarse_tile_factor=20,\n tile_size=250,\n tile_bounds=None,\n image_scale=1,\n max_workers=8\n):\n\n\n \"\"\"\n Processes cell boundary data and divides it into spatial tiles based on the provided technology.\n Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles.\n The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.\n\n Parameters\n ----------\n technology : str\n The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\".\n path_cell_boundaries : str\n Path to the file containing the cell boundaries (Parquet format).\n path_meta_cell_micron : str\n Path to the file containing cell metadata (CSV format).\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV format).\n path_output : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional, default=20.\n scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional, default=500\n Size of each fine-grain tile in microns.\n tile_bounds : dict, optional\n Dictionary containing the minimum and maximum bounds for x and y coordinates.\n image_scale : float, optional, default=1\n Scale factor to apply to the geometry data.\n max_workers : int, optional, default=8\n Maximum number of parallel workers for processing tiles.\n\n Returns\n -------\n None\n \"\"\"\n\n def numpy_affine_transform(coords, matrix):\n \"\"\"Apply affine transformation to numpy coordinates.\"\"\"\n # Homogeneous coordinates for affine transformation\n coords = np.hstack([coords, np.ones((coords.shape[0], 1))])\n transformed_coords = coords @ matrix.T\n return transformed_coords[:, :2] # Drop the homogeneous coordinate\n\n def batch_transform_geometries(geometries, transformation_matrix, scale):\n \"\"\"\n Batch transform geometries using numpy for optimized performance.\n \"\"\"\n # Extract affine transformation parameters into a 3x3 matrix for numpy\n affine_matrix = np.array([\n [transformation_matrix[0, 0], transformation_matrix[0, 1], transformation_matrix[0, 2]],\n [transformation_matrix[1, 0], transformation_matrix[1, 1], transformation_matrix[1, 2]],\n [0, 0, 1]\n ])\n\n transformed_geometries = []\n\n for polygon in geometries:\n # Extract coordinates and transform them\n if isinstance(polygon, MultiPolygon):\n polygon = next(polygon.geoms) # Use the first geometry\n\n # Transform the exterior of the polygon\n exterior_coords = np.array(polygon.exterior.coords)\n\n # Apply the affine transformation and scale\n transformed_coords = numpy_affine_transform(exterior_coords, affine_matrix) / scale\n\n # Append the result to the transformed_geometries list\n transformed_geometries.append([transformed_coords.tolist()])\n\n return transformed_geometries\n\n\n def filter_and_save_fine_boundary(coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output):\n cell_ids = coarse_tile.index.values\n\n tile_filter = (\n (coarse_tile[\"center_x\"] >= fine_tile_x_min) & (coarse_tile[\"center_x\"] < fine_tile_x_max) &\n (coarse_tile[\"center_y\"] >= fine_tile_y_min) & (coarse_tile[\"center_y\"] < fine_tile_y_max)\n )\n filtered_indices = np.where(tile_filter)[0]\n\n keep_cells = cell_ids[filtered_indices]\n fine_tile_cells = coarse_tile.loc[keep_cells, [\"GEOMETRY\"]]\n fine_tile_cells = fine_tile_cells.assign(name=fine_tile_cells.index)\n\n if not fine_tile_cells.empty:\n filename = f\"{path_output}/cell_tile_{fine_i}_{fine_j}.parquet\"\n fine_tile_cells.to_parquet(filename)\n\n def process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y):\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n futures.append(executor.submit(\n filter_and_save_fine_boundary, coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output\n ))\n\n for future in futures:\n future.result()\n\n tile_size_x = tile_size\n tile_size_y = tile_size\n\n transformation_matrix = pd.read_csv(path_transformation_matrix, header=None, sep=\" \").values\n\n # Load cell boundary data based on the technology\n if technology == \"MERSCOPE\":\n df_meta = pd.read_parquet(f\"{path_output.replace('cell_segmentation','cell_metadata.parquet')}\")\n entity_to_cell_id_dict = pd.Series(df_meta.index.values, index=df_meta.EntityID).to_dict()\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n cells_orig['cell_id'] = cells_orig['EntityID'].map(entity_to_cell_id_dict)\n cells_orig = cells_orig[cells_orig[\"ZIndex\"] == 1]\n\n # Correct cell_id issues with meta_cell\n meta_cell = pd.read_csv(path_meta_cell_micron)\n meta_cell['cell_id'] = meta_cell['EntityID'].map(entity_to_cell_id_dict)\n cells_orig.index = meta_cell[meta_cell[\"cell_id\"].isin(cells_orig['cell_id'])].index\n\n # Correct 'MultiPolygon' to 'Polygon'\n cells_orig[\"geometry\"] = cells_orig[\"Geometry\"].apply(\n lambda x: list(x.geoms)[0] if isinstance(x, MultiPolygon) else x\n )\n\n cells_orig.set_index('cell_id', inplace=True)\n\n elif technology == \"Xenium\":\n xenium_cells = pd.read_parquet(path_cell_boundaries)\n grouped = xenium_cells.groupby(\"cell_id\")[[\"vertex_x\", \"vertex_y\"]].agg(lambda x: x.tolist())\n grouped[\"geometry\"] = grouped.apply(lambda row: Polygon(zip(row[\"vertex_x\"], row[\"vertex_y\"])), axis=1)\n cells_orig = gpd.GeoDataFrame(grouped, geometry=\"geometry\")[[\"geometry\"]]\n\n elif technology == \"custom\":\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n\n # Transform geometries\n cells_orig[\"GEOMETRY\"] = batch_transform_geometries(cells_orig[\"geometry\"], transformation_matrix, image_scale)\n\n # Convert transformed geometries to polygons and calculate centroids\n cells_orig[\"polygon\"] = cells_orig[\"GEOMETRY\"].apply(lambda x: Polygon(x[0]))\n gdf_cells = gpd.GeoDataFrame(geometry=cells_orig[\"polygon\"])\n gdf_cells[\"center_x\"] = gdf_cells.geometry.centroid.x\n gdf_cells[\"center_y\"] = gdf_cells.geometry.centroid.y\n gdf_cells[\"GEOMETRY\"] = cells_orig[\"GEOMETRY\"]\n\n # Ensure the output directory exists\n if not os.path.exists(path_output):\n os.makedirs(path_output)\n\n # Calculate tile bounds and fine/coarse tiles\n x_min, x_max = tile_bounds[\"x_min\"], tile_bounds[\"x_max\"]\n y_min, y_max = tile_bounds[\"y_min\"], tile_bounds[\"y_max\"]\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Process coarse tiles in parallel\n for i in tqdm(range(n_coarse_tiles_x), desc=\"Processing coarse tiles\"):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n coarse_tile = gdf_cells[\n (gdf_cells[\"center_x\"] >= coarse_tile_x_min) & (gdf_cells[\"center_x\"] < coarse_tile_x_max) &\n (gdf_cells[\"center_y\"] >= coarse_tile_y_min) & (gdf_cells[\"center_y\"] < coarse_tile_y_max)\n ]\n if not coarse_tile.empty:\n process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid","title":"<code>make_deepzoom_pyramid(image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix='.jpeg')</code>","text":"<p>Create a DeepZoom image pyramid from a JPEG image</p>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid--parameters","title":"Parameters","text":"<p>image_path : str Path to the JPEG image file tile_size : int (default=512) Tile size for the DeepZoom pyramid overlap : int (default=0) Overlap size for the DeepZoom pyramid suffix : str (default='jpeg') Suffix for the DeepZoom pyramid tiles</p>"},{"location":"python/api/#celldega.pre.make_deepzoom_pyramid--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_deepzoom_pyramid(\n image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix=\".jpeg\"\n):\n \"\"\"\n Create a DeepZoom image pyramid from a JPEG image\n\n Parameters\n ----------\n image_path : str\n Path to the JPEG image file\n tile_size : int (default=512)\n Tile size for the DeepZoom pyramid\n overlap : int (default=0)\n Overlap size for the DeepZoom pyramid\n suffix : str (default='jpeg')\n Suffix for the DeepZoom pyramid tiles\n\n Returns\n -------\n None\n\n \"\"\"\n\n # Define the output path\n output_path = Path(output_path)\n\n # Load the JPEG image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # check if the output path exists and create it if it does not\n output_path.mkdir(parents=True, exist_ok=True)\n\n # append the pyramid name to the output path\n output_path = output_path / pyramid_name\n\n # Save the image as a DeepZoom image pyramid\n image.dzsave(output_path, tile_size=tile_size, overlap=overlap, suffix=suffix)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord","title":"<code>make_meta_cell_image_coord(technology, path_transformation_matrix, path_meta_cell_micron, path_meta_cell_image, image_scale)</code>","text":"<p>Apply an affine transformation to the cell coordinates in microns and save the transformed coordinates in pixels</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_transformation_matrix : str Path to the transformation matrix file path_meta_cell_micron : str Path to the meta cell file with coordinates in microns path_meta_cell_image : str Path to save the meta cell file with coordinates in pixels</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/api/#celldega.pre.make_meta_cell_image_coord--examples","title":"Examples","text":"<p>make_meta_cell_image_coord( ... technology='Xenium', ... path_transformation_matrix='data/transformation_matrix.txt', ... path_meta_cell_micron='data/meta_cell_micron.csv', ... path_meta_cell_image='data/meta_cell_image.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_cell_image_coord(\n technology,\n path_transformation_matrix,\n path_meta_cell_micron,\n path_meta_cell_image,\n image_scale\n):\n \"\"\"\n Apply an affine transformation to the cell coordinates in microns and save\n the transformed coordinates in pixels\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_transformation_matrix : str\n Path to the transformation matrix file\n path_meta_cell_micron : str\n Path to the meta cell file with coordinates in microns\n path_meta_cell_image : str\n Path to save the meta cell file with coordinates in pixels\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_cell_image_coord(\n ... technology='Xenium',\n ... path_transformation_matrix='data/transformation_matrix.txt',\n ... path_meta_cell_micron='data/meta_cell_micron.csv',\n ... path_meta_cell_image='data/meta_cell_image.parquet'\n ... )\n\n \"\"\"\n\n transformation_matrix = pd.read_csv(\n path_transformation_matrix, header=None, sep=\" \"\n ).values\n\n if technology == \"MERSCOPE\":\n meta_cell = pd.read_csv(path_meta_cell_micron, usecols=[\"EntityID\", \"center_x\", \"center_y\"])\n meta_cell = convert_long_id_to_short(meta_cell)\n meta_cell[\"name\"] = meta_cell[\"cell_id\"]\n meta_cell = meta_cell.set_index('cell_id')\n elif technology == \"Xenium\":\n usecols = [\"cell_id\", \"x_centroid\", \"y_centroid\"]\n meta_cell = pd.read_csv(path_meta_cell_micron, index_col=0, usecols=usecols)\n meta_cell.columns = [\"center_x\", \"center_y\"]\n meta_cell[\"name\"] = pd.Series(meta_cell.index, index=meta_cell.index)\n\n # Adding a ones column to accommodate for affine transformation\n meta_cell[\"ones\"] = 1\n\n # Preparing the data for matrix multiplication\n points = meta_cell[[\"center_x\", \"center_y\", \"ones\"]].values\n\n # Applying the transformation matrix\n transformed_points = np.dot(transformation_matrix, points.T).T\n\n # Updating the DataFrame with transformed coordinates\n meta_cell[\"center_x\"] = transformed_points[:, 0]\n meta_cell[\"center_y\"] = transformed_points[:, 1]\n\n # Dropping the ones column as it's no longer needed\n meta_cell.drop(columns=[\"ones\"], inplace=True)\n\n meta_cell[\"center_x\"] = meta_cell[\"center_x\"] / image_scale\n meta_cell[\"center_y\"] = meta_cell[\"center_y\"] / image_scale\n\n meta_cell[\"geometry\"] = meta_cell.apply(\n lambda row: [row[\"center_x\"], row[\"center_y\"]], axis=1\n )\n\n if technology == \"MERSCOPE\":\n meta_cell = meta_cell[[\"name\", \"geometry\", \"EntityID\"]]\n else:\n meta_cell = meta_cell[[\"name\", \"geometry\"]]\n\n\n meta_cell.to_parquet(path_meta_cell_image)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_meta_gene","title":"<code>make_meta_gene(technology, path_cbg, path_output)</code>","text":"<p>Create a DataFrame with genes and their assigned colors</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_cbg : str Path to the cell-by-gene matrix data (the data format can vary based on technology) path_output : str Path to save the meta gene file</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/api/#celldega.pre.make_meta_gene--examples","title":"Examples","text":"<p>make_meta_gene( ... technology='Xenium', ... path_cbg='data/', ... path_output='data/meta_gene.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_gene(technology, path_cbg, path_output):\n \"\"\"\n Create a DataFrame with genes and their assigned colors\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_cbg : str\n Path to the cell-by-gene matrix data (the data format can vary based on technology)\n path_output : str\n Path to save the meta gene file\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_gene(\n ... technology='Xenium',\n ... path_cbg='data/',\n ... path_output='data/meta_gene.parquet'\n ... )\n \"\"\"\n\n if technology == \"MERSCOPE\":\n cbg = pd.read_csv(path_cbg, index_col=0)\n genes = cbg.columns.tolist()\n elif technology == \"Xenium\":\n # genes = pd.read_csv(path_cbg + 'features.tsv.gz', sep='\\t', header=None)[1].values.tolist()\n cbg = read_cbg_mtx(path_cbg)\n genes = cbg.columns.tolist()\n\n # Get all categorical color palettes from Matplotlib and flatten them into a single list of colors\n palettes = [plt.get_cmap(name).colors for name in plt.colormaps() if \"tab\" in name]\n flat_colors = [color for palette in palettes for color in palette]\n\n # Convert RGB tuples to hex codes\n flat_colors_hex = [to_hex(color) for color in flat_colors]\n\n # Use modular arithmetic to assign a color to each gene, white for genes with \"Blank\"\n colors = [\n flat_colors_hex[i % len(flat_colors_hex)] if \"Blank\" not in gene else \"#FFFFFF\"\n for i, gene in enumerate(genes)\n ]\n\n # Create a DataFrame with genes and their assigned colors\n ser_color = pd.Series(colors, index=genes)\n\n # calculate gene expression metadata\n meta_gene = calc_meta_gene_data(cbg)\n meta_gene['color'] = ser_color\n\n # Identify sparse columns\n sparse_cols = [col for col in meta_gene.columns if pd.api.types.is_sparse(meta_gene[col])]\n\n # Convert sparse columns to dense\n for col in sparse_cols:\n meta_gene[col] = meta_gene[col].sparse.to_dense()\n\n meta_gene.to_parquet(path_output)\n</code></pre>"},{"location":"python/api/#celldega.pre.make_trx_tiles","title":"<code>make_trx_tiles(technology, path_trx, path_transformation_matrix, path_trx_tiles, coarse_tile_factor=10, tile_size=250, chunk_size=1000000, verbose=False, image_scale=1, max_workers=8)</code>","text":"<p>Processes transcript data by dividing it into coarse-grain and fine-grain tiles, applying transformations, and saving the results in a parallelized manner.</p>"},{"location":"python/api/#celldega.pre.make_trx_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\"). path_trx : str Path to the file containing the transcript data. path_transformation_matrix : str Path to the file containing the transformation matrix (CSV file). path_trx_tiles : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional Scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional Size of each fine-grain tile in microns (default is 250). chunk_size : int, optional Number of rows to process per chunk for memory efficiency (default is 1000000). verbose : bool, optional Flag to enable verbose output (default is False). image_scale : float, optional Scale factor to apply to the transcript coordinates (default is 0.5). max_workers : int, optional Maximum number of parallel workers for processing tiles (default is 8).</p>"},{"location":"python/api/#celldega.pre.make_trx_tiles--returns","title":"Returns","text":"<p>dict A dictionary containing the bounds of the processed data in both x and y directions.</p> Source code in <code>src/celldega/pre/trx_tile.py</code> <pre><code>def make_trx_tiles(\n technology,\n path_trx,\n path_transformation_matrix,\n path_trx_tiles,\n coarse_tile_factor=10,\n tile_size=250,\n chunk_size=1000000,\n verbose=False,\n image_scale=1,\n max_workers=8\n):\n \"\"\"\n Processes transcript data by dividing it into coarse-grain and fine-grain tiles,\n applying transformations, and saving the results in a parallelized manner.\n\n Parameters\n ----------\n technology : str\n The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\").\n path_trx : str\n Path to the file containing the transcript data.\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV file).\n path_trx_tiles : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional\n Scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional\n Size of each fine-grain tile in microns (default is 250).\n chunk_size : int, optional\n Number of rows to process per chunk for memory efficiency (default is 1000000).\n verbose : bool, optional\n Flag to enable verbose output (default is False).\n image_scale : float, optional\n Scale factor to apply to the transcript coordinates (default is 0.5).\n max_workers : int, optional\n Maximum number of parallel workers for processing tiles (default is 8).\n\n Returns\n -------\n dict\n A dictionary containing the bounds of the processed data in both x and y directions.\n \"\"\"\n\n def process_coarse_tile(trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers):\n # Filter the entire dataset for the current coarse tile\n coarse_tile = trx.filter(\n (pl.col(\"transformed_x\") >= coarse_tile_x_min) & (pl.col(\"transformed_x\") < coarse_tile_x_max) &\n (pl.col(\"transformed_y\") >= coarse_tile_y_min) & (pl.col(\"transformed_y\") < coarse_tile_y_max)\n )\n\n if not coarse_tile.is_empty():\n # Now process fine tiles using global fine tile indices\n process_fine_tiles(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers) \n\n\n def process_fine_tiles(coarse_tile, coarse_i, coarse_j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers=8):\n\n # Use ThreadPoolExecutor for parallel processing of fine-grain tiles within the coarse tile\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n\n # Iterate over fine-grain tiles within the global bounds\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n # Submit the task for each fine tile to process in parallel\n futures.append(executor.submit(\n filter_and_save_fine_tile, coarse_tile, coarse_i, coarse_j, fine_i, fine_j, \n fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles\n ))\n\n # Wait for all futures to complete\n for future in concurrent.futures.as_completed(futures):\n future.result() # Raise exceptions if any occurred during execution\n\n\n def filter_and_save_fine_tile(coarse_tile, coarse_i, coarse_j, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles):\n\n # Filter the coarse tile for the current fine tile's boundaries\n fine_tile_trx = coarse_tile.filter(\n (pl.col(\"transformed_x\") >= fine_tile_x_min) & (pl.col(\"transformed_x\") < fine_tile_x_max) &\n (pl.col(\"transformed_y\") >= fine_tile_y_min) & (pl.col(\"transformed_y\") < fine_tile_y_max)\n )\n\n if not fine_tile_trx.is_empty():\n # Add geometry column as a list of [x, y] pairs\n fine_tile_trx = fine_tile_trx.with_columns(\n pl.concat_list([pl.col(\"transformed_x\"), pl.col(\"transformed_y\")]).alias(\"geometry\")\n ).drop(['transformed_x', 'transformed_y'])\n\n # Define the filename based on fine tile coordinates\n filename = f\"{path_trx_tiles}/transcripts_tile_{fine_i}_{fine_j}.parquet\"\n\n # Save the filtered DataFrame to a Parquet file\n fine_tile_trx.to_pandas().to_parquet(filename)\n\n\n # Load transformation matrix\n transformation_matrix = np.loadtxt(path_transformation_matrix)\n\n # Load the transcript data based on the technology using Polars\n if technology == \"MERSCOPE\":\n trx_ini = pl.read_csv(path_trx, columns=[\"gene\", \"global_x\", \"global_y\"])\n trx_ini = trx_ini.with_columns([\n pl.col(\"global_x\").alias(\"x\"),\n pl.col(\"global_y\").alias(\"y\"),\n pl.col(\"gene\").alias(\"name\")\n ]).select([\"name\", \"x\", \"y\"])\n\n elif technology == \"Xenium\":\n trx_ini = pl.read_parquet(path_trx).select([\n pl.col(\"feature_name\").alias(\"name\"),\n pl.col(\"x_location\").alias(\"x\"),\n pl.col(\"y_location\").alias(\"y\")\n ])\n\n # Process the data in chunks and apply transformations\n all_chunks = []\n\n for start_row in tqdm(range(0, trx_ini.height, chunk_size), desc=\"Processing chunks\"):\n chunk = trx_ini.slice(start_row, chunk_size)\n\n # Apply transformation matrix to the coordinates\n points = np.hstack([chunk.select([\"x\", \"y\"]).to_numpy(), np.ones((chunk.height, 1))])\n transformed_points = np.dot(points, transformation_matrix.T)[:, :2]\n\n # Create new transformed columns and drop original x, y columns\n transformed_chunk = chunk.with_columns([\n (pl.Series(transformed_points[:, 0]) * image_scale).round(2).alias(\"transformed_x\"),\n (pl.Series(transformed_points[:, 1]) * image_scale).round(2).alias(\"transformed_y\")\n ]).drop([\"x\", \"y\"])\n all_chunks.append(transformed_chunk)\n\n # Concatenate all chunks after processing\n trx = pl.concat(all_chunks)\n\n # Ensure the output directory exists\n if not os.path.exists(path_trx_tiles):\n os.makedirs(path_trx_tiles)\n\n # Get min and max x, y values\n x_min, x_max = trx.select([\n pl.col(\"transformed_x\").min().alias(\"x_min\"),\n pl.col(\"transformed_x\").max().alias(\"x_max\")\n ]).row(0)\n\n y_min, y_max = trx.select([\n pl.col(\"transformed_y\").min().alias(\"y_min\"),\n pl.col(\"transformed_y\").max().alias(\"y_max\")\n ]).row(0)\n\n # Calculate the number of fine-grain tiles globally\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n\n # Calculate the number of coarse-grain tiles\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Use ThreadPoolExecutor for parallel processing of coarse-grain tiles\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for i in range(n_coarse_tiles_x):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n # Submit each coarse tile for parallel processing\n futures.append(executor.submit(\n process_coarse_tile, trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers\n ))\n\n # Wait for all coarse tiles to complete\n for future in tqdm(concurrent.futures.as_completed(futures), desc=\"Processing coarse tiles\", unit=\"tile\"):\n future.result() # Raise exceptions if any occurred during execution\n\n # Return the tile bounds\n tile_bounds = {\n \"x_min\": x_min,\n \"x_max\": x_max,\n \"y_min\": y_min,\n \"y_max\": y_max,\n }\n\n return tile_bounds\n</code></pre>"},{"location":"python/api/#celldega.pre.reduce_image_size","title":"<code>reduce_image_size(image_path, scale_image=0.5, path_landscape_files='')</code>","text":""},{"location":"python/api/#celldega.pre.reduce_image_size--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file scale_image : float (default=0.5) Scale factor for the image resize</p>"},{"location":"python/api/#celldega.pre.reduce_image_size--returns","title":"Returns","text":"<p>new_image_path : str Path to the resized image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def reduce_image_size(image_path, scale_image=0.5, path_landscape_files=\"\"):\n \"\"\"\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n scale_image : float (default=0.5)\n Scale factor for the image resize\n\n Returns\n -------\n new_image_path : str\n Path to the resized image file\n \"\"\"\n\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n resized_image = image.resize(scale_image)\n\n new_image_name = image_path.split(\"/\")[-1].replace(\".tif\", \"_downsize.tif\")\n new_image_path = f\"{path_landscape_files}/{new_image_name}\"\n resized_image.write_to_file(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/api/#celldega.pre.save_landscape_parameters","title":"<code>save_landscape_parameters(technology, path_landscape_files, image_name='dapi_files', tile_size=1000, image_info={}, image_format='.webp')</code>","text":"<p>Save the landscape parameters to a JSON file.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def save_landscape_parameters(\n technology, path_landscape_files, image_name=\"dapi_files\", tile_size=1000, image_info={}, image_format='.webp'\n):\n \"\"\"\n Save the landscape parameters to a JSON file.\n \"\"\"\n\n path_image_pyramid = f\"{path_landscape_files}/pyramid_images/{image_name}\"\n\n print(path_image_pyramid)\n\n max_pyramid_zoom = get_max_zoom_level(path_image_pyramid)\n\n landscape_parameters = {\n \"technology\": technology,\n \"max_pyramid_zoom\": max_pyramid_zoom,\n \"tile_size\": tile_size,\n \"image_info\": image_info,\n \"image_format\": image_format\n }\n\n path_landscape_parameters = f\"{path_landscape_files}/landscape_parameters.json\"\n\n with open(path_landscape_parameters, \"w\") as file:\n json.dump(landscape_parameters, file, indent=4)\n</code></pre>"},{"location":"python/api/#celldega.viz.Landscape","title":"<code>Landscape</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of spatial omics data. This widget currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)</p> <p>Parameters:</p> Name Type Description Default <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> required <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> required <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> required <code>token</code> <code>str</code> <p>The token traitlet.</p> required <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> required <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize. This will show up in the user interface bar.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>technology</code> <code>str</code> <p>The technology used.</p> <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> <code>token</code> <code>str</code> <p>The token traitlet.</p> <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> <code>ini_z</code> <code>float</code> <p>The initial z-coordinate of the view.</p> <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize.</p> <code>update_trigger</code> <code>dict</code> <p>The dictionary to trigger updates.</p> <code>cell_clusters</code> <code>dict</code> <p>The dictionary containing cell cluster information.</p> <p>Returns:</p> Name Type Description <code>Landscape</code> <p>A widget for visualizing a 'landscape' view of spatial omics data.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Landscape(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of spatial omics data. This widget\n currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)\n\n Args:\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n token (str): The token traitlet.\n base_url (str): The base URL for the widget.\n dataset_name (str, optional): The name of the dataset to visualize. This will show up in the user interface bar.\n\n Attributes:\n component (str): The name of the component.\n technology (str): The technology used.\n base_url (str): The base URL for the widget.\n token (str): The token traitlet.\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_z (float): The initial z-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n dataset_name (str): The name of the dataset to visualize.\n update_trigger (dict): The dictionary to trigger updates.\n cell_clusters (dict): The dictionary containing cell cluster information.\n\n Returns:\n Landscape: A widget for visualizing a 'landscape' view of spatial omics data.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n component = traitlets.Unicode(\"Landscape\").tag(sync=True)\n\n technology = traitlets.Unicode(\"sst\").tag(sync=True)\n base_url = traitlets.Unicode(\"\").tag(sync=True)\n token = traitlets.Unicode(\"\").tag(sync=True)\n ini_x = traitlets.Float(1000).tag(sync=True)\n ini_y = traitlets.Float(1000).tag(sync=True)\n ini_z = traitlets.Float(0).tag(sync=True)\n ini_zoom = traitlets.Float(0).tag(sync=True)\n square_tile_size = traitlets.Float(1.4).tag(sync=True)\n dataset_name = traitlets.Unicode(\"\").tag(sync=True)\n region = traitlets.Dict({}).tag(sync=True)\n\n update_trigger = traitlets.Dict().tag(sync=True)\n cell_clusters = traitlets.Dict().tag(sync=True)\n\n width = traitlets.Int(0).tag(sync=True)\n height = traitlets.Int(800).tag(sync=True)\n\n def trigger_update(self, new_value):\n # This method updates the update_trigger traitlet with a new value\n # You can pass any information necessary for the update, or just a timestamp\n self.update_trigger = new_value\n\n def update_cell_clusters(self, new_clusters):\n # Convert the new_clusters to a JSON serializable format if necessary\n self.cell_clusters = new_clusters\n</code></pre>"},{"location":"python/api/#celldega.viz.Matrix","title":"<code>Matrix</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of a hierarchically clustered matrix.</p> <p>Parameters:</p> Name Type Description Default <code>value</code> <code>int</code> <p>The value traitlet.</p> required <code>component</code> <code>str</code> <p>The component traitlet.</p> required <code>network</code> <code>dict</code> <p>The network traitlet.</p> required <code>click_info</code> <code>dict</code> <p>The click_info traitlet.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>network</code> <code>dict</code> <p>The network dictionary.</p> <code>click_info</code> <code>dict</code> <p>The click_info dictionary.</p> <p>Returns:</p> Name Type Description <code>Matrix</code> <p>A widget for visualizing a hierarchically clustered matrix.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Matrix(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of a hierarchically clustered matrix.\n\n Args:\n value (int): The value traitlet.\n component (str): The component traitlet.\n network (dict): The network traitlet.\n click_info (dict): The click_info traitlet.\n\n Attributes:\n component (str): The name of the component.\n network (dict): The network dictionary.\n click_info (dict): The click_info dictionary.\n\n Returns:\n Matrix: A widget for visualizing a hierarchically clustered matrix.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n value = traitlets.Int(0).tag(sync=True)\n component = traitlets.Unicode(\"Matrix\").tag(sync=True)\n\n network = traitlets.Dict({}).tag(sync=True)\n click_info = traitlets.Dict({}).tag(sync=True)\n</code></pre>"},{"location":"python/pre/api/","title":"Pre Module API Reference","text":"<p>Module for pre-processing to generate LandscapeFiles from ST data.</p>"},{"location":"python/pre/api/#celldega.pre.convert_long_id_to_short","title":"<code>convert_long_id_to_short(df)</code>","text":"<p>Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.</p> <p>Parameters:</p> Name Type Description Default <code>df</code> <code>DataFrame</code> <p>The DataFrame containing the EntityID.</p> required <p>Returns: pd.DataFrame: The original DataFrame with an additional column named <code>cell_id</code> containing the shortened cell IDs.</p> <p>The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates it to create a shorter identifier that is added as a new column to the DataFrame.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_long_id_to_short(df):\n \"\"\"\n Converts a column of long integer cell IDs in a DataFrame to a shorter, hash-based representation.\n\n Args:\n df (pd.DataFrame): The DataFrame containing the EntityID.\n Returns:\n pd.DataFrame: The original DataFrame with an additional column named `cell_id`\n containing the shortened cell IDs.\n\n The function applies a SHA-256 hash to each cell ID, encodes the hash using base64, and truncates\n it to create a shorter identifier that is added as a new column to the DataFrame.\n \"\"\"\n # Function to hash and encode the cell ID\n def hash_and_shorten_id(cell_id):\n # Create a hash of the cell ID\n cell_id_bytes = str(cell_id).encode('utf-8')\n hash_object = hashlib.sha256(cell_id_bytes)\n hash_digest = hash_object.digest()\n\n # Encode the hash to a base64 string to mix letters and numbers, truncate to 9 characters\n short_id = base64.urlsafe_b64encode(hash_digest).decode('utf-8')[:9]\n return short_id\n\n # Apply the hash_and_shorten_id function to each cell ID in the specified column\n df['cell_id'] = df['EntityID'].apply(hash_and_shorten_id)\n\n return df\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg","title":"<code>convert_to_jpeg(image_path, quality=80)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_jpeg--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_jpeg(image_path, quality=80):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".jpeg\")\n image.jpegsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_png","title":"<code>convert_to_png(image_path)</code>","text":"<p>Convert a TIFF image to a JPEG image with a quality of score</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_png--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=80) Quality score for the JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_png--returns","title":"Returns","text":"<p>new_image_path : str Path to the JPEG image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_png(image_path):\n \"\"\"\n Convert a TIFF image to a JPEG image with a quality of score\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=80)\n Quality score for the JPEG image\n\n Returns\n -------\n new_image_path : str\n Path to the JPEG image file\n\n \"\"\"\n\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a JPEG with a quality of 80\n new_image_path = image_path.replace(\".tif\", \".png\")\n image.pngsave(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp","title":"<code>convert_to_webp(image_path, quality=100)</code>","text":"<p>Convert a TIFF image to a WEBP image with a specified quality score.</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file quality : int (default=100) Quality score for the WEBP image (higher is better quality)</p>"},{"location":"python/pre/api/#celldega.pre.convert_to_webp--returns","title":"Returns","text":"<p>new_image_path : str Path to the WEBP image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def convert_to_webp(image_path, quality=100):\n \"\"\"\n Convert a TIFF image to a WEBP image with a specified quality score.\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n quality : int (default=100)\n Quality score for the WEBP image (higher is better quality)\n\n Returns\n -------\n new_image_path : str\n Path to the WEBP image file\n \"\"\"\n # Load the TIFF image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # Save the image as a WEBP with specified quality\n new_image_path = image_path.replace(\".tif\", \".webp\")\n image.webpsave(new_image_path, Q=quality)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.get_max_zoom_level","title":"<code>get_max_zoom_level(path_image_pyramid)</code>","text":"<p>Returns the maximum zoom level based on the highest-numbered directory in the specified path_image_pyramid.</p> <p>Parameters:</p> Name Type Description Default <code>path_image_pyramid</code> <code>str</code> <p>The path to the directory containing zoom level directories.</p> required <p>Returns:</p> Name Type Description <code>max_pyramid_zoom</code> <code>int</code> <p>The maximum zoom level.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def get_max_zoom_level(path_image_pyramid):\n \"\"\"\n Returns the maximum zoom level based on the highest-numbered directory\n in the specified path_image_pyramid.\n\n Parameters:\n path_image_pyramid (str): The path to the directory containing zoom level directories.\n\n Returns:\n max_pyramid_zoom (int): The maximum zoom level.\n \"\"\"\n # List all entries in the path_image_pyramid that are directories and can be converted to integers\n zoom_levels = [\n entry\n for entry in os.listdir(path_image_pyramid)\n if os.path.isdir(os.path.join(path_image_pyramid, entry)) and entry.isdigit()\n ]\n\n # Convert to integer and find the maximum value\n max_pyramid_zoom = max(map(int, zoom_levels)) if zoom_levels else None\n\n return max_pyramid_zoom\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles","title":"<code>make_cell_boundary_tiles(technology, path_cell_boundaries, path_meta_cell_micron, path_transformation_matrix, path_output, coarse_tile_factor=20, tile_size=250, tile_bounds=None, image_scale=1, max_workers=8)</code>","text":"<p>Processes cell boundary data and divides it into spatial tiles based on the provided technology. Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles. The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.</p>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\". path_cell_boundaries : str Path to the file containing the cell boundaries (Parquet format). path_meta_cell_micron : str Path to the file containing cell metadata (CSV format). path_transformation_matrix : str Path to the file containing the transformation matrix (CSV format). path_output : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional, default=20. scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional, default=500 Size of each fine-grain tile in microns. tile_bounds : dict, optional Dictionary containing the minimum and maximum bounds for x and y coordinates. image_scale : float, optional, default=1 Scale factor to apply to the geometry data. max_workers : int, optional, default=8 Maximum number of parallel workers for processing tiles.</p>"},{"location":"python/pre/api/#celldega.pre.make_cell_boundary_tiles--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/boundary_tile.py</code> <pre><code>def make_cell_boundary_tiles(\n technology,\n path_cell_boundaries,\n path_meta_cell_micron,\n path_transformation_matrix,\n path_output,\n coarse_tile_factor=20,\n tile_size=250,\n tile_bounds=None,\n image_scale=1,\n max_workers=8\n):\n\n\n \"\"\"\n Processes cell boundary data and divides it into spatial tiles based on the provided technology.\n Reads cell boundary data, applies affine transformations, and divides the data into coarse and fine tiles.\n The resulting tiles are saved as Parquet files, each containing the geometries of cells in that tile.\n\n Parameters\n ----------\n technology : str\n The technology used to generate the cell boundary data, e.g., \"MERSCOPE\", \"Xenium\", or \"custom\".\n path_cell_boundaries : str\n Path to the file containing the cell boundaries (Parquet format).\n path_meta_cell_micron : str\n Path to the file containing cell metadata (CSV format).\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV format).\n path_output : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional, default=20.\n scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional, default=500\n Size of each fine-grain tile in microns.\n tile_bounds : dict, optional\n Dictionary containing the minimum and maximum bounds for x and y coordinates.\n image_scale : float, optional, default=1\n Scale factor to apply to the geometry data.\n max_workers : int, optional, default=8\n Maximum number of parallel workers for processing tiles.\n\n Returns\n -------\n None\n \"\"\"\n\n def numpy_affine_transform(coords, matrix):\n \"\"\"Apply affine transformation to numpy coordinates.\"\"\"\n # Homogeneous coordinates for affine transformation\n coords = np.hstack([coords, np.ones((coords.shape[0], 1))])\n transformed_coords = coords @ matrix.T\n return transformed_coords[:, :2] # Drop the homogeneous coordinate\n\n def batch_transform_geometries(geometries, transformation_matrix, scale):\n \"\"\"\n Batch transform geometries using numpy for optimized performance.\n \"\"\"\n # Extract affine transformation parameters into a 3x3 matrix for numpy\n affine_matrix = np.array([\n [transformation_matrix[0, 0], transformation_matrix[0, 1], transformation_matrix[0, 2]],\n [transformation_matrix[1, 0], transformation_matrix[1, 1], transformation_matrix[1, 2]],\n [0, 0, 1]\n ])\n\n transformed_geometries = []\n\n for polygon in geometries:\n # Extract coordinates and transform them\n if isinstance(polygon, MultiPolygon):\n polygon = next(polygon.geoms) # Use the first geometry\n\n # Transform the exterior of the polygon\n exterior_coords = np.array(polygon.exterior.coords)\n\n # Apply the affine transformation and scale\n transformed_coords = numpy_affine_transform(exterior_coords, affine_matrix) / scale\n\n # Append the result to the transformed_geometries list\n transformed_geometries.append([transformed_coords.tolist()])\n\n return transformed_geometries\n\n\n def filter_and_save_fine_boundary(coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output):\n cell_ids = coarse_tile.index.values\n\n tile_filter = (\n (coarse_tile[\"center_x\"] >= fine_tile_x_min) & (coarse_tile[\"center_x\"] < fine_tile_x_max) &\n (coarse_tile[\"center_y\"] >= fine_tile_y_min) & (coarse_tile[\"center_y\"] < fine_tile_y_max)\n )\n filtered_indices = np.where(tile_filter)[0]\n\n keep_cells = cell_ids[filtered_indices]\n fine_tile_cells = coarse_tile.loc[keep_cells, [\"GEOMETRY\"]]\n fine_tile_cells = fine_tile_cells.assign(name=fine_tile_cells.index)\n\n if not fine_tile_cells.empty:\n filename = f\"{path_output}/cell_tile_{fine_i}_{fine_j}.parquet\"\n fine_tile_cells.to_parquet(filename)\n\n def process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y):\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n futures.append(executor.submit(\n filter_and_save_fine_boundary, coarse_tile, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_output\n ))\n\n for future in futures:\n future.result()\n\n tile_size_x = tile_size\n tile_size_y = tile_size\n\n transformation_matrix = pd.read_csv(path_transformation_matrix, header=None, sep=\" \").values\n\n # Load cell boundary data based on the technology\n if technology == \"MERSCOPE\":\n df_meta = pd.read_parquet(f\"{path_output.replace('cell_segmentation','cell_metadata.parquet')}\")\n entity_to_cell_id_dict = pd.Series(df_meta.index.values, index=df_meta.EntityID).to_dict()\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n cells_orig['cell_id'] = cells_orig['EntityID'].map(entity_to_cell_id_dict)\n cells_orig = cells_orig[cells_orig[\"ZIndex\"] == 1]\n\n # Correct cell_id issues with meta_cell\n meta_cell = pd.read_csv(path_meta_cell_micron)\n meta_cell['cell_id'] = meta_cell['EntityID'].map(entity_to_cell_id_dict)\n cells_orig.index = meta_cell[meta_cell[\"cell_id\"].isin(cells_orig['cell_id'])].index\n\n # Correct 'MultiPolygon' to 'Polygon'\n cells_orig[\"geometry\"] = cells_orig[\"Geometry\"].apply(\n lambda x: list(x.geoms)[0] if isinstance(x, MultiPolygon) else x\n )\n\n cells_orig.set_index('cell_id', inplace=True)\n\n elif technology == \"Xenium\":\n xenium_cells = pd.read_parquet(path_cell_boundaries)\n grouped = xenium_cells.groupby(\"cell_id\")[[\"vertex_x\", \"vertex_y\"]].agg(lambda x: x.tolist())\n grouped[\"geometry\"] = grouped.apply(lambda row: Polygon(zip(row[\"vertex_x\"], row[\"vertex_y\"])), axis=1)\n cells_orig = gpd.GeoDataFrame(grouped, geometry=\"geometry\")[[\"geometry\"]]\n\n elif technology == \"custom\":\n cells_orig = gpd.read_parquet(path_cell_boundaries)\n\n # Transform geometries\n cells_orig[\"GEOMETRY\"] = batch_transform_geometries(cells_orig[\"geometry\"], transformation_matrix, image_scale)\n\n # Convert transformed geometries to polygons and calculate centroids\n cells_orig[\"polygon\"] = cells_orig[\"GEOMETRY\"].apply(lambda x: Polygon(x[0]))\n gdf_cells = gpd.GeoDataFrame(geometry=cells_orig[\"polygon\"])\n gdf_cells[\"center_x\"] = gdf_cells.geometry.centroid.x\n gdf_cells[\"center_y\"] = gdf_cells.geometry.centroid.y\n gdf_cells[\"GEOMETRY\"] = cells_orig[\"GEOMETRY\"]\n\n # Ensure the output directory exists\n if not os.path.exists(path_output):\n os.makedirs(path_output)\n\n # Calculate tile bounds and fine/coarse tiles\n x_min, x_max = tile_bounds[\"x_min\"], tile_bounds[\"x_max\"]\n y_min, y_max = tile_bounds[\"y_min\"], tile_bounds[\"y_max\"]\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Process coarse tiles in parallel\n for i in tqdm(range(n_coarse_tiles_x), desc=\"Processing coarse tiles\"):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n coarse_tile = gdf_cells[\n (gdf_cells[\"center_x\"] >= coarse_tile_x_min) & (gdf_cells[\"center_x\"] < coarse_tile_x_max) &\n (gdf_cells[\"center_y\"] >= coarse_tile_y_min) & (gdf_cells[\"center_y\"] < coarse_tile_y_max)\n ]\n if not coarse_tile.empty:\n process_fine_boundaries(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_output, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid","title":"<code>make_deepzoom_pyramid(image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix='.jpeg')</code>","text":"<p>Create a DeepZoom image pyramid from a JPEG image</p>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid--parameters","title":"Parameters","text":"<p>image_path : str Path to the JPEG image file tile_size : int (default=512) Tile size for the DeepZoom pyramid overlap : int (default=0) Overlap size for the DeepZoom pyramid suffix : str (default='jpeg') Suffix for the DeepZoom pyramid tiles</p>"},{"location":"python/pre/api/#celldega.pre.make_deepzoom_pyramid--returns","title":"Returns","text":"<p>None</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_deepzoom_pyramid(\n image_path, output_path, pyramid_name, tile_size=512, overlap=0, suffix=\".jpeg\"\n):\n \"\"\"\n Create a DeepZoom image pyramid from a JPEG image\n\n Parameters\n ----------\n image_path : str\n Path to the JPEG image file\n tile_size : int (default=512)\n Tile size for the DeepZoom pyramid\n overlap : int (default=0)\n Overlap size for the DeepZoom pyramid\n suffix : str (default='jpeg')\n Suffix for the DeepZoom pyramid tiles\n\n Returns\n -------\n None\n\n \"\"\"\n\n # Define the output path\n output_path = Path(output_path)\n\n # Load the JPEG image\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n # check if the output path exists and create it if it does not\n output_path.mkdir(parents=True, exist_ok=True)\n\n # append the pyramid name to the output path\n output_path = output_path / pyramid_name\n\n # Save the image as a DeepZoom image pyramid\n image.dzsave(output_path, tile_size=tile_size, overlap=overlap, suffix=suffix)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord","title":"<code>make_meta_cell_image_coord(technology, path_transformation_matrix, path_meta_cell_micron, path_meta_cell_image, image_scale)</code>","text":"<p>Apply an affine transformation to the cell coordinates in microns and save the transformed coordinates in pixels</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_transformation_matrix : str Path to the transformation matrix file path_meta_cell_micron : str Path to the meta cell file with coordinates in microns path_meta_cell_image : str Path to save the meta cell file with coordinates in pixels</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_cell_image_coord--examples","title":"Examples","text":"<p>make_meta_cell_image_coord( ... technology='Xenium', ... path_transformation_matrix='data/transformation_matrix.txt', ... path_meta_cell_micron='data/meta_cell_micron.csv', ... path_meta_cell_image='data/meta_cell_image.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_cell_image_coord(\n technology,\n path_transformation_matrix,\n path_meta_cell_micron,\n path_meta_cell_image,\n image_scale\n):\n \"\"\"\n Apply an affine transformation to the cell coordinates in microns and save\n the transformed coordinates in pixels\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_transformation_matrix : str\n Path to the transformation matrix file\n path_meta_cell_micron : str\n Path to the meta cell file with coordinates in microns\n path_meta_cell_image : str\n Path to save the meta cell file with coordinates in pixels\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_cell_image_coord(\n ... technology='Xenium',\n ... path_transformation_matrix='data/transformation_matrix.txt',\n ... path_meta_cell_micron='data/meta_cell_micron.csv',\n ... path_meta_cell_image='data/meta_cell_image.parquet'\n ... )\n\n \"\"\"\n\n transformation_matrix = pd.read_csv(\n path_transformation_matrix, header=None, sep=\" \"\n ).values\n\n if technology == \"MERSCOPE\":\n meta_cell = pd.read_csv(path_meta_cell_micron, usecols=[\"EntityID\", \"center_x\", \"center_y\"])\n meta_cell = convert_long_id_to_short(meta_cell)\n meta_cell[\"name\"] = meta_cell[\"cell_id\"]\n meta_cell = meta_cell.set_index('cell_id')\n elif technology == \"Xenium\":\n usecols = [\"cell_id\", \"x_centroid\", \"y_centroid\"]\n meta_cell = pd.read_csv(path_meta_cell_micron, index_col=0, usecols=usecols)\n meta_cell.columns = [\"center_x\", \"center_y\"]\n meta_cell[\"name\"] = pd.Series(meta_cell.index, index=meta_cell.index)\n\n # Adding a ones column to accommodate for affine transformation\n meta_cell[\"ones\"] = 1\n\n # Preparing the data for matrix multiplication\n points = meta_cell[[\"center_x\", \"center_y\", \"ones\"]].values\n\n # Applying the transformation matrix\n transformed_points = np.dot(transformation_matrix, points.T).T\n\n # Updating the DataFrame with transformed coordinates\n meta_cell[\"center_x\"] = transformed_points[:, 0]\n meta_cell[\"center_y\"] = transformed_points[:, 1]\n\n # Dropping the ones column as it's no longer needed\n meta_cell.drop(columns=[\"ones\"], inplace=True)\n\n meta_cell[\"center_x\"] = meta_cell[\"center_x\"] / image_scale\n meta_cell[\"center_y\"] = meta_cell[\"center_y\"] / image_scale\n\n meta_cell[\"geometry\"] = meta_cell.apply(\n lambda row: [row[\"center_x\"], row[\"center_y\"]], axis=1\n )\n\n if technology == \"MERSCOPE\":\n meta_cell = meta_cell[[\"name\", \"geometry\", \"EntityID\"]]\n else:\n meta_cell = meta_cell[[\"name\", \"geometry\"]]\n\n\n meta_cell.to_parquet(path_meta_cell_image)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene","title":"<code>make_meta_gene(technology, path_cbg, path_output)</code>","text":"<p>Create a DataFrame with genes and their assigned colors</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--parameters","title":"Parameters","text":"<p>technology : str The technology used to generate the data, Xenium and MERSCOPE are supported. path_cbg : str Path to the cell-by-gene matrix data (the data format can vary based on technology) path_output : str Path to save the meta gene file</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--returns","title":"Returns","text":"<p>None</p>"},{"location":"python/pre/api/#celldega.pre.make_meta_gene--examples","title":"Examples","text":"<p>make_meta_gene( ... technology='Xenium', ... path_cbg='data/', ... path_output='data/meta_gene.parquet' ... )</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def make_meta_gene(technology, path_cbg, path_output):\n \"\"\"\n Create a DataFrame with genes and their assigned colors\n\n Parameters\n ----------\n technology : str\n The technology used to generate the data, Xenium and MERSCOPE are supported.\n path_cbg : str\n Path to the cell-by-gene matrix data (the data format can vary based on technology)\n path_output : str\n Path to save the meta gene file\n\n Returns\n -------\n None\n\n Examples\n --------\n >>> make_meta_gene(\n ... technology='Xenium',\n ... path_cbg='data/',\n ... path_output='data/meta_gene.parquet'\n ... )\n \"\"\"\n\n if technology == \"MERSCOPE\":\n cbg = pd.read_csv(path_cbg, index_col=0)\n genes = cbg.columns.tolist()\n elif technology == \"Xenium\":\n # genes = pd.read_csv(path_cbg + 'features.tsv.gz', sep='\\t', header=None)[1].values.tolist()\n cbg = read_cbg_mtx(path_cbg)\n genes = cbg.columns.tolist()\n\n # Get all categorical color palettes from Matplotlib and flatten them into a single list of colors\n palettes = [plt.get_cmap(name).colors for name in plt.colormaps() if \"tab\" in name]\n flat_colors = [color for palette in palettes for color in palette]\n\n # Convert RGB tuples to hex codes\n flat_colors_hex = [to_hex(color) for color in flat_colors]\n\n # Use modular arithmetic to assign a color to each gene, white for genes with \"Blank\"\n colors = [\n flat_colors_hex[i % len(flat_colors_hex)] if \"Blank\" not in gene else \"#FFFFFF\"\n for i, gene in enumerate(genes)\n ]\n\n # Create a DataFrame with genes and their assigned colors\n ser_color = pd.Series(colors, index=genes)\n\n # calculate gene expression metadata\n meta_gene = calc_meta_gene_data(cbg)\n meta_gene['color'] = ser_color\n\n # Identify sparse columns\n sparse_cols = [col for col in meta_gene.columns if pd.api.types.is_sparse(meta_gene[col])]\n\n # Convert sparse columns to dense\n for col in sparse_cols:\n meta_gene[col] = meta_gene[col].sparse.to_dense()\n\n meta_gene.to_parquet(path_output)\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles","title":"<code>make_trx_tiles(technology, path_trx, path_transformation_matrix, path_trx_tiles, coarse_tile_factor=10, tile_size=250, chunk_size=1000000, verbose=False, image_scale=1, max_workers=8)</code>","text":"<p>Processes transcript data by dividing it into coarse-grain and fine-grain tiles, applying transformations, and saving the results in a parallelized manner.</p>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles--parameters","title":"Parameters","text":"<p>technology : str The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\"). path_trx : str Path to the file containing the transcript data. path_transformation_matrix : str Path to the file containing the transformation matrix (CSV file). path_trx_tiles : str Directory path where the output files (Parquet files) for each tile will be saved. coarse_tile_factor : int, optional Scaling factor of each coarse-grain tile comparing to the fine tile size. tile_size : int, optional Size of each fine-grain tile in microns (default is 250). chunk_size : int, optional Number of rows to process per chunk for memory efficiency (default is 1000000). verbose : bool, optional Flag to enable verbose output (default is False). image_scale : float, optional Scale factor to apply to the transcript coordinates (default is 0.5). max_workers : int, optional Maximum number of parallel workers for processing tiles (default is 8).</p>"},{"location":"python/pre/api/#celldega.pre.make_trx_tiles--returns","title":"Returns","text":"<p>dict A dictionary containing the bounds of the processed data in both x and y directions.</p> Source code in <code>src/celldega/pre/trx_tile.py</code> <pre><code>def make_trx_tiles(\n technology,\n path_trx,\n path_transformation_matrix,\n path_trx_tiles,\n coarse_tile_factor=10,\n tile_size=250,\n chunk_size=1000000,\n verbose=False,\n image_scale=1,\n max_workers=8\n):\n \"\"\"\n Processes transcript data by dividing it into coarse-grain and fine-grain tiles,\n applying transformations, and saving the results in a parallelized manner.\n\n Parameters\n ----------\n technology : str\n The technology used for generating the transcript data (e.g., \"MERSCOPE\" or \"Xenium\").\n path_trx : str\n Path to the file containing the transcript data.\n path_transformation_matrix : str\n Path to the file containing the transformation matrix (CSV file).\n path_trx_tiles : str\n Directory path where the output files (Parquet files) for each tile will be saved.\n coarse_tile_factor : int, optional\n Scaling factor of each coarse-grain tile comparing to the fine tile size.\n tile_size : int, optional\n Size of each fine-grain tile in microns (default is 250).\n chunk_size : int, optional\n Number of rows to process per chunk for memory efficiency (default is 1000000).\n verbose : bool, optional\n Flag to enable verbose output (default is False).\n image_scale : float, optional\n Scale factor to apply to the transcript coordinates (default is 0.5).\n max_workers : int, optional\n Maximum number of parallel workers for processing tiles (default is 8).\n\n Returns\n -------\n dict\n A dictionary containing the bounds of the processed data in both x and y directions.\n \"\"\"\n\n def process_coarse_tile(trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers):\n # Filter the entire dataset for the current coarse tile\n coarse_tile = trx.filter(\n (pl.col(\"transformed_x\") >= coarse_tile_x_min) & (pl.col(\"transformed_x\") < coarse_tile_x_max) &\n (pl.col(\"transformed_y\") >= coarse_tile_y_min) & (pl.col(\"transformed_y\") < coarse_tile_y_max)\n )\n\n if not coarse_tile.is_empty():\n # Now process fine tiles using global fine tile indices\n process_fine_tiles(coarse_tile, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers) \n\n\n def process_fine_tiles(coarse_tile, coarse_i, coarse_j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers=8):\n\n # Use ThreadPoolExecutor for parallel processing of fine-grain tiles within the coarse tile\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n\n # Iterate over fine-grain tiles within the global bounds\n for fine_i in range(n_fine_tiles_x):\n fine_tile_x_min = x_min + fine_i * tile_size\n fine_tile_x_max = fine_tile_x_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_x_min >= coarse_tile_x_min and fine_tile_x_max <= coarse_tile_x_max):\n continue\n\n for fine_j in range(n_fine_tiles_y):\n fine_tile_y_min = y_min + fine_j * tile_size\n fine_tile_y_max = fine_tile_y_min + tile_size\n\n # Process only if the fine tile falls within the current coarse tile's bounds\n if not (fine_tile_y_min >= coarse_tile_y_min and fine_tile_y_max <= coarse_tile_y_max):\n continue\n\n # Submit the task for each fine tile to process in parallel\n futures.append(executor.submit(\n filter_and_save_fine_tile, coarse_tile, coarse_i, coarse_j, fine_i, fine_j, \n fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles\n ))\n\n # Wait for all futures to complete\n for future in concurrent.futures.as_completed(futures):\n future.result() # Raise exceptions if any occurred during execution\n\n\n def filter_and_save_fine_tile(coarse_tile, coarse_i, coarse_j, fine_i, fine_j, fine_tile_x_min, fine_tile_x_max, fine_tile_y_min, fine_tile_y_max, path_trx_tiles):\n\n # Filter the coarse tile for the current fine tile's boundaries\n fine_tile_trx = coarse_tile.filter(\n (pl.col(\"transformed_x\") >= fine_tile_x_min) & (pl.col(\"transformed_x\") < fine_tile_x_max) &\n (pl.col(\"transformed_y\") >= fine_tile_y_min) & (pl.col(\"transformed_y\") < fine_tile_y_max)\n )\n\n if not fine_tile_trx.is_empty():\n # Add geometry column as a list of [x, y] pairs\n fine_tile_trx = fine_tile_trx.with_columns(\n pl.concat_list([pl.col(\"transformed_x\"), pl.col(\"transformed_y\")]).alias(\"geometry\")\n ).drop(['transformed_x', 'transformed_y'])\n\n # Define the filename based on fine tile coordinates\n filename = f\"{path_trx_tiles}/transcripts_tile_{fine_i}_{fine_j}.parquet\"\n\n # Save the filtered DataFrame to a Parquet file\n fine_tile_trx.to_pandas().to_parquet(filename)\n\n\n # Load transformation matrix\n transformation_matrix = np.loadtxt(path_transformation_matrix)\n\n # Load the transcript data based on the technology using Polars\n if technology == \"MERSCOPE\":\n trx_ini = pl.read_csv(path_trx, columns=[\"gene\", \"global_x\", \"global_y\"])\n trx_ini = trx_ini.with_columns([\n pl.col(\"global_x\").alias(\"x\"),\n pl.col(\"global_y\").alias(\"y\"),\n pl.col(\"gene\").alias(\"name\")\n ]).select([\"name\", \"x\", \"y\"])\n\n elif technology == \"Xenium\":\n trx_ini = pl.read_parquet(path_trx).select([\n pl.col(\"feature_name\").alias(\"name\"),\n pl.col(\"x_location\").alias(\"x\"),\n pl.col(\"y_location\").alias(\"y\")\n ])\n\n # Process the data in chunks and apply transformations\n all_chunks = []\n\n for start_row in tqdm(range(0, trx_ini.height, chunk_size), desc=\"Processing chunks\"):\n chunk = trx_ini.slice(start_row, chunk_size)\n\n # Apply transformation matrix to the coordinates\n points = np.hstack([chunk.select([\"x\", \"y\"]).to_numpy(), np.ones((chunk.height, 1))])\n transformed_points = np.dot(points, transformation_matrix.T)[:, :2]\n\n # Create new transformed columns and drop original x, y columns\n transformed_chunk = chunk.with_columns([\n (pl.Series(transformed_points[:, 0]) * image_scale).round(2).alias(\"transformed_x\"),\n (pl.Series(transformed_points[:, 1]) * image_scale).round(2).alias(\"transformed_y\")\n ]).drop([\"x\", \"y\"])\n all_chunks.append(transformed_chunk)\n\n # Concatenate all chunks after processing\n trx = pl.concat(all_chunks)\n\n # Ensure the output directory exists\n if not os.path.exists(path_trx_tiles):\n os.makedirs(path_trx_tiles)\n\n # Get min and max x, y values\n x_min, x_max = trx.select([\n pl.col(\"transformed_x\").min().alias(\"x_min\"),\n pl.col(\"transformed_x\").max().alias(\"x_max\")\n ]).row(0)\n\n y_min, y_max = trx.select([\n pl.col(\"transformed_y\").min().alias(\"y_min\"),\n pl.col(\"transformed_y\").max().alias(\"y_max\")\n ]).row(0)\n\n # Calculate the number of fine-grain tiles globally\n n_fine_tiles_x = int(np.ceil((x_max - x_min) / tile_size))\n n_fine_tiles_y = int(np.ceil((y_max - y_min) / tile_size))\n\n # Calculate the number of coarse-grain tiles\n n_coarse_tiles_x = int(np.ceil((x_max - x_min) / (coarse_tile_factor * tile_size)))\n n_coarse_tiles_y = int(np.ceil((y_max - y_min) / (coarse_tile_factor * tile_size)))\n\n # Use ThreadPoolExecutor for parallel processing of coarse-grain tiles\n with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor:\n futures = []\n for i in range(n_coarse_tiles_x):\n coarse_tile_x_min = x_min + i * (coarse_tile_factor * tile_size)\n coarse_tile_x_max = coarse_tile_x_min + (coarse_tile_factor * tile_size)\n\n for j in range(n_coarse_tiles_y):\n coarse_tile_y_min = y_min + j * (coarse_tile_factor * tile_size)\n coarse_tile_y_max = coarse_tile_y_min + (coarse_tile_factor * tile_size)\n\n # Submit each coarse tile for parallel processing\n futures.append(executor.submit(\n process_coarse_tile, trx, i, j, coarse_tile_x_min, coarse_tile_x_max, coarse_tile_y_min, coarse_tile_y_max, tile_size, path_trx_tiles, x_min, y_min, n_fine_tiles_x, n_fine_tiles_y, max_workers\n ))\n\n # Wait for all coarse tiles to complete\n for future in tqdm(concurrent.futures.as_completed(futures), desc=\"Processing coarse tiles\", unit=\"tile\"):\n future.result() # Raise exceptions if any occurred during execution\n\n # Return the tile bounds\n tile_bounds = {\n \"x_min\": x_min,\n \"x_max\": x_max,\n \"y_min\": y_min,\n \"y_max\": y_max,\n }\n\n return tile_bounds\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.reduce_image_size","title":"<code>reduce_image_size(image_path, scale_image=0.5, path_landscape_files='')</code>","text":""},{"location":"python/pre/api/#celldega.pre.reduce_image_size--parameters","title":"Parameters","text":"<p>image_path : str Path to the image file scale_image : float (default=0.5) Scale factor for the image resize</p>"},{"location":"python/pre/api/#celldega.pre.reduce_image_size--returns","title":"Returns","text":"<p>new_image_path : str Path to the resized image file</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def reduce_image_size(image_path, scale_image=0.5, path_landscape_files=\"\"):\n \"\"\"\n\n Parameters\n ----------\n image_path : str\n Path to the image file\n scale_image : float (default=0.5)\n Scale factor for the image resize\n\n Returns\n -------\n new_image_path : str\n Path to the resized image file\n \"\"\"\n\n image = pyvips.Image.new_from_file(image_path, access=\"sequential\")\n\n resized_image = image.resize(scale_image)\n\n new_image_name = image_path.split(\"/\")[-1].replace(\".tif\", \"_downsize.tif\")\n new_image_path = f\"{path_landscape_files}/{new_image_name}\"\n resized_image.write_to_file(new_image_path)\n\n return new_image_path\n</code></pre>"},{"location":"python/pre/api/#celldega.pre.save_landscape_parameters","title":"<code>save_landscape_parameters(technology, path_landscape_files, image_name='dapi_files', tile_size=1000, image_info={}, image_format='.webp')</code>","text":"<p>Save the landscape parameters to a JSON file.</p> Source code in <code>src/celldega/pre/__init__.py</code> <pre><code>def save_landscape_parameters(\n technology, path_landscape_files, image_name=\"dapi_files\", tile_size=1000, image_info={}, image_format='.webp'\n):\n \"\"\"\n Save the landscape parameters to a JSON file.\n \"\"\"\n\n path_image_pyramid = f\"{path_landscape_files}/pyramid_images/{image_name}\"\n\n print(path_image_pyramid)\n\n max_pyramid_zoom = get_max_zoom_level(path_image_pyramid)\n\n landscape_parameters = {\n \"technology\": technology,\n \"max_pyramid_zoom\": max_pyramid_zoom,\n \"tile_size\": tile_size,\n \"image_info\": image_info,\n \"image_format\": image_format\n }\n\n path_landscape_parameters = f\"{path_landscape_files}/landscape_parameters.json\"\n\n with open(path_landscape_parameters, \"w\") as file:\n json.dump(landscape_parameters, file, indent=4)\n</code></pre>"},{"location":"python/viz/api/","title":"Viz Module API Reference","text":""},{"location":"python/viz/api/#widget-classes","title":"Widget Classes","text":""},{"location":"python/viz/api/#celldega.viz.widget.Landscape","title":"<code>Landscape</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of spatial omics data. This widget currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)</p> <p>Parameters:</p> Name Type Description Default <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> required <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> required <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> required <code>token</code> <code>str</code> <p>The token traitlet.</p> required <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> required <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize. This will show up in the user interface bar.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>technology</code> <code>str</code> <p>The technology used.</p> <code>base_url</code> <code>str</code> <p>The base URL for the widget.</p> <code>token</code> <code>str</code> <p>The token traitlet.</p> <code>ini_x</code> <code>float</code> <p>The initial x-coordinate of the view.</p> <code>ini_y</code> <code>float</code> <p>The initial y-coordinate of the view.</p> <code>ini_z</code> <code>float</code> <p>The initial z-coordinate of the view.</p> <code>ini_zoom</code> <code>float</code> <p>The initial zoom level of the view.</p> <code>dataset_name</code> <code>str</code> <p>The name of the dataset to visualize.</p> <code>update_trigger</code> <code>dict</code> <p>The dictionary to trigger updates.</p> <code>cell_clusters</code> <code>dict</code> <p>The dictionary containing cell cluster information.</p> <p>Returns:</p> Name Type Description <code>Landscape</code> <p>A widget for visualizing a 'landscape' view of spatial omics data.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Landscape(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of spatial omics data. This widget\n currently supports iST (Xenium and MERSCOPE) and sST (Visium HD data)\n\n Args:\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n token (str): The token traitlet.\n base_url (str): The base URL for the widget.\n dataset_name (str, optional): The name of the dataset to visualize. This will show up in the user interface bar.\n\n Attributes:\n component (str): The name of the component.\n technology (str): The technology used.\n base_url (str): The base URL for the widget.\n token (str): The token traitlet.\n ini_x (float): The initial x-coordinate of the view.\n ini_y (float): The initial y-coordinate of the view.\n ini_z (float): The initial z-coordinate of the view.\n ini_zoom (float): The initial zoom level of the view.\n dataset_name (str): The name of the dataset to visualize.\n update_trigger (dict): The dictionary to trigger updates.\n cell_clusters (dict): The dictionary containing cell cluster information.\n\n Returns:\n Landscape: A widget for visualizing a 'landscape' view of spatial omics data.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n component = traitlets.Unicode(\"Landscape\").tag(sync=True)\n\n technology = traitlets.Unicode(\"sst\").tag(sync=True)\n base_url = traitlets.Unicode(\"\").tag(sync=True)\n token = traitlets.Unicode(\"\").tag(sync=True)\n ini_x = traitlets.Float(1000).tag(sync=True)\n ini_y = traitlets.Float(1000).tag(sync=True)\n ini_z = traitlets.Float(0).tag(sync=True)\n ini_zoom = traitlets.Float(0).tag(sync=True)\n square_tile_size = traitlets.Float(1.4).tag(sync=True)\n dataset_name = traitlets.Unicode(\"\").tag(sync=True)\n region = traitlets.Dict({}).tag(sync=True)\n\n update_trigger = traitlets.Dict().tag(sync=True)\n cell_clusters = traitlets.Dict().tag(sync=True)\n\n width = traitlets.Int(0).tag(sync=True)\n height = traitlets.Int(800).tag(sync=True)\n\n def trigger_update(self, new_value):\n # This method updates the update_trigger traitlet with a new value\n # You can pass any information necessary for the update, or just a timestamp\n self.update_trigger = new_value\n\n def update_cell_clusters(self, new_clusters):\n # Convert the new_clusters to a JSON serializable format if necessary\n self.cell_clusters = new_clusters\n</code></pre>"},{"location":"python/viz/api/#celldega.viz.widget.Matrix","title":"<code>Matrix</code>","text":"<p> Bases: <code>AnyWidget</code></p> <p>A widget for interactive visualization of a hierarchically clustered matrix.</p> <p>Parameters:</p> Name Type Description Default <code>value</code> <code>int</code> <p>The value traitlet.</p> required <code>component</code> <code>str</code> <p>The component traitlet.</p> required <code>network</code> <code>dict</code> <p>The network traitlet.</p> required <code>click_info</code> <code>dict</code> <p>The click_info traitlet.</p> required <p>Attributes:</p> Name Type Description <code>component</code> <code>str</code> <p>The name of the component.</p> <code>network</code> <code>dict</code> <p>The network dictionary.</p> <code>click_info</code> <code>dict</code> <p>The click_info dictionary.</p> <p>Returns:</p> Name Type Description <code>Matrix</code> <p>A widget for visualizing a hierarchically clustered matrix.</p> Source code in <code>src/celldega/viz/widget.py</code> <pre><code>class Matrix(anywidget.AnyWidget):\n \"\"\"\n A widget for interactive visualization of a hierarchically clustered matrix.\n\n Args:\n value (int): The value traitlet.\n component (str): The component traitlet.\n network (dict): The network traitlet.\n click_info (dict): The click_info traitlet.\n\n Attributes:\n component (str): The name of the component.\n network (dict): The network dictionary.\n click_info (dict): The click_info dictionary.\n\n Returns:\n Matrix: A widget for visualizing a hierarchically clustered matrix.\n \"\"\"\n _esm = pathlib.Path(__file__).parent / \"../static\" / \"widget.js\"\n _css = pathlib.Path(__file__).parent / \"../static\" / \"widget.css\"\n value = traitlets.Int(0).tag(sync=True)\n component = traitlets.Unicode(\"Matrix\").tag(sync=True)\n\n network = traitlets.Dict({}).tag(sync=True)\n click_info = traitlets.Dict({}).tag(sync=True)\n</code></pre>"},{"location":"technologies/","title":"Technologies","text":"<p>Celldega utilizes a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit.</p>"},{"location":"technologies/#visualization-technologies","title":"Visualization Technologies","text":"<p>Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega <code>viz</code> module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio).</p> <p>The Celldega Landscape visualization method (see Gallery) utilizes novel vector tiling approaches to enable interactive visualization of large ST datasets in a notebook environment or as a stand-alone webpage. This approach allows Celldega to visualize larger datasets than currently available open-source tools (e.g., datasets with hundreds of millions of transcripts). We also utilize modern web image data formats (WebP) to reduce the data storage burden for interactive visualization. The resulting LandscapeFiles data format serves as a compact and highly performant visualization-specific data format.</p>"},{"location":"technologies/#terrabio","title":"Terra.bio","text":"<p>Terra.bio is a cloud-based compute and data storage platform that is being developed by the Broad Institute of MIT and Harvard. We are utilizing Terra.bio to help Spatial Technology Platform clients access, analyze, and visualize their ST data.</p>"},{"location":"technologies/#jupyter-widget","title":"Jupyter Widget","text":"<p>We utilize the Jupyter Widget ecosystem to build interactive spatial and data visualizations that enable users to perform two way communication between JavaScript (front-end) and Python (back-end). We are utilizing the AnyWidget implementation to build our custom widgets.</p>"},{"location":"technologies/#deckgl","title":"Deck.gl","text":"<p>Celldega uses the GPU-powered data visualization library deck.gl to create high-performance spatial- and data-visualizations.</p>"},{"location":"technologies/#apache-parquet","title":"Apache Parquet","text":"<p>Celldega uses the Apache Parquet file format for storing vectorized spatial data and metadata. This file format in combination with the JavaScript library ParquetWASM and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see GeoArrow and GeoParquet in deck.gl).</p>"},{"location":"technologies/#parquetwasm-and-apache-arrow","title":"ParquetWASM and Apache Arrow","text":"<p>ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a columnar in-memory format that is used for storing data from Apache Parquet files and efficiently passing to deck.gl. For more information please see GeoArrow and GeoParquet in deck.gl.</p>"},{"location":"technologies/#webp","title":"WebP","text":"<p>A modern image format developed by Google, offering efficient lossless compression and designed specifically for the web.</p>"},{"location":"technologies/#deep-zoom","title":"Deep Zoom","text":"<p>We utilize the Deep Zoom image schema, developed by Microsoft, to enable efficient visualization of large multi-channel microscopy image. Deep Zoom tile images are stored using the WebP image format.</p>"},{"location":"technologies/#clustergrammer-visualization","title":"Clustergrammer Visualization","text":""},{"location":"technologies/#data-analysis-technologies","title":"Data Analysis Technologies","text":""},{"location":"technologies/#geopandas","title":"GeoPandas","text":""},{"location":"technologies/#libpysal-python-spatial-analysis-library-core","title":"LibPySal: Python Spatial Analysis Library Core","text":""},{"location":"technologies/#clustergrammer-data-analysis","title":"Clustergrammer Data Analysis","text":""}]} \ No newline at end of file diff --git a/technologies/index.html b/technologies/index.html index 4f22d2ff..6e7042c8 100644 --- a/technologies/index.html +++ b/technologies/index.html @@ -989,9 +989,9 @@ </li> <li class="md-nav__item"> - <a href="#deepzoom" class="md-nav__link"> + <a href="#deep-zoom" class="md-nav__link"> <span class="md-ellipsis"> - DeepZoom + Deep Zoom </span> </a> @@ -1069,9 +1069,9 @@ <h1 id="technologies">Technologies</h1> -<p>Celldega is utilizing a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit.</p> +<p>Celldega utilizes a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit.</p> <h2 id="visualization-technologies">Visualization Technologies</h2> -<p>Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is easily linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega <a href="../python/viz/"><code>viz</code></a> module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio).</p> +<p>Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega <a href="../python/viz/"><code>viz</code></a> module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio).</p> <p>The Celldega Landscape visualization method (see <a href="../gallery">Gallery</a>) utilizes novel vector tiling approaches to enable interactive visualization of large ST datasets in a notebook environment or as a stand-alone webpage. This approach allows Celldega to visualize larger datasets than currently available open-source tools (e.g., datasets with hundreds of millions of transcripts). We also utilize modern web image data formats (<a href="#webp">WebP</a>) to reduce the data storage burden for interactive visualization. The resulting <a href="../overview/file_formats/#landscapefiles">LandscapeFiles</a> data format serves as a compact and highly performant visualization-specific data format.</p> <h3 id="terrabio">Terra.bio</h3> <p>Terra.bio is a cloud-based compute and data storage platform that is being developed by the <a href='https://www.broadinstitute.org/spatial-technology-platform' target='_blank'>Broad Institute of MIT and Harvard</a>. We are utilizing Terra.bio to help <a href='https://www.broadinstitute.org/spatial-technology-platform' target='_blank'>Spatial Technology Platform</a> clients access, analyze, and visualize their ST data.</p> @@ -1080,11 +1080,13 @@ <h3 id="jupyter-widget">Jupyter Widget</h3> <h3 id="deckgl">Deck.gl</h3> <p>Celldega uses the GPU-powered data visualization library <a href='https://deck.gl/' target='_blank'>deck.gl</a> to create high-performance spatial- and data-visualizations.</p> <h3 id="apache-parquet">Apache Parquet</h3> -<p>Celldega uses the <a href='https://parquet.apache.org/' target='_blank'>Apache Parquet</a> file format for storing spatial data. This file format in combination with the JavaScript library <a href="#parquetwasm-and-apache-arrow">ParquetWASM</a> and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>).</p> +<p>Celldega uses the <a href='https://parquet.apache.org/' target='_blank'>Apache Parquet</a> file format for storing vectorized spatial data and metadata. This file format in combination with the JavaScript library <a href="#parquetwasm-and-apache-arrow">ParquetWASM</a> and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>).</p> <h3 id="parquetwasm-and-apache-arrow">ParquetWASM and Apache Arrow</h3> -<p>ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a complementary columnar in-memory format for storing data from Apache Parquet files. See <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a> for more information.</p> +<p>ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a columnar in-memory format that is used for storing data from Apache Parquet files and efficiently passing to deck.gl. For more information please see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>.</p> <h3 id="webp">WebP</h3> -<h3 id="deepzoom">DeepZoom</h3> +<p>A modern image format developed by Google, offering efficient lossless compression and designed specifically for the web.</p> +<h3 id="deep-zoom">Deep Zoom</h3> +<p>We utilize the Deep Zoom image schema, developed by Microsoft, to enable efficient visualization of large multi-channel microscopy image. Deep Zoom tile images are stored using the WebP image format.</p> <h3 id="clustergrammer-visualization">Clustergrammer Visualization</h3> <h2 id="data-analysis-technologies">Data Analysis Technologies</h2> <h3 id="geopandas">GeoPandas</h3> diff --git a/technologies/index.md b/technologies/index.md index 53262e5c..dcb4953d 100644 --- a/technologies/index.md +++ b/technologies/index.md @@ -1,8 +1,8 @@ # Technologies -Celldega is utilizing a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit. +Celldega utilizes a suite of complementary technologies to develop an efficient web-based spatial-omics analysis and visualization toolkit. ## Visualization Technologies -Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is easily linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega [`viz`](../python/viz/) module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio). +Spatial transcriptomics (ST) datasets can be very large and difficult for researchers to analyze and visualize collaboratively. Additionally, visualization that is linked to analysis is key to extracting biological insights. To address these issues, we built the Celldega [`viz`](../python/viz/) module to help researchers interactively visualize large ST datasets within notebook-based workflows on the cloud (e.g., Terra.bio). The Celldega Landscape visualization method (see [Gallery](../gallery)) utilizes novel vector tiling approaches to enable interactive visualization of large ST datasets in a notebook environment or as a stand-alone webpage. This approach allows Celldega to visualize larger datasets than currently available open-source tools (e.g., datasets with hundreds of millions of transcripts). We also utilize modern web image data formats ([WebP](#webp)) to reduce the data storage burden for interactive visualization. The resulting [LandscapeFiles](../overview/file_formats.md#landscapefiles) data format serves as a compact and highly performant visualization-specific data format. @@ -16,17 +16,20 @@ We utilize the Jupyter Widget ecosystem to build interactive spatial and data vi Celldega uses the GPU-powered data visualization library <a href='https://deck.gl/' target='_blank'>deck.gl</a> to create high-performance spatial- and data-visualizations. ### Apache Parquet -Celldega uses the <a href='https://parquet.apache.org/' target='_blank'>Apache Parquet</a> file format for storing spatial data. This file format in combination with the JavaScript library [ParquetWASM](#parquetwasm-and-apache-arrow) and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>). +Celldega uses the <a href='https://parquet.apache.org/' target='_blank'>Apache Parquet</a> file format for storing vectorized spatial data and metadata. This file format in combination with the JavaScript library [ParquetWASM](#parquetwasm-and-apache-arrow) and Apache Arrow in memory representation is used to build Celldega's high-performance vector tiling spatial visualization functionality (see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>). ### ParquetWASM and Apache Arrow -ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a complementary columnar in-memory format for storing data from Apache Parquet files. See <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a> for more information. +ParquetWASM is a JavaScript library for reading Parquet files into Apache Arrow memory and utilizes Web Assembly (WASM) to run Rust in a browser environment. The Apache Arrow in-memory format is a columnar in-memory format that is used for storing data from Apache Parquet files and efficiently passing to deck.gl. For more information please see <a href='https://observablehq.com/@kylebarron/geoarrow-and-geoparquet-in-deck-gl' target='_blank'>GeoArrow and GeoParquet in deck.gl</a>. ### WebP +A modern image format developed by Google, offering efficient lossless compression and designed specifically for the web. -### DeepZoom +### Deep Zoom +We utilize the Deep Zoom image schema, developed by Microsoft, to enable efficient visualization of large multi-channel microscopy image. Deep Zoom tile images are stored using the WebP image format. ### Clustergrammer Visualization + ## Data Analysis Technologies ### GeoPandas