Panorai

Panorai is a Python package for spherical (360°) image processing, specifically focusing on sampling, projection, forward/backward transformations, and unsharp masking for panoramic or spherical data. It provides a modular pipeline to handle various steps, including:

• Resizing input images
• Sampling tangent points on a sphere (e.g., cube-based, Fibonacci-based)
• Projecting spherical images into local tangential (rectilinear) planes
• Re-projecting these rectilinear images back into the equirectangular space
• Configuring and applying unsharp masking to sharpen the projected images
• Logging and comparing configuration results (e.g., MSE) via an automated test suite

This README will guide you through the repository structure, installation, usage, and testing of Panorai.

---

Table of Contents

1. Overview
2. Directory Structure
3. Installation
4. Examples
5. Key Modules and Classes
6. Running Tests
7. Extending Panorai
8. License

---

Overview

Panorai is a Python library designed for advanced geometric transformations, projections, and sampling on spherical and equirectangular data. It provides a highly modular framework for implementing gnomonic projections, backward projections, and blending strategies, suitable for 360-degree image processing and panoramic data transformations.

---

Directory Structure

/
├── panorai/
│   ├── __pycache__/
│   ├── common/
│   │   ├── __pycache__/
│   │   ├── __init__.py
│   │   └── shared_config.py
│   ├── pipeline/
│   │   ├── __pycache__/
│   │   ├── utils/
│   │   │   ├── __pycache__/
│   │   │   ├── __init__.py
│   │   │   └── resizer.py
│   │   ├── __init__.py
│   │   ├── pipeline.py
│   │   ├── pipeline_data.py
│   │   └── pipeline_old.py
│   ├── projection/
│   │   ├── __pycache__/
│   │   ├── utils/
│   │   │   ├── __pycache__/
│   │   │   ├── __init__.py
│   │   │   ├── remapper.py
│   │   │   └── unsharp.py
│   │   ├── __init__.py
│   │   ├── projector.py
│   │   └── projector_deprecated.py
│   ├── readme/
│   │   └── gnomonic.md
│   ├── sampler/
│   │   ├── __pycache__/
│   │   ├── __init__.py
│   │   └── sampler.py
│   ├── __init__.py
├── panorai.egg-info/
│   ├── dependency_links.txt
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   └── top_level.txt
├── tests/
│   ├── __pycache__/
│   └── test_config_params.py
├── .gitignore
├── best_configs_chart.png
├── requirements.txt
├── setup.py
└── test_results.db

Notable Contents

• panorai/
  Main package directory containing submodules:

  • pipeline/: The pipeline logic for forward/backward projection and data handling, plus utility functions like resizer.
  • projection/: Projector classes (e.g., GnomonicProjector) and remapping/unsharp utility code.
  • sampler/: Sphere sampling strategies (cube, icosahedron, Fibonacci).
  • readme/: Additional notes/documentation (e.g., gnomonic.md).

• setup.py
  A setuptools-based installation script.

• requirements.txt
  Lists dependencies needed to run the code (e.g., NumPy, OpenCV, etc.).

---

Installation

1. Clone the Repository:

   git clone https://github.com/yourusername/panorai.git
   cd panorai

2. Install Dependencies:

   pip install -r requirements.txt

3. Install Panorai (Editable Mode or Standard):

   pip install -e .
   # or
   python setup.py install

---

Examples

1. Data Preparation

Start by loading your input data, typically stored in an .npz file containing rgb, depth, and other channels.

import numpy as np
from panorai.pipeline.pipeline_data import PipelineData

# Load data from an NPZ file
filename = 'path/to/sample.npz'  # Replace with your file path
arr = np.load(filename)

rgb = arr['rgb']  # Shape: (H, W, 3)
depth = np.sqrt(np.sum(arr['z']**2, axis=-1))[:, :, None]  # Shape: (H, W, 1)
xyz = arr['z']  # Shape: (H, W, 3)

# Create a PipelineData instance
data = PipelineData.from_dict({
    "rgb": rgb,
    "depth": depth,
    "xyz_depth": xyz
})

2. Preprocessing Data

Adjust for shadow angle and optionally rotate the equirectangular image:

from panorai.pipeline.utils.preprocess_eq import PreprocessEquirectangularImage

# Visualize the original data
import matplotlib.pyplot as plt
plt.imshow(data.data['rgb'])
plt.show()

# Preprocess the data (e.g., handle shadow angle)
data.preprocess(shadow_angle=30)
plt.imshow(data.data['rgb'])
plt.show()

3. Using Projections

3.1. Forward Projection

Project equirectangular data into a gnomonic projection.

from panorai.submodules.projections import ProjectionRegistry

# Access the gnomonic projection
proj = ProjectionRegistry.get_projection('gnomonic', return_processor=True)

# Perform forward projection
face = proj.forward(data.data['rgb'])
plt.imshow(face)
plt.show()

3.2. Backward Projection

Reconstruct the equirectangular image from a projection:

# Perform backward projection
eq_img = proj.backward(face)
plt.imshow(eq_img)
plt.show()

4. Using the ProjectionPipeline

The ProjectionPipeline provides a high-level API for projections.

from panorai.pipeline.pipeline import ProjectionPipeline

# Initialize the pipeline
pipe = ProjectionPipeline(projection_name='gnomonic')

# Forward projection
face = pipe.project(data)
plt.imshow(face['rgb'].astype(np.uint8))
plt.show()

4.1. Using Samplers

Use samplers to generate multiple projections (e.g., cube or icosahedron samplers):

pipe = ProjectionPipeline(projection_name='gnomonic', sampler_name='CubeSampler')
faces = pipe.project(data)

# Visualize a specific face
plt.imshow(faces['point_1']['rgb'].astype(np.uint8))
plt.show()

4.2. Blending Projections

Blend multiple projections into a seamless equirectangular image:

# Reconstruct equirectangular image
reconstructed = pipe.backward(faces)
plt.imshow(reconstructed['stacked'])
plt.show()

4.3. Custom Configurations

Modify the pipeline configuration to customize the behavior:

pipe = ProjectionPipeline(projection_name='gnomonic', sampler_name='IcosahedronSampler')
faces = pipe.project(data, subdivisions=2, fov_deg=40)
reconstructed = pipe.backward(faces)
plt.imshow(reconstructed['stacked'])
plt.show()

---

Key Modules and Classes

1. PipelineData
   A container for storing and stacking multiple image channels (e.g., rgb, depth).

2. ProjectionPipeline
   Manages the forward/backward transformations using a chosen sampler and projector.

3. Samplers

   • CubeSampler: Tangent points for cube-based projections.
   • IcosahedronSampler: Icosahedron-based tangent points.
   • FibonacciSampler: Fibonacci sphere sampling for uniform distribution.

4. ProjectionRegistry

   • Project that implements the projection modules https://github.com/RobinsonGarcia/projections/tree/main

---

Extending Panorai

• Add new samplers: Implement a new class inheriting from Sampler and register it in SAMPLER_CLASSES.
• Add new projectors: Implement a new class inheriting from ProjectionStrategy and add it to PROJECTOR_CLASSES.
• HF integration to handle depth estimation: Implement seamless integration with HF to load and run DA inference

---

License

This project is licensed under the MIT License. See the LICENSE file for details.

---

Contact

For questions or feedback, contact the maintainers:

• Your Name: [email protected]
• GitHub: https://github.com/RobinsonGarcia

---

Enjoy using Panorai for your panoramic image processing!