Spectral Extension and Automatic Segmentation workflows

[rafaspadilha]

Hi, FarmVibes.AI users! 👋

We are happy to announce a new release!

This time, we have a new workflow that combines Sentinel-2 and drone imagery to produce high resolution images. We are also releasing an extension of the segmentation workflows that allows automatic segmentation of rasters without the previous requirement of user-provided prompts. These are followed by new notebooks and important backend improvements to our worker and caching services.

There are two breaking changes that need your attention:

• We created a new module vibe_notebook that groups utility functions and classes shared among the notebooks in our repo. These were previously located within notebooks/shared_nb_lib folder. Because of this change, most notebook environments will need to be recreated.
• We updated the scripts/export_sam_models.py to export ONNX models that can process multiple prompts in a batch, which is required for the new segmentation workflows. Users will need to run the script again, exporting the new models to the FarmVibes.AI cluster.

Please, refer to the summarized release notes below for more details.

Now, to update your existing clusters, if you cloned the repo, do a

git pull

to update your local copy. Then, you will have to update (or install) the vibe_core library, with

pip install ./src/vibe_core

Finally, you can upgrade your cluster by running

farmvibes-ai local update

Thank you for using FarmVibes.AI!

---

Added 🎉

• New workflow to perform spectral extension. The workflow runs a model to combine information from an RGB drone image with a Sentinel-2 product to generate a high-resolution (0.125m/px) image with 8 Sentinel-2 bands.
• New notebook showcasing the spectral extension workflow. The user may provide their own UAV image in order to generate the image.
• Two automatic segmentation workflows based on SAM (Segment Anything Model). These will automatically segment entities in Sentinel-2 rasters (farm_ai/segmentation/auto_segment_s2) and BingMaps basemaps (farm_ai/segmentation/auto_segment_basemap) without requiring the user to explictly define a set of prompts (i.e., points and/or bounding box).
• Two notebooks showcasing the automatic segmentation workflows over Sentinel-2 rasters and BingMaps basemaps. The notebooks explores several of the workflow parameters, highlighting how they affect the output segmentation masks.

Changed ↪️

• [💔 BREAKING CHANGE] We created a new module vibe_notebook with all utility methods that are shared by the example notebooks in FarmVibes.AI. Previously these methods were organized in the notebooks/shared_nb_lib directory, which was imported by the notebooks using them. Because of this, most notebook environments
  will need to be updated.
• [💔 BREAKING CHANGE] With the new segmentation workflows, we updated scripts/export_sam_models.py so the exported ONNX models can process multiple prompts in a batch. Because of this change, the segmentation workflows (both prompt-based and automatic segmentation) will not work with ONNX models exported prior to this release. Please, make sure to delete the old ONNX models in the onnx_resources folder in your storage, and re-run the exportation script.
• We updated scripts/export_sam_models.py to support adding the ONNX models to remote clusters. The script now has a new --cluster parameter to specify the cluster type (local or remote) to which the script will export the model to.
• The workflow data_processing/clip/clip now allows hard clip (explicitly clipping the raster to the input geometry) in addition to soft clip (which would only update the geometry metadata of the raster, without actually cropping the raster).
• We improved the post request of ADMA by accessing a skip token in the response. This enhancement allows users to download response results in smaller, manageable chunks.
• Both the worker and the cache services now use gRPC for internal communication, hopefully reducing latency a bit and improving reliability as well. This should fix Issue with SpaceEye Pipeline Stalling on Rerun #163.
• We updated the description of workflows that download Sentinel-1, as well as the documentation page on secrets,
  to reflect the need for a Planetary Computer API key.

Fixed ⚒️

• Fixed micromamba environment creation on Windows (see Setup - Unable to setup via pip on Windows machine and few other installation questions #164).
• Stopped logging the azurerm provider's key when creating or updating AKS clusters.

---

Co-authored-by: Bruno Silva [email protected]
Co-authored-by: Eduardo Rodrigues [email protected]
Co-authored-by: Naga Bilwanth Gangarapu [email protected]
Co-authored-by: Rafael Padilha [email protected]
Co-authored-by: Renato Luiz de Freitas Cunha [email protected]
Co-authored-by: Roberto de Moura Estevão Filho [email protected]