Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

setup backend code to make satellite predictions #3613

Merged
merged 9 commits into from
Oct 11, 2024
Merged

setup backend code to make satellite predictions #3613

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
897348c
setup backend code for satellite stuff
Mnoble-19 Oct 10, 2024
30d3a0d
code changes
Mnoble-19 Oct 10, 2024
c7e9640
Update satellite_predictions.py
Mnoble-19 Oct 10, 2024
b0c6083
Merge branch 'staging' into satellite-endpoints
Mnoble-19 Oct 11, 2024
687379c
Update src/spatial/configure.py
Mnoble-19 Oct 11, 2024
6838c69
Update requirements.txt
Mnoble-19 Oct 11, 2024
f4d81b4
Merge branch 'satellite-endpoints' of https://github.com/airqo-platfo…
Mnoble-19 Oct 11, 2024
63a53a0
Update requirements.txt
Mnoble-19 Oct 11, 2024
d7acd3d
Update Dockerfile
Mnoble-19 Oct 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
101 changes: 100 additions & 1 deletion src/spatial/configure.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
#configure.py
import os
from pathlib import Path

import gcsfs
import joblib
Mnoble-19 marked this conversation as resolved.
Show resolved Hide resolved
from dotenv import load_dotenv

BASE_DIR = Path(__file__).resolve().parent
Expand Down Expand Up @@ -41,4 +44,100 @@ class TestingConfig(Config):
environment = os.getenv("FLASK_ENV", "staging")
print("ENVIRONMENT", environment or "staging")

configuration = app_config.get(environment, "staging")
configuration = app_config.get(environment, "staging")

satellite_collections = {
'COPERNICUS/S5P/OFFL/L3_SO2': [
'SO2_column_number_density',
'SO2_column_number_density_amf',
'SO2_slant_column_number_density',
'absorbing_aerosol_index',
'cloud_fraction',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle',
'SO2_column_number_density_15km'
],
'COPERNICUS/S5P/OFFL/L3_CO': [
'CO_column_number_density',
'H2O_column_number_density',
'cloud_height',
'sensor_altitude',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
],
'COPERNICUS/S5P/OFFL/L3_NO2': [
'NO2_column_number_density',
'tropospheric_NO2_column_number_density',
'stratospheric_NO2_column_number_density',
'NO2_slant_column_number_density',
'tropopause_pressure',
'absorbing_aerosol_index',
'cloud_fraction',
'sensor_altitude',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
],
'COPERNICUS/S5P/OFFL/L3_HCHO': [
'tropospheric_HCHO_column_number_density',
'tropospheric_HCHO_column_number_density_amf',
'HCHO_slant_column_number_density',
'cloud_fraction',
'solar_zenith_angle',
'solar_azimuth_angle',
'sensor_zenith_angle',
'sensor_azimuth_angle'
],
'COPERNICUS/S5P/OFFL/L3_O3': [
'O3_column_number_density',
'O3_effective_temperature',
'cloud_fraction',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
],
'COPERNICUS/S5P/OFFL/L3_AER_AI': [
'absorbing_aerosol_index',
'sensor_altitude',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
],
'COPERNICUS/S5P/OFFL/L3_CH4': [
'CH4_column_volume_mixing_ratio_dry_air',
'aerosol_height',
'aerosol_optical_depth',
'sensor_zenith_angle',
'sensor_azimuth_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
],
'COPERNICUS/S5P/OFFL/L3_CLOUD': [
'cloud_fraction',
'cloud_top_pressure',
'cloud_top_height',
'cloud_base_pressure',
'cloud_base_height',
'cloud_optical_depth',
'surface_albedo',
'sensor_azimuth_angle',
'sensor_zenith_angle',
'solar_azimuth_angle',
'solar_zenith_angle'
]
Mnoble-19 marked this conversation as resolved.
Show resolved Hide resolved
}


def get_trained_model_from_gcs(project_name, bucket_name, source_blob_name):
fs = gcsfs.GCSFileSystem(project=project_name)
fs.ls(bucket_name)
with fs.open(bucket_name + "/" + source_blob_name, "rb") as handle:
job = joblib.load(handle)
Mnoble-19 marked this conversation as resolved.
Show resolved Hide resolved
return job
7 changes: 6 additions & 1 deletion src/spatial/controllers/controllers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from views.getis_confidence_services import SpatialDataHandler_confidence
from views.localmoran_services import SpatialDataHandler_moran
from views.derived_pm2_5 import PM25View, PM25_aod_Model_daily, Sentinel5PView, Satellite_data
from views.satellite_predictions import SatellitePredictionView
from views.site_category_view import SiteCategorizationView
from views.site_selection_views import SiteSelectionView

Expand Down Expand Up @@ -45,4 +46,8 @@ def categorize_site():

@controller_bp.route('/site_location', methods=['GET'])
def site_selection():
return SiteSelectionView.site_selection()
return SiteSelectionView.site_selection()

@controller_bp.route('/satellite_prediction', methods=['POST'])
def get_satellite_prediction():
return SatellitePredictionView.make_predictions()
48 changes: 48 additions & 0 deletions src/spatial/models/SatellitePredictionModel.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
from datetime import datetime, timezone
from typing import Dict

import ee
from google.oauth2 import service_account

from configure import Config, satellite_collections


class SatellitePredictionModel:
@staticmethod
def initialize_earth_engine():
ee.Initialize(credentials=service_account.Credentials.from_service_account_file(
Config.CREDENTIALS,
scopes=['https://www.googleapis.com/auth/earthengine']
), project=Config.GOOGLE_CLOUD_PROJECT_ID)
Mnoble-19 marked this conversation as resolved.
Show resolved Hide resolved

@staticmethod
def extract_single_point_data(longitude: float, latitude: float) -> Dict[str, float]:
aoi = ee.Geometry.Point([longitude, latitude])
current_time = datetime.now(timezone.utc)

all_features = {}

for collection, fields in satellite_collections.items():
image = ee.ImageCollection(collection) \
.filterDate(current_time.strftime('%Y-%m-%d')) \
.filterBounds(aoi) \
.first()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Ensure proper handling when no images are found in the collection

When filtering the ImageCollection, there may be cases where no images match the criteria, resulting in image being None. It's important to handle this scenario to avoid unexpected errors downstream.

You might consider adding a check to continue the loop if no image is found:

 for collection, fields in satellite_collections.items():
     image = ee.ImageCollection(collection) \
         .filterDate(current_time.strftime('%Y-%m-%d')) \
         .filterBounds(aoi) \
         .first()

+    if not image:
+        print(f"No images found for collection {collection} on {current_time.strftime('%Y-%m-%d')}")
+        continue
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for collection, fields in satellite_collections.items():
image = ee.ImageCollection(collection) \
.filterDate(current_time.strftime('%Y-%m-%d')) \
.filterBounds(aoi) \
.first()
for collection, fields in satellite_collections.items():
image = ee.ImageCollection(collection) \
.filterDate(current_time.strftime('%Y-%m-%d')) \
.filterBounds(aoi) \
.first()
if not image:
print(f"No images found for collection {collection} on {current_time.strftime('%Y-%m-%d')}")
continue

if image:
values = image.reduceRegion(
reducer=ee.Reducer.mean(),
geometry=aoi,
scale=1113.2
).getInfo()

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Handle potential exceptions during data retrieval with 'getInfo()'

The getInfo() call communicates with the Earth Engine servers and may raise exceptions due to network issues or server errors. To enhance robustness, consider wrapping this call in a try-except block.

Here's how you might implement this:

 if image:
     try:
         values = image.reduceRegion(
             reducer=ee.Reducer.mean(),
             geometry=aoi,
             scale=1113.2
         ).getInfo()
+    except Exception as e:
+        print(f"Error retrieving data from collection {collection}: {e}")
+        continue
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
if image:
values = image.reduceRegion(
reducer=ee.Reducer.mean(),
geometry=aoi,
scale=1113.2
).getInfo()
if image:
try:
values = image.reduceRegion(
reducer=ee.Reducer.mean(),
geometry=aoi,
scale=1113.2
).getInfo()
except Exception as e:
print(f"Error retrieving data from collection {collection}: {e}")
continue

for field in fields:
all_features[f"{collection}_{field}"] = values.get(field)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Handle cases where 'values' may be empty or missing keys

There may be situations where values is empty or doesn't contain all the expected fields. To prevent issues, it's prudent to handle missing keys when populating all_features.

Consider using a default value when accessing fields:

 for field in fields:
-    all_features[f"{collection}_{field}"] = values.get(field)
+    all_features[f"{collection}_{field}"] = values.get(field, None)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
for field in fields:
all_features[f"{collection}_{field}"] = values.get(field)
for field in fields:
all_features[f"{collection}_{field}"] = values.get(field, None)


# Add time-related features
all_features['year'] = current_time.year
all_features['month'] = current_time.month
all_features['day'] = current_time.day
all_features['dayofweek'] = current_time.weekday()
all_features['week'] = int(current_time.strftime('%V'))

return all_features
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Enhance code clarity with detailed docstrings and type annotations

Adding comprehensive docstrings and precise type hints improves code readability and assists other developers in understanding the code's purpose and usage.

Here's how you can augment the method:

 @staticmethod
-def extract_single_point_data(longitude: float, latitude: float) -> Dict[str, float]:
+def extract_single_point_data(longitude: float, latitude: float) -> Dict[str, Optional[float]]:
+    """
+    Extract satellite data features for a given geographic point.
+
+    Parameters:
+        longitude (float): The longitude of the point of interest.
+        latitude (float): The latitude of the point of interest.
+
+    Returns:
+        Dict[str, Optional[float]]: A dictionary containing satellite data features and time-related features.
+    """
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
@staticmethod
def extract_single_point_data(longitude: float, latitude: float) -> Dict[str, float]:
aoi = ee.Geometry.Point([longitude, latitude])
current_time = datetime.now(timezone.utc)
all_features = {}
for collection, fields in satellite_collections.items():
image = ee.ImageCollection(collection) \
.filterDate(current_time.strftime('%Y-%m-%d')) \
.filterBounds(aoi) \
.first()
if image:
values = image.reduceRegion(
reducer=ee.Reducer.mean(),
geometry=aoi,
scale=1113.2
).getInfo()
for field in fields:
all_features[f"{collection}_{field}"] = values.get(field)
# Add time-related features
all_features['year'] = current_time.year
all_features['month'] = current_time.month
all_features['day'] = current_time.day
all_features['dayofweek'] = current_time.weekday()
all_features['week'] = int(current_time.strftime('%V'))
return all_features
@staticmethod
def extract_single_point_data(longitude: float, latitude: float) -> Dict[str, Optional[float]]:
"""
Extract satellite data features for a given geographic point.
Parameters:
longitude (float): The longitude of the point of interest.
latitude (float): The latitude of the point of interest.
Returns:
Dict[str, Optional[float]]: A dictionary containing satellite data features and time-related features.
"""
aoi = ee.Geometry.Point([longitude, latitude])
current_time = datetime.now(timezone.utc)
all_features = {}
for collection, fields in satellite_collections.items():
image = ee.ImageCollection(collection) \
.filterDate(current_time.strftime('%Y-%m-%d')) \
.filterBounds(aoi) \
.first()
if image:
values = image.reduceRegion(
reducer=ee.Reducer.mean(),
geometry=aoi,
scale=1113.2
).getInfo()
for field in fields:
all_features[f"{collection}_{field}"] = values.get(field)
# Add time-related features
all_features['year'] = current_time.year
all_features['month'] = current_time.month
all_features['day'] = current_time.day
all_features['dayofweek'] = current_time.weekday()
all_features['week'] = int(current_time.strftime('%V'))
return all_features

45 changes: 45 additions & 0 deletions src/spatial/views/satellite_predictions.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
from datetime import datetime, timezone

import gcsfs
import joblib
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Remove Unused Imports: gcsfs and joblib

It appears that gcsfs and joblib are imported but not used in this file. Removing unused imports helps keep the code clean and maintainable.

Apply this diff to remove the unused imports:

- import gcsfs
- import joblib
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import gcsfs
import joblib
🧰 Tools
🪛 Ruff

3-3: gcsfs imported but unused

Remove unused import: gcsfs

(F401)

4-4: joblib imported but unused

Remove unused import: joblib

(F401)

import numpy as np
from flask import request, jsonify

from configure import get_trained_model_from_gcs
from models.SatellitePredictionModel import SatellitePredictionModel


class SatellitePredictionView:
@staticmethod


@staticmethod
def make_predictions():
try:
data = request.json
latitude = data.get('latitude')
longitude = data.get('longitude')

if not latitude or not longitude:
return jsonify({'error': 'Latitude and longitude are required'}), 400

SatellitePredictionModel.initialize_earth_engine()

features = SatellitePredictionModel.extract_single_point_data(longitude, latitude)

feature_array = np.array(list(features.values())).reshape(1, -1)

prediction = get_trained_model_from_gcs(
project_name, bucket_name, f"satellite_model.pkl"
).predict(feature_array)[0]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue

Undefined Variables: project_name and bucket_name

The variables project_name and bucket_name used in get_trained_model_from_gcs are not defined within the scope of this function. Ensure that these variables are properly defined or passed as parameters to avoid NameError exceptions.

Would you like assistance in defining these variables or retrieving them from your configuration?

🧰 Tools
🪛 Ruff

33-33: Undefined name project_name

(F821)

33-33: Undefined name bucket_name

(F821)

33-33: f-string without any placeholders

Remove extraneous f prefix

(F541)


return jsonify({
'pm2_5_prediction': float(prediction),
'latitude': latitude,
'longitude': longitude,
'timestamp': datetime.now(timezone.utc).isoformat()
})

except Exception as e:
return jsonify({'error': str(e)}), 500

Check warning

Code scanning / CodeQL

Information exposure through an exception Medium

Stack trace information
Loading
flows to this location and may be exposed to an external user.

Copilot Autofix AI 4 months ago

To fix the problem, we need to ensure that detailed error information is not exposed to the end user. Instead, we should log the error details on the server and return a generic error message to the user. This can be achieved by using Python's logging module to log the exception details and then returning a generic error message in the response.

Suggested changeset 1
src/spatial/views/satellite_predictions.py

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/src/spatial/views/satellite_predictions.py b/src/spatial/views/satellite_predictions.py
--- a/src/spatial/views/satellite_predictions.py
+++ b/src/spatial/views/satellite_predictions.py
@@ -4,2 +4,3 @@
 from flask import request, jsonify
+import logging
 
@@ -9,2 +10,4 @@
 
+logging.basicConfig(level=logging.ERROR)
+
 class SatellitePredictionView:
@@ -50,3 +53,4 @@
         except Exception as e:
-            return jsonify({'error': str(e)}), 500
+            app.logger.error('An error occurred: %s', str(e))
+            return jsonify({'error': 'An internal error has occurred!'}), 500
 
EOF
@@ -4,2 +4,3 @@
from flask import request, jsonify
import logging

@@ -9,2 +10,4 @@

logging.basicConfig(level=logging.ERROR)

class SatellitePredictionView:
@@ -50,3 +53,4 @@
except Exception as e:
return jsonify({'error': str(e)}), 500
app.logger.error('An error occurred: %s', str(e))
return jsonify({'error': 'An internal error has occurred!'}), 500

Copilot is powered by AI and may make mistakes. Always verify output.
Positive Feedback
Negative Feedback

Provide additional feedback

Please help us improve GitHub Copilot by sharing more details about this comment.

Please select one or more of the options
Mnoble-19 marked this conversation as resolved.
Show resolved Hide resolved

Loading