Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move to production #3785

Merged
merged 10 commits into from
Oct 28, 2024
Merged

move to production #3785

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
a3ba14c
Update AirQo exceedance production image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
742a81e
Update KCCA exceedance production image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
f1ad5e2
Update workflows staging image tag to stage-f4c5a75e-1730099832
github-actions[bot] Oct 28, 2024
5af9e20
Update device registry production image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
ebdde1d
Update workflows prod image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
e095ac4
Update analytics production image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
fb58fdb
Update predict production image tag to prod-cf1605be-1730099912
github-actions[bot] Oct 28, 2024
669c536
Add settings for consumer to optimize connection
NicholasTurner23 Oct 28, 2024
9063106
Merge branch 'staging' into update/Kafka-implementations
NicholasTurner23 Oct 28, 2024
0957541
Merge pull request #3784 from NicholasTurner23/update/Kafka-implement…
Baalmart Oct 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion k8s/analytics/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ images:
celeryWorker: eu.gcr.io/airqo-250220/airqo-analytics-celery-worker
reportJob: eu.gcr.io/airqo-250220/airqo-analytics-report-job
devicesSummaryJob: eu.gcr.io/airqo-250220/airqo-analytics-devices-summary-job
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
api:
name: airqo-analytics-api
label: analytics-api
Expand Down
2 changes: 1 addition & 1 deletion k8s/device-registry/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-device-registry-api
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-airqo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/airqo-exceedance-job
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-kcca.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/kcca-exceedance-job
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/predict/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ images:
predictJob: eu.gcr.io/airqo-250220/airqo-predict-job
trainJob: eu.gcr.io/airqo-250220/airqo-train-job
predictPlaces: eu.gcr.io/airqo-250220/airqo-predict-places-air-quality
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
api:
name: airqo-prediction-api
label: prediction-api
Expand Down
2 changes: 1 addition & 1 deletion k8s/workflows/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ images:
initContainer: eu.gcr.io/airqo-250220/airqo-workflows-xcom
redisContainer: eu.gcr.io/airqo-250220/airqo-redis
containers: eu.gcr.io/airqo-250220/airqo-workflows
tag: prod-bbf47c3d-1729854005
tag: prod-cf1605be-1730099912
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/workflows/values-stage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ images:
initContainer: eu.gcr.io/airqo-250220/airqo-stage-workflows-xcom
redisContainer: eu.gcr.io/airqo-250220/airqo-stage-redis
containers: eu.gcr.io/airqo-250220/airqo-stage-workflows
tag: stage-b3d5eb96-1729853938
tag: stage-f4c5a75e-1730099832
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
4 changes: 2 additions & 2 deletions src/workflows/airqo_etl_utils/airqo_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1058,8 +1058,8 @@ def get_devices(group_id: str) -> pd.DataFrame:
except json.JSONDecodeError as e:
logger.exception(f"Error decoding JSON: {e}")
continue
if not key or not value.get("device_id"):
logger.info(
if not key or not value.get("_id"):
logger.warning(
f"Skipping message with key: {key}, missing 'device_id'."
)
continue
Expand Down
2 changes: 2 additions & 0 deletions src/workflows/airqo_etl_utils/message_broker_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -232,6 +232,8 @@ def consume_from_topic(
"auto.offset.reset": auto_offset_reset,
"enable.auto.commit": "true" if auto_commit else "false",
"fetch.message.max.bytes": 2 * 1024 * 1024,
"api.version.request.timeout.ms": 15000,
"socket.timeout.ms": 30000,
}
)
consumer = Consumer(consumer_config)
Expand Down
Loading