Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

move to production #3731

Merged
merged 11 commits into from
Oct 22, 2024
Merged

move to production #3731

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
2346f24
Update auth service staging image tag to stage-6631dc02-1729615305
github-actions[bot] Oct 22, 2024
cc9a5ba
Update AirQo exceedance production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
8e99b43
Update KCCA exceedance production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
867df4e
Update device registry production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
c14fff6
Update auth service production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
67e09e5
Clean up
NicholasTurner23 Oct 22, 2024
6757a1d
Update analytics production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
e33d351
Update workflows prod image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
8595bb8
Update spatial production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
69fbbcc
Update predict production image tag to prod-65aa1824-1729615373
github-actions[bot] Oct 22, 2024
143b27e
Merge pull request #3730 from NicholasTurner23/update/Kafka-implement…
Baalmart Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion k8s/analytics/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ images:
celeryWorker: eu.gcr.io/airqo-250220/airqo-analytics-celery-worker
reportJob: eu.gcr.io/airqo-250220/airqo-analytics-report-job
devicesSummaryJob: eu.gcr.io/airqo-250220/airqo-analytics-devices-summary-job
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
api:
name: airqo-analytics-api
label: analytics-api
Expand Down
2 changes: 1 addition & 1 deletion k8s/auth-service/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-auth-api
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/auth-service/values-stage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 2
image:
repository: eu.gcr.io/airqo-250220/airqo-stage-auth-api
tag: stage-b55d4116-1729602971
tag: stage-6631dc02-1729615305
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/device-registry/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-device-registry-api
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-airqo.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/airqo-exceedance-job
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/exceedance/values-prod-kcca.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ app:
configmap: env-exceedance-production
image:
repository: eu.gcr.io/airqo-250220/kcca-exceedance-job
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
2 changes: 1 addition & 1 deletion k8s/predict/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ images:
predictJob: eu.gcr.io/airqo-250220/airqo-predict-job
trainJob: eu.gcr.io/airqo-250220/airqo-train-job
predictPlaces: eu.gcr.io/airqo-250220/airqo-predict-places-air-quality
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
api:
name: airqo-prediction-api
label: prediction-api
Expand Down
2 changes: 1 addition & 1 deletion k8s/spatial/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ app:
replicaCount: 3
image:
repository: eu.gcr.io/airqo-250220/airqo-spatial-api
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
2 changes: 1 addition & 1 deletion k8s/workflows/values-prod.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ images:
initContainer: eu.gcr.io/airqo-250220/airqo-workflows-xcom
redisContainer: eu.gcr.io/airqo-250220/airqo-redis
containers: eu.gcr.io/airqo-250220/airqo-workflows
tag: prod-d58d02be-1729614498
tag: prod-65aa1824-1729615373
nameOverride: ''
fullnameOverride: ''
podAnnotations: {}
Expand Down
5 changes: 3 additions & 2 deletions src/workflows/airqo_etl_utils/message_broker_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -336,9 +336,9 @@ def publish_to_topic(
logger.info(f"Preparing to publish data to topic: {topic}")
data.replace(np.nan, None, inplace=True)
dataframe_list = data.to_dict("records")
message_counts = 0
if column_key:
logger.info(f"Using '{column_key}' as the key for messages")
message_counts = 0
for row in dataframe_list:
key = row.pop(column_key, None)
if key is None:
Expand All @@ -351,14 +351,15 @@ def publish_to_topic(
selected_partition = (
None if auto_partition else self.__get_least_loaded_partition()
)
message_counts += 1
self._send_message(producer, topic, key, message, selected_partition)
if not auto_partition:
self.partition_loads[selected_partition] += 1
message_counts += 1

else:
logger.info("No key provided, splitting data into chunks and publishing")
for chunk_data in self._generate_chunks(dataframe_list):
message_counts += len(chunk_data)
message = json.dumps({"data": chunk_data}).encode("utf-8")

selected_partition = (
Expand Down
Loading