chore(eap): write script to send scrubbed data into a gcs bucket #6698
Conversation
| file_name = f"scrubbed_spans_data_{current_time}.csv.gz" | ||
|
|
||
| query = f""" | ||
| INSERT INTO FUNCTION gcs('https://storage.googleapis.com/{self.gcp_bucket_name}/{file_name}', |
There was a problem hiding this comment.
Dump that into a directory with the date of the dump.
You can probably make that one variable to be called later in the log.
|
|
||
| def execute(self, logger: JobLogger) -> None: | ||
| cluster = get_cluster(StorageSetKey.EVENTS_ANALYTICS_PLATFORM) | ||
| connection = cluster.get_query_connection(ClickhouseClientSettings.QUERY) |
There was a problem hiding this comment.
Let's do one query that does an INSERT INTO <gcs_bucket> (SELECT ... FROM eap_spans_2_local) instead.
And make sure it's running asynchronous so this script can stop with success.
There was a problem hiding this comment.
Let's do one query that does an INSERT INTO <gcs_bucket> (SELECT ... FROM eap_spans_2_local) instead.
How is this different from what I already have?
And make sure it's running asynchronous so this script can stop with success.
I don't think snuba supports running async queries, but I think we can add support for it with https://github.com/mymarilyn/aioch
There was a problem hiding this comment.
Oh indeed, that's what you are doing, I got confused with how you it was laid out. My bad.
As for the async comment, I'm thinking using https://clickhouse.com/docs/en/operations/settings/settings#async_insert with https://clickhouse.com/docs/en/operations/settings/settings#wait_for_async_insert to false if that's possible so we let the client return immediately, the job to complete and can monitor the query via other ways since we know it will take time to complete otherwise.
There was a problem hiding this comment.
I was looking into using those settings, but when I tested them out, they didn't work for inserts into a gcs bucket. It seems like they're mainly intended for inserts into actual clickhouse tables.
❌ 1 Tests Failed:
View the top 1 failed tests by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
No description provided.