From d396b5e7af31f85cabd4e2ff0dde16dea1e23d3f Mon Sep 17 00:00:00 2001 From: Ramachandran A G <106139410+ag-ramachandran@users.noreply.github.com> Date: Wed, 1 Jan 2025 10:55:01 +0530 Subject: [PATCH] Update docs/KustoSink.md Co-authored-by: ohad bitton <32278684+ohadbitt@users.noreply.github.com> --- docs/KustoSink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/KustoSink.md b/docs/KustoSink.md index 917ac295..05847307 100644 --- a/docs/KustoSink.md +++ b/docs/KustoSink.md @@ -78,7 +78,7 @@ All the options that can be used in the Kusto Sink can be found in KustoSinkOpti **'KustoStreaming'** mode - uses [stream ingestion](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Cjava) to load data into Kusto. Streaming ingestion is useful for loading data when you need low latency between ingestion and query. As ADX Streaming ingestion has a [data size limit](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming) of 4 MB, for each partition over a batched rdd the connector will ingest **4MB** chunks of data, ingesting each individually. For each such batch - The connector will try 3 times to stream the data and if fails it will fallback to uploading it to blob storage and queue the ingestion. - It is therefore recommended to configure the rate of Spark stream to produce around 10MB of data per batch and avoid using Stream. + It is therefore recommended to configure the rate of Spark stream to produce around 10MB of data per batch and avoid using KustoStreaming. It is also recommended to tune the target table [ingestion batching policy](https://learn.microsoft.com/azure/data-explorer/kusto/management/batching-policy) as this will effect the fallback flow latency. > **Note** - [Streaming ingestion policy](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/streamingingestionpolicy) must be enabled on the destination table or database and enabled on cluster configuration (see [documentation](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Cjava) for details). Streaming ingestion has certain [performance and operational considerations](https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Ccsharp#performance-and-operational-considerations) please validate that the scenario for KustoStreaming mode is inline with these limits.