Skip to content

Commit

Permalink
Update docs/KustoSink.md
Browse files Browse the repository at this point in the history
Co-authored-by: ohad bitton <[email protected]>
  • Loading branch information
ag-ramachandran and ohadbitt authored Jan 1, 2025
1 parent 92ee97f commit d396b5e
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion docs/KustoSink.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,7 @@ All the options that can be used in the Kusto Sink can be found in KustoSinkOpti
**'KustoStreaming'** mode - uses [stream ingestion](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Cjava)
to load data into Kusto. Streaming ingestion is useful for loading data when you need low latency between ingestion and query.
As ADX Streaming ingestion has a [data size limit](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming) of 4 MB, for each partition over a batched rdd the connector will ingest **4MB** chunks of data, ingesting each individually. For each such batch - The connector will try 3 times to stream the data and if fails it will fallback to uploading it to blob storage and queue the ingestion.
It is therefore recommended to configure the rate of Spark stream to produce around 10MB of data per batch and avoid using Stream.
It is therefore recommended to configure the rate of Spark stream to produce around 10MB of data per batch and avoid using KustoStreaming.
It is also recommended to tune the target table [ingestion batching policy](https://learn.microsoft.com/azure/data-explorer/kusto/management/batching-policy) as this will effect the fallback flow latency.

> **Note** - [Streaming ingestion policy](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/streamingingestionpolicy) must be enabled on the destination table or database and enabled on cluster configuration (see [documentation](https://learn.microsoft.com/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Cjava) for details). Streaming ingestion has certain [performance and operational considerations](https://learn.microsoft.com/en-us/azure/data-explorer/ingest-data-streaming?tabs=azure-portal%2Ccsharp#performance-and-operational-considerations) please validate that the scenario for KustoStreaming mode is inline with these limits.
Expand Down

0 comments on commit d396b5e

Please sign in to comment.