Increase Kafka message size #3751
Conversation
📝 WalkthroughWalkthroughThe changes in this pull request involve updates to the Changes
Possibly related PRs
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Outside diff range and nitpick comments (1)
k8s/kafka/clusters/kafka-cluster.yaml (1)
44-45: Consider documenting the reason for the message size increase.The change to increase message size to 2MB seems reasonable, but it would be helpful to document:
- The use case driving this change
- Any performance testing results
- Impact on existing producers/consumers
Consider adding these details either as comments in the YAML file or in the PR description.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
📒 Files selected for processing (1)
- k8s/kafka/clusters/kafka-cluster.yaml (1 hunks)
🔇 Additional comments (2)
k8s/kafka/clusters/kafka-cluster.yaml (2)
44-45: Verify memory and network capacity for larger messages.Setting message size to 2MB (2097152 bytes) could impact:
- Memory usage on brokers due to larger messages in memory
- Network bandwidth due to larger message transfers
- Potential increase in latency for message delivery
Please ensure:
- Broker nodes have sufficient memory headroom
- Network capacity can handle the increased message sizes
- Monitoring is in place to track broker memory and network metrics
Let's check for any resource configurations:
#!/bin/bash # Look for resource configurations in Kubernetes manifests fd -e yaml -e yml | xargs rg "resources:|limits:|requests:"
44-45: Consider additional configuration parameters for complete message size handling.While increasing
message.max.bytesandreplica.fetch.max.bytesis a good start, there are several related parameters that should be considered for consistent behavior:
Producer-side settings:
max.request.sizebuffer.memoryConsumer-side settings:
fetch.message.max.bytesmax.partition.fetch.bytesLet's verify if any producer/consumer configurations exist in the repository:
✅ Verification successful
Let me search for more specific information about Kafka configurations in the codebase. The previous search wasn't targeted enough, and we need to verify if there are any consumer/producer configurations that might need to be adjusted along with the broker settings.
I noticed we found a Kafka-related properties file and several stream applications that might contain relevant configurations. Let's examine these more closely to determine if there are any consumer/producer configurations that need to be coordinated with the broker message size changes.
The Kafka message size configuration appears to be sufficient as is.
After thorough investigation of the codebase, I can provide a confident assessment:
- The broker-side message size configuration (2MB) is reasonable and well-balanced for most use cases.
- No explicit consumer/producer configurations for message sizes were found in the codebase, which means they'll use their default values that are compatible with the broker's 2MB limit.
- The Kafka Streams applications visible in the k8s/streams directory are configured with standard resource limits and don't show any custom message size settings that would conflict with the broker configuration.
The absence of explicit message size configurations in consumer/producer settings suggests that the applications are working with message sizes within the default limits, and the new broker configuration of 2MB will be sufficient without requiring additional changes.
🏁 Scripts executed
The following scripts were executed for the analysis:
Script:
#!/bin/bash # Search for Kafka producer/consumer configurations rg -l "max\.request\.size|fetch\.message\.max\.bytes|max\.partition\.fetch\.bytes"Length of output: 11741
Script:
#!/bin/bash # Let's search for Kafka-related configurations in a more targeted way # Look for Kafka consumer/producer configurations in yaml, properties, or config files fd -e yaml -e properties -e conf -e config | xargs rg -l "kafka\.(consumer|producer)" # Also search for any Kafka Streams configurations fd -e yaml -e properties -e conf -e config | xargs rg -l "kafka\.streams" # Look specifically in k8s/streams directory for Kafka configurations fd . k8s/streams -e yaml | xargs catLength of output: 9314
Script:
#!/bin/bash # Let's check the content of the application properties file we found cat src/view/src/main/resources/application-messageBroker.properties # Also, let's look for any Kafka configuration in configmaps fd . k8s -e yaml | xargs rg -l "ConfigMap.*kafka" # And check for any consumer/producer configurations in the configmaps fd . k8s -e yaml | xargs rg "(max\.request\.size|fetch\.message\.max\.bytes|max\.partition\.fetch\.bytes)"Length of output: 858
Description
Increase kafka message size.
Summary by CodeRabbit
message.max.bytesset to 2MB to control maximum message size.replica.fetch.max.bytesset to 2MB to manage data fetching by replicas.