Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

KAFKA-18364 migrating from zk to kraft document #18365

Merged
merged 11 commits into from
Jan 15, 2025
Merged

KAFKA-18364 migrating from zk to kraft document #18365

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
f8c91d2
add password section
m1a2st Dec 31, 2024
024b00b
improvement password encoded section
m1a2st Dec 31, 2024
1b237aa
wip
m1a2st Dec 31, 2024
fa8c93f
add section for broker.id series
m1a2st Dec 31, 2024
a1f4a40
remove ZkConfigs
m1a2st Dec 31, 2024
7fc245c
add a.k.a
m1a2st Jan 1, 2025
fa12729
update the document
m1a2st Jan 8, 2025
adf962b
add metrics section
m1a2st Jan 8, 2025
2c8a9ce
add new ZooKeeperClientMetrics section
m1a2st Jan 11, 2025
03f4c8c
Merge branch 'trunk' into KAFKA-18364
m1a2st Jan 14, 2025
0f078f4
update the header tag
m1a2st Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/upgrade.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,12 @@ <h5><a id="upgrade_400_notable" href="#upgrade_400_notable">Notable changes in 4
taken when it comes to kafka clients that are not part of Apache Kafka, please see
<a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-896%3A+Remove+old+client+protocol+API+versions+in+Kafka+4.0">KIP-896</a> for the details.
</li>
<li>
Before updating your Kafka cluster which build on Zookeeper to Kafka 4.0, you must understand what the updates and changes
are between Zookeeper mode and Kraft mode. See the following page to learn about some of the significant changes in
latest Kafka releases.
<!--#include virtual="zk2kraft.html" -->
</li>
<li>A number of deprecated classes, methods, configurations and tools have been removed.
<ul>
<li><b>Common</b>
Expand Down
159 changes: 159 additions & 0 deletions docs/zk2kraft.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one or more
contributor license agreements. See the NOTICE file distributed with
this work for additional information regarding copyright ownership.
The ASF licenses this file to You under the Apache License, Version 2.0
(the "License"); you may not use this file except in compliance with
the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->

<script><!--#include virtual="js/templateData.js" --></script>

<script id="zk2kraft-template" type="text/x-handlebars-template"></script>

<div class="p-zk2kraft">
<h4 class="anchor-heading">Significant Changes in Kafka 4.0 Release</h4>
<p>The following are some of the updates in Kafka 4.0 release:</p>
<h5 class="anchor-heading">Removal Zookeeper configs</h5>
<ul>
<li>
<p>
The password encoder-related configurations have been removed. These configurations were used in
ZooKeeper mode to define the key and backup key for encrypting sensitive data (e.g., passwords),
specify the algorithm and key generation method for password encryption (e.g., AES, RSA), and control
the key length and encryption strength.
</p>
<ul>
<li><code>password.encoder.secret</code></li>
<li><code>password.encoder.old.secret</code></li>
<li><code>password.encoder.keyfactory.algorithm</code></li>
<li><code>password.encoder.cipher.algorithm</code></li>
<li><code>password.encoder.key.length</code></li>
<li><code>password.encoder.iterations</code></li>
</ul>
<p>
In Kraft mode, Kafka stores sensitive data in records, and the data is not encrypted in Kafka.
</p>
</li>
<li>
<p>
Removed <code>control.plane.listener.name</code>. Kafka relies on ZooKeeper to manage metadata, but some
internal operations (e.g., communication between controllers (a.k.a., broker controller) and brokers)
still require Kafka’s internal control plane for coordination.
</p>
<p>
In KRaft mode, Kafka eliminates its dependency on ZooKeeper, and the control plane functionality is
fully integrated into Kafka itself. The process roles are clearly separated: brokers handle data-related
requests, while the controllers (a.k.a., quorum controller) manages metadata-related requests. The controllers
use the Raft protocol for internal communication, which operates differently from the ZooKeeper model. Use the
following parameters to configure the control plane listener:
</p>
<ul>
<li><code>controller.listener.names</code></li>
<li><code>listeners</code></li>
<li><code>listener.security.protocol.map</code></li>
</ul>
</li>
<li>
<p>
Removed graceful broker shutdowns-related configurations. These configurations were used in ZooKeeper mode
to define the maximum number of retries and the retry backoff time for controlled shutdowns. It can
reduce the risk of unplanned leader changes and data inconsistencies.
</p>
<ul>
<li><code>controlled.shutdown.max.retries</code></li>
<li><code>controlled.shutdown.retry.backoff.ms</code></li>
</ul>
<p>
In KRaft mode, Kafka uses the Raft protocol to manage metadata. The broker shutdown process differs from
ZooKeeper mode as it is managed by the quorum-based controller. The shutdown process is more reliable
and efficient due to automated leader transfers and metadata updates handled by the controller.
</p>
</li>
<li>
<p>
Remove the broker id generation-related configurations. These configurations were used in ZooKeeper mode
to define the broker id, specify the broker id auto generation, and control the broker id generation process.
</p>
<ul>
<li><code>reserved.broker.max.id</code></li>
<li><code>broker.id.generation.enable</code></li>
<li><code>broker.id</code></li>
</ul>
<p>
Kafka use the node id in Kraft mode to identify servers.
</p>
<ul>
<li><code>node.id</code></li>
</ul>
</li>
<li>
<p>
Removed Zookeeper related configurations.
</p>
<ul>
<li><code>zookeeper.connect</code></li>
<li><code>zookeeper.session.timeout.ms</code></li>
<li><code>zookeeper.connection.timeout.ms</code></li>
<li><code>zookeeper.set.acl</code></li>
<li><code>zookeeper.max.in.flight.requests</code></li>
<li><code>zookeeper.ssl.client.enable</code></li>
<li><code>zookeeper.clientCnxnSocket</code></li>
<li><code>zookeeper.ssl.keystore.location</code></li>
<li><code>zookeeper.ssl.keystore.password</code></li>
<li><code>zookeeper.ssl.keystore.type</code></li>
<li><code>zookeeper.ssl.truststore.location</code></li>
<li><code>zookeeper.ssl.truststore.password</code></li>
<li><code>zookeeper.ssl.truststore.type</code></li>
<li><code>zookeeper.ssl.protocol</code></li>
<li><code>zookeeper.ssl.enabled.protocols</code></li>
<li><code>zookeeper.ssl.cipher.suites</code></li>
<li><code>zookeeper.ssl.endpoint.identification.algorithm</code></li>
<li><code>zookeeper.ssl.crl.enable</code></li>
<li><code>zookeeper.ssl.ocsp.enable</code></li>
</ul>
</li>
</ul>
<h5 class="anchor-heading">Removal metrics</h5>
<ul>
<li>
<p>
Remove the following metrics related to ZooKeeper.
<code>ControlPlaneNetworkProcessorAvgIdlePercent</code>
is to monitor the average fraction of time the network processors are idle. The other <code>ControlPlaneExpiredConnectionsKilledCount</code>
is to monitor the total number of connections disconnected, across all processors.
</p>
<ul>
<li><code>ControlPlaneNetworkProcessorAvgIdlePercent</code></li>
<li><code>ControlPlaneExpiredConnectionsKilledCount</code></li>
</ul>
<p>
In Kraft mode, Kafka also provides metrics to monitor the network processors and expired connections.
Use the following metrics to monitor the network processors and expired connections:
</p>
<ul>
<li><code>NetworkProcessorAvgIdlePercent</code></li>
<li><code>ExpiredConnectionsKilledCount</code></li>
</ul>
</li>
<li>
<p>
Remove the metrics which is monitoring the latency in milliseconds for ZooKeeper requests from broker.
</p>
<ul>
<li><code>kafka.server:type=ZooKeeperClientMetrics,name=ZooKeeperRequestLatencyMs</code></li>
</ul>
<p>
In Kraft mode, Zookeeper is not used, so the metrics is removed.
</p>
</li>
</ul>
</div>
Loading