Steps of finding out and deleting missing ledgers on the Pulsar side

[hangc0276]

1. Delete unrecoverable ledgers in under replicated state in the BookKeeper cluster

1.1 list all unrecoverable ledgers without deleting them

bin/sn-pulsar-tools bookie delete-underreplicate-ledgers --conf ./bookkeeper.conf --dry-run

Please confirm the results are expected.

1.2 list and delete all unrecoverable ledgers

bin/sn-pulsar-tools bookie delete-underreplicate-ledgers --conf ./bookkeeper.conf 

2. Find out ledgers that existed on Pulsar managedLedger list, but not existed in the BookKeeper cluster.

root@pulsar-toolset-0:/pulsar/sn-pulsar-tools/bin# ./sn-pulsar-tools pck bookie detect-missing-ledger --zookeeper pulsar-zookeeper:2181 --service-url http://pulsar-broker:8080 -n public/default --auth-plugin org.apache.pulsar.client.impl.auth.AuthenticationToken --auth-params file:///pulsar/tokens/client/token

3. Delete missing ledgers from Pulsar managedLedger list (Run it in the toolset container).

3.1 Replace the generated python Protobuf file MLDataFormats_pb2.py with the given one. (This python file is generated from Pulsar 2.9.3.23, which matched the customer's Pulsar version. The new python file list is below)

3.2 From step 2's output, we can the missing ledgerId with the topic name. For example, the missing ledgerId is [2,3] and the related topic name is persistent://public/default/test_v1-partition-0
Use the following command to get the ledger list of the topic and confirm the list containers missing ledgerIds

bin/pulsar-managed-ledger-admin print-managed-ledger -zk ip:port --managedLedgerPath public/default/persistent/test_v1-partition-0

After confirmingd, we can use the following command to delete ledgerIds [2, 3] from the topic's ledger list.

bin/pulsar-managed-ledger-admin delete-managed-ledger-ids -zk ip:port --ledgerIds 2,3 --managedLedgerPath public/default/persistent/test_v1-partition-0

Use the following command to confirm the ledgerIds has been deletd.

bin/pulsar-managed-ledger-admin print-managed-ledger -zk ip:port --managedLedgerPath public/default/persistent/test_v1-partition-0

Notice:

1. Please test the above command with one topic first to confirm it works as expected.
2. Please pay attention to the parameter --managedLedgerPath. This parameter represents the ZNode path in Zookeeper, we can find it in Zookeeper's Znode. If the topic name is persistent://public/default/test_v1-partition-0 and the parameter is public/default/persistent/test_v1-partition-0

Ricks:
This delete command directly updates ZNode. However, the broker maybe also update the ZNode at the same time. It has the risk of this tool updating the ZNode with the old content and may lead to data loss. It may happens with low probability.

[hangc0276]

MLDataFormats_pb2.py.tar.gz

[github-actions]

The issue had no activity for 30 days, mark with Stale label.