Shrink_Ceph_OSDs.md

Shrink Ceph OSDs

This procedure describes how to remove an OSD(s) from a Ceph cluster. Once the OSD is removed, the cluster is also rebalanced to account for the changes. Use this procedure to reduce the size of a cluster or to replace hardware.

Prerequisites

This procedure requires administrative privileges.

Procedure

1. Log in as root on the first master node (ncn-m001).

2. Monitor the progress of the OSDs that have been added.

   ceph -s

3. View the status of each OSD and see where they reside.

   ceph osd tree

4. Reweigh the OSD(s) being removed to rebalance the cluster.

   The first two substeps below can be skipped if there is a down drive and OSD.

   1. Change the weight of the OSD being removed to 0.

      The OSD_ID value should be replaced with the ID of the OSD being removed. For example, if the ID is osd.1, the OSD_ID value would be 1 in the command below.

      ceph osd reweight osd.OSD_ID 0

   2. Change the weight in the CRUSH map to 0.

      ceph osd crush reweight osd.OSD_ID 0

   3. Prevent the removed OSD from getting marked up.

      ceph osd set noup

5. Remove the OSD after the reweighing work is complete.

   1. Take down the OSD being removed.

      ceph osd down osd.OSD_ID

   2. Destroy the OSD.

      ceph osd destroy osd.OSD_ID

   3. Remove the OSD authentication key.

      ceph auth rm osd.OSD_ID

   4. Remove the OSD.

      ceph osd rm osd.OSD_ID

   5. Remove the OSD from the CRUSH map.

      ceph osd crush rm osd.OSD_ID

   6. Remove references to the OSDs on the storage node(s) they were located on.

      The following commands must be run on the storage node(s) that held the OSDs being removed.

      umount /var/lib/ceph/osd/ceph-OSD_ID
      rm -rf /var/lib/ceph/osd/ceph-OSD_ID

6. Clear the flags that were set earlier in the procedure.

   ceph osd unset noup

7. Monitor the cluster until the rebalancing is complete.

   ceph -s