Recovering from exceeded quota requires defrag - should it?

[ee-github]

Bug report criteria

• This bug report is not security related, security issues should be disclosed privately via [email protected].
• This is not a support request, support requests should be raised in the etcd discussion forums.
• You have read the etcd bug reporting guidelines.
• Existing open issues along with etcd frequently asked questions have been checked and this is not a duplicate.

What happened?

Hello. I was testing breaching quota limit and recovery in etcd 3.5.9 3-node cluster. It seems to recover a compact and defrag are both needed. Though the documentation states "internal fragmentation is space that is free to use by the backend", which suggests that just compacting should be required, if fragmented space is 'free to use'. So is the behavior incorrect, or the documentation, or my understanding of it (very possible)?

What did you expect to happen?

I expected to only have delete some keys, compact and disarm alarms to get back to a read-write state, NOT defrag.

How can we reproduce it (as minimally and precisely as possible)?

My steps:

1. Exceeded quota (writing via REST API until HTTP error 429 received)
2. Confirm alarm NOSPACE is set on all members via 'endpoint status --cluster'
3. Delete many (all) keys
4. Compact to current revision
5. Clear alarm using 'alarm disarm'
6. Confirm NO alarms are set via 'endpoint status --cluster'
7. Write a key, which fails with 'mvcc: database space exceeded'
8. Confirm NOSPACE alarm is set again via 'endpoint status --cluster'

If I insert a defrag step between 4 and 5 this problem doesn't happen.

Anything else we need to know?

Test machine is not easily accessible to paste exact log outputs, but I can jump through some hoops if it becomes really necessary to make it happen.

Etcd version (please run commands below)

Test machine is not easily accessible to paste exact log outputs, but I can jump through some hoops if it becomes really necessary to make it happen.

Etcd configuration (command line flags or environment variables)

Test machine is not easily accessible to paste exact log outputs, but I can jump through some hoops if it becomes really necessary to make it happen.

Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)

Test machine is not easily accessible to paste exact log outputs, but I can jump through some hoops if it becomes really necessary to make it happen.

Relevant log output

No response

[jmhbnz]

Hey @ee-github - My understanding is that recovering from an exceeded space quote should require a defragment.

After compacting the keyspace, the backend database may exhibit internal fragmentation. Any internal fragmentation is space that is free to use by the backend but still consumes storage space. Compacting old revisions internally fragments etcd by leaving gaps in backend database. Fragmented space is available for use by etcd but unavailable to the host filesystem. In other words, deleting application data does not reclaim the space on disk.
https://etcd.io/docs/v3.5/op-guide/maintenance/#defragmentation

The key fragments of that paragraph to me are "but still consumes storage space" and "Fragmented space is available for use by etcd but unavailable to the host filesystem"

So if we read a bit further down we highlight that defragmenting is mandatory in relation to recovering from exceeding a storage space quota.

The space quota in etcd ensures the cluster operates in a reliable fashion. Without a space quota, etcd may suffer from poor performance if the keyspace grows excessively large, or it may simply run out of storage space, leading to unpredictable cluster behavior. If the keyspace’s backend database for any member exceeds the space quota, etcd raises a cluster-wide alarm that puts the cluster into a maintenance mode which only accepts key reads and deletes. Only after freeing enough space in the keyspace and defragmenting the backend database, along with clearing the space quota alarm can the cluster resume normal operation.
https://etcd.io/docs/v3.5/op-guide/maintenance/#space-quota

[ee-github]

Thank you confirming. That certainly lines up with my observations/testing.

I'm left with one doubt then: Does this mean that EVERY instance of etcd MUST employ a regular defrag schedule? Because as I understand it, key/value history always grows, and compacting/auto-compacting is not sufficient to avoid a NOSPACE condition eventually?

[jmhbnz]

Yes it is explicitly suggested as part of our operations maintenance guide linked above to monitor the status of etcd storage fragmentation and run defragment as required. Many users do this in response to certain monitoring thresholds based on etcd metrics, or simply on a cronjob schedule.

Refer to https://github.com/ahrtr/etcd-defrag to assist with operationalising the defrag process.

[ee-github]

Thank you once again. I've seen various discussion here about safety concerns regarding defragmentation, including a link to that tool that aims to implement it as safely as possible. That said it seems like the consensus is still to schedule it outside of critical business hours.

With that in mind, and how etcd uses storage, it may not work for my non-kubernetes workload. I'll gladly take any opinion here, even if subjective.

In my use case as a generic k-v store, I estimate needing about 1000 keys, each of about 200KB in size. So a total of 200MB of data. I need to refresh the value of each key once per minute. So in a 10 hour (600 refreshes) workday that would be 120GB of data stored on disk if I compact/defragment after hours. For safety, it would be prudent to allocate double, or 240GB. 240GB of disk (even 120GB) to ultimately just store 200MB of data is obviously not feasible.

Unless I'm missing something. It doesn't seem like there is a way to disable history retention (which I don't need).

Any input appreciated.

[tjungblu]

@ee-github you might also want to check the auto compaction feature:

  --auto-compaction-retention '0'
    Auto compaction retention length. 0 means disable auto compaction.
  --auto-compaction-mode 'periodic'
    Interpret 'auto-compaction-retention' one of: periodic|revision. 'periodic' for duration based retention, defaulting to hours if no time unit is provided (e.g. '5m'). 'revision' for revision number based retention.

Defrag of 200mb should not take more than a few milliseconds on a modern nvme drive. I highly doubt it would cause severe availability issues for a once-a-minute-write workload.

[ee-github]

Hi @tjungblu. Thanks for the pointer.

I did initially review the auto-compaction settings so I was aware of them going into this testing. However, what threw me off was that it seemed like compaction did NOT give back space for writing new key values. The reason is that in my testing, after compaction the this metric doesn't decrease (so I though defrag was also necessary):

etcd_mvcc_db_total_size_in_use_in_bytes

However, as soon as I write the next key (after compaction) the above metric does drop to the expected value. Not sure if that is a bug (obviously cosmetic if so - I'm guessing not many users have long idling instances outside of testing).

So as auto-compaction does seem to be running every 5m as advertised, my storage should only swell to 5m worth of 'extra' writes, which is a totally reasonable disk allocation, and I have a way forward.

(Understood that recovery of an exceeded quota still requires defrag, which is fine)

In short thank you and @jmhbnz for taking the time to respond. Your responses spurred more testing and a solution.