Information needed on ETCD

[ankitg-03]

Hi @ahrtr
I have a few queries regarding the ETCD. Could you please help me with the below queries:

If ETCD is experiencing high PVC usage, but I don't want to increase the PVC size, rather I believe we can delete the .wal and .snap files that are occupying space which exceeds the configured PVC value.
So,

1. Can we delete .wal and .snap files from ETCD deployment (we have snap and wal files present at - /data/member/snap and /data/member/wal respectively)?
2. If "YES", is there any impact of deleting .wal and .snap files or is there any kind of risk?
3. How many .wal and .snap files can be deleted ?

[chaochn47]

It's generally not recommended to delete .wal and .snap files manually. You can decrease the number of such files etcd retains by lowing --max-wals and max-snapshots etcd command line arguments.

However, I don't think above actions can mitigate high PVC usage (I guess it meant disk usage) because each WAL file is just 64MiB and snapfile is a magnitude of hundreds of KiB. I would suggest look into if disk capacity is big enough to for etcd db growth, specifically file /data/member/snap/db, which could be as large as a magnitude of GiBs.

[ahrtr]

On top of @chaochn47 's comment, I suggest to read https://etcd.io/blog/2023/how_to_debug_large_db_size_issue/ and consider to run defragmentation using https://github.com/ahrtr/etcd-defrag/ when needed.

[ankitg-03]

Hi @ahrtr & @chaochn47
Thanks for the support.
Could you please tell how much time does it take for wal files to purge automatically to meet the configured value once we reduce the value of ETCD_MAX_WAL variable?

In my case I was having 2 wal files and ETCD_MAX_WALS is configured as 5, I configured the ETCD_MAX_WALS = 1, so ideally there should have been 1 wal file remaining and the older one should have been purged. But after configuring variable to 1 I still see 2 wal files.

[ahrtr]

A WAL file will only be removed when both the following two conditions are true:

• It exceeds the --max-wals or ETCD_MAX_WALS;
• The WAL file has been released due to snapshot creation. FYI.

  etcd/server/etcdserver/server.go

  Line 2089 in 0efa1c1

  if err = s.r.storage.Release(snap); err != nil {

[ankitg-03]

How can I hit the 2nd condition manually, can you also please elaborate a little more on that?
@ahrtr

[chaochn47]

You can try increasing the frequency of snapshot creation by lowering snapshot-count.

etcd/server/etcdserver/server.go

Lines 1156 to 1158 in 0efa1c1

	func (s *EtcdServer) shouldSnapshot(ep *etcdProgress) bool {
	return (s.forceSnapshot && ep.appliedi != ep.snapi) || (ep.appliedi-ep.snapi > s.Cfg.SnapshotCount)
	}

[ankitg-03]

If I make the snapshot count lower, then will it impact deployment in any way either in long or short term ?
@ahrtr @chaochn47

[chaochn47]

It could potentially impact cluster availability if one of the followers is slow to catch up with leader due to bursty traffic and slow network. So it's kind of depending on your environment setup and workload pattern.

etcd/server/embed/config.go

Lines 154 to 161 in a7344da

	SnapshotCount uint64 `json:"snapshot-count"`
	// SnapshotCatchUpEntries is the number of entries for a slow follower
	// to catch-up after compacting the raft storage entries.
	// We expect the follower has a millisecond level latency with the leader.
	// The max throughput is around 10K. Keep a 5K entries is enough for helping
	// follower to catch up.
	SnapshotCatchUpEntries uint64 `json:"experimental-snapshot-catch-up-entries"`