Fix issue in VolumeReplace feature that may block replacement when combined with an annotation change

[anish-db]

What problem does this PR solve?

Fix an issue with Volume Replacement feature that can cause it to be blocked when user requests a metadata / annotation change, clubbed with a disk replacement.

What is changed and how does it work?

Remove check in pvc_replacer that checks if sts is upgrading before recreating sts.

If a disk change is clubbed with pod spec changes (but not metadata changes), it gets reverted here:

tidb-operator/pkg/manager/member/tikv_member_manager.go

Line 276 in 6de4871

newSet.Spec.Template.Spec = *podSpec

So the statefulset would not show any changes (and would not fail upgrade check, so issue doesn't happen).

However metadata changes like (annotation change), do not get reverted, and will be applied to the sts, but since volume replace marks the update strategy to be OnDelete :

tidb-operator/pkg/manager/member/tikv_member_manager.go

Line 691 in 6de4871

updateStrategy.Type = apps.OnDeleteStatefulSetStrategyType

The annotation won't be applied, and will be stuck.

Further pvc_replacer will be stuck forever on this check (which is removed in this PR)

The check itself is unnecessary since, before the code reaches here we are already in volume replace mode, which means sts should have OnDelete strategy and sts updates will only be performed by pvc_replacer clubbed with disk replace.

Code changes

• Has Go code change
• Has CI related scripts change

Tests

• Unit test
• E2E test
• Manual test
• No code

Side effects

• Breaking backward compatibility
• Other side effects:

Related changes

• Need to cherry-pick to the release branch
• Need to update the documentation

Release Notes

Please refer to Release Notes Language Style Guide before writing the release note.

Fix an issue where VolumeReplacement feature can get stuck when a disk change is clubbed with a metadata(annotation) change

[codecov-commenter]

Codecov Report

Merging #5382 (6eeead0) into master (6f07084) will decrease coverage by 0.03%.
The diff coverage is n/a.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5382      +/-   ##
==========================================
- Coverage   61.61%   61.58%   -0.03%     
==========================================
  Files         228      228              
  Lines       28866    28864       -2     
==========================================
- Hits        17785    17776       -9     
- Misses       9339     9344       +5     
- Partials     1742     1744       +2     

Flag	Coverage Δ
unittest	61.58% <ø> (-0.03%)	⬇️

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: csuzhangxc

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [csuzhangxc]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2023-11-10 08:52:51.817522485 +0000 UTC m=+3807169.404632630: ☑️ agreed by csuzhangxc.

[csuzhangxc]

/cherry-pick release-1.5

[ti-chi-bot]

@csuzhangxc: once the present PR merges, I will cherry-pick it on top of release-1.5 in the new PR and assign it to you.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

@csuzhangxc: new pull request created to branch release-1.5: #5383.

In response to this:

/cherry-pick release-1.5

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.