update blocked by cloud-controller-manager

[kellmer]

Hi,

I'm trying to update our okd 4.12.0-0.okd-2023-04-16-041331 cluster to version 4.13.0-0.okd-2023-05-22-052007.
We are in the stable-4 channel. For some reason the cloud-controller-manager operator blocks the update withe the message:
"Message: Cluster operator cloud-controller-manager should not be upgraded between minor versions: "

But I don't see any specific reason why it should not be updated.

`
oc adm upgrade
Cluster version is 4.12.0-0.okd-2023-04-16-041331

Upgradeable=False

Reason: AsExpected
Message: Cluster operator cloud-controller-manager should not be upgraded between minor versions:

Upstream: https://amd64.origin.releases.ci.openshift.org/graph
Channel: stable-4
Updates:

VERSION IMAGE
4.13.0-0.okd-2023-05-22-052007 registry.ci.openshift.org/origin/release@sha256:16b7227d0c0de198d65c4478617e402d7f67ffc0dee312110a9d4c85877e5e4d
`

In the clusteroperator view in the webUI I see only that the operator can not be updated.

Any suggestion how I can solve this problem?

Regards
Florian

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[vrutkovs]

At least ClusterVersion and ClusterOperator status. Most likely it means "current update is not yet completed, so don't start another one without letting this one finish". It could however also means something is broken too

[kellmer]

clusteroperator.txt
clusterversion.txt

I've uploaded clusteroperator and clusterversion as yml, had to rename then wit .txt to attach.

As you will see there is an issue with vSphere and the default datastore. this warning is present and in the past it doesn't blocked us the update

[vrutkovs]

Check CredentialsRequest objects - seems some permissions are lacking and/or need manual intervention

[kellmer]

@vrutkovs thanks for your help, i really apprechiate it

Do you think the vSphere credentials are wrong? We checked that as first step and they are correct. we can deploy new nodes with the credentials.

[kellmer]

We re-checked the permissions and the seems so be right, at least all the vSphere releated credentials.

[peterroth]

I saw that message every time we reached 4.10 and updated further up. The update process didn't stop for me and in the end, it finished successfully, even though on a nearly empty cluster the entire process took 2+ hours.
I don't say it is what I'd like to see during an update, but after a time the cluster came online with the desired version.

[kellmer]

The problem here is that I can't start the update prozess at all. In the WebUI, for example, the Version I would like to update to is just gray an can not be picked.

[peterroth]

Isn't there any failing clusteroperator? Is it a newly deployed cluster or already an updated one? If the later, was there any error during the last update?

[kellmer]

we did a few updates on this cluster. as far as I can tell the previous updates where all successfully.

[bo0ts]

This comes late, but we just ran into issues updating an old cluster with exactly this problem. @kellmer Have you found any solution to this problem?

[khakimbao-herc]

Have you found a solution to this problem? because I am experiencing the exact same thing right now :(

[Frewx]

Do you have a solution for this problem ? I am facing the same issue with the same okd versions. Trying to upgrade from 4.12.0-0.okd-2023-04-16-041331 to 4.13.0-0.okd-2023-05-22-052007. Upgrade is blocked and cannot start.

[Tyrone-W]

What do the logs of cluster-version-operator show? Have you followed https://access.redhat.com/articles/6958394 to acknowledge the removed API's?

[Frewx]

I have ack'ed the APIs. I have 3 OKD clusters with the same configuration same version and all. 2 of them can show the upgrade path but one of them shows it's blocked like I've mentioned in the previous message.

This is the cluster-version-operator logs. All of my clusters logs the same messages.

I0730 14:02:54.001450       1 cvo.go:570] Started syncing cluster version "openshift-cluster-version/version", spec changes, status, and payload (2024-07-30 14:02:54.001433607 +0000 UTC m=+16236.430929812)
I0730 14:02:54.001550       1 cvo.go:602] Desired version from operator is v1.Update{Version:"4.12.0-0.okd-2023-04-16-041331", Image:"quay.io/openshift/okd@sha256:2b3d90157565bb1e227c1cd182154b498c4cf76360d8a57cc5d6d5a4a63794cb", Force:false}
I0730 14:02:54.001591       1 sync_worker.go:486] Update work is equal to current target; no change required

Also I have seen this https://access.redhat.com/solutions/7041078 article that shows solution as "do minor upgrade until 4.12.18". I can not do minor upgrade as well. When I change the channel from stable-4 to stable-4.12 it says I'm already up to date. But of course I'm not up to date.

[khakimbao-herc]

Have you found a solution to this problem? because I am experiencing the exact same thing right now.
change the upgrade channel to stable-4.12 and exactly the same result. i cannot find any minor update after version 4.12.0-0.okd-2023-04-16-041331

[nevidanniu]

for those who has met same problem: you can manually patch status for problem operator via script and continue update:

package main

import (
  "context"
  "fmt"
  v1 "github.com/openshift/api/config/v1"
  config "github.com/openshift/client-go/config/clientset/versioned/typed/config/v1"
  metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
  "k8s.io/client-go/tools/clientcmd"
  "time"
)

func main() {
  kubeconfig, err := clientcmd.BuildConfigFromFlags("", "kubeconfig")
  if err != nil {
    panic(err.Error())
  }
  clientset, err := config.NewForConfig(kubeconfig)
  if err != nil {
    panic(err.Error())
  }
  ctx := context.Background()
  obj, err := clientset.ClusterOperators().Get(ctx, "cloud-controller-manager", metav1.GetOptions{})
  if err != nil {
    panic(err.Error())
  }
  fmt.Println(obj)
  var conds []v1.ClusterOperatorStatusCondition
  for _, cond := range obj.Status.Conditions {

    if cond.Type != "Upgradeable" {
      conds = append(conds, cond)
    } else {
      nc := v1.ClusterOperatorStatusCondition{
        Type:               cond.Type,
        Status:             "True",
        LastTransitionTime: metav1.Time{Time: time.Now()},
        Reason:             cond.Reason,
      }
      conds = append(conds, nc)
    }
  }
  newObj := obj.DeepCopy()
  newObj.Status.Conditions = conds
  fmt.Println(newObj)
  _, err = clientset.ClusterOperators().UpdateStatus(ctx, newObj, metav1.UpdateOptions{})
  if err != nil {
    panic(err.Error())
  }
}