Install 4.12.0-0.okd-2023-03-05-022504 manual mode CCO STS bootstrap issues

[apmlima]

Install 4.12.0-0.okd-2023-03-05-022504 manual mode CCO STS bootstrap failed/issues after a new warning I noticed:
openshift-install create manifests --dir t8
level=info msg=Credentials loaded from default AWS environment variables
level=info msg=Consuming Install Config from target directory
level=warning msg=Discarding the Openshift Manifests that was provided in the target directory because its dependencies are dirty and it needs to be regenerated
level=info msg=Manifests created in: t8/manifests and t8/openshift

This isn't occurring in 4.11.
Kind regards.

4.12.0-0.okd-2023-03-05-022504 AWS

I see issues/problems in the bootstrap, sometimes doesn't complete.

failed-units.txt

• release-image-pivot.service loaded failed failed Pivot bootstrap to the OpenShift Release Image

gather.log
Gathering bootstrap containers ... sudo: crictl: command not found cp: cannot stat '/var/log/bootstrap-control-plane/': No such file or directory Gathering bootstrap rpm-ostree info ... Gathering rendered assets... Using /opt/openshift/auth/kubeconfig as KUBECONFIG Using /opt/openshift/auth/kubeconfig as KUBECONFIG Gathering cluster resources ... sudo: oc: command not found ... sudo: oc: command not found Waiting for logs ... sudo: oc: command not found sudo: oc: command not found Gather remote logs Collecting info from 10.35.197.137 scp: Connection closed ssh: connect to host 10.35.197.137 port 22: Connection refused

release-image-pivot.json
... { "timestamp": "2023-03-16T13:26:37Z", "stage": "rebase-to-okd-os-image", "phase": "stage end", "result": "failure", "errorLine": "1 source ./pre-pivot.sh", "errorMessage": "++ caller 1\n+ error_line='1 source ./pre-pivot.sh'\n++ journalctl --unit=release-image-pivot.service --lines=3 --output=cat" }, { "timestamp": "2023-03-16T13:26:37Z", "phase": "service end", "result": "failure", "errorLine": "1 source ./pre-pivot.sh", "errorMessage": "++ caller 1\n+ error_line='1 source ./pre-pivot.sh'\n++ journalctl --unit=release-image-pivot.service --lines=3 --output=cat" }

[vrutkovs]

Please attach log bundle

[apmlima]

No, I even installed helm bin previously using remote network. Everything works fine on latest stable 4.11. I wanted to check 4.12 changed my ci/cd code to get new install.gz client.gz, that I've done many times already since 4.6, and bootstrap giving errors after that new warning on the manifests...

[apmlima]

Tried again with the same results, 3 clusters failed install on bootstrap... You guys made changes to Manual mode CCO STS?

[vrutkovs]

I don't think this has anything to do with secrets. What I see here is rpm-ostree can't run initial rebase cause the node can't resolve one of Quay's CDN domains

[apmlima]

Tried today with the just released 4.12.0-0.okd-2023-03-18-084815 and is fine with this. Thanks!

[vrutkovs]

I don't think we did any specific fix for this fwiw, I'm still convinced this is some infra problem (i.e. one of Quay CDN domains is blocked)