[Bug]: beam_CleanupGCPResources is failing with log "The operation was canceled"

[chamikaramj]

What happened?

https://github.com/apache/beam/actions/workflows/beam_CleanUpGCPResources.yml

For example,
https://github.com/apache/beam/actions/runs/12363928955/job/34506222289

> Task :beam-test-tools:removeStaleK8sWorkload
namespace "beam-performancetests-singlestoreio-12352184229" deleted
Error: The operation was canceled.

Issue Priority

Priority: 1 (data loss / total loss of function)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Infrastructure
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[chamikaramj]

cc: @Abacn

[chamikaramj]

The error log doesn't seem to be very useful unfortunately.

[chamikaramj]

@Abacn assigning to you since you probably have the most context regarding this workflow. Feel free to forward if needed.

[Abacn]

for future reference, it is due to a namespace (beam-performancetests-singlestoreio-12352184229) with finalizer stuck at teardown

To fix, follow https://cloud.google.com/knowledge/kb/deleted-namespace-remains-in-terminating-status-000004867

[chamikaramj]

From where would you run kubectl ? Probably good to document exact commands you used to resolve the error.

[Abacn]

the command I used was gcloud container clusters get-credentials io-datastores --zone us-central1-a --project apache-beam-testing. It is given by GCP console. One can run in cloud shell or locally.

[Abacn]

Re-open this issue to track the task that makes clean up script working for such workload with non-empty finalizer, see #31846 (comment)