ArgoCD sometimes fails to template Application manifest when using Helm Secrets

[Benjou02]

Checklist

• I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• I've included steps to reproduce the bug.
• I've pasted the output of argocd version.

Describe the bug

Sometimes, when syncing an app with some secrets:// values file that is indeed where it should be, the sync will fail with the following error, found in the UI and the repo server: Error: open secrets://secrets/prod.sops.yaml: no such file or directory.

(Trying) To Reproduce

We have not been able to reproduce consistently the issue, it seems to occur randomly when syncing such an app. However, here are some findings we discovered when trying to reproduce the behavior.

We did not manage to reproduce the bug manually, directly using the same helm template command (shown in the logs below) from within the repo server. However, the only way we found to reproduce this precise error message, was to launch the command without having installed the helm-secrets plugin.

When the file is missing and helm-secrets is installed, the message is different:

[helm-secrets] File does not exist: secrets/prod.sops.yaml 
Error: plugin "scripts/run.sh downloader" exited with error

The fact that this error seems random but always mentions the secrets:// file, combined with the fact that we could not obtain this exact error other than by removing the plugin, suggests that the problem is in the integration of the plugin.

I am opening this issue in the ArgoCD repository as it is the only place where we saw the problem occur, since we were not able to reproduce it using directly Helm.

Expected behavior

The sync should never fail with this error when the file is not missing.

Versions

argocd: v2.11.7+e4a0246
helm: version.BuildInfo{Version:"v3.14.4", GitCommit:"81c902a123462fd4052bc5e9aa9c513c4c8fc142", GitTreeState:"clean", GoVersion:"go1.21.9"}
helm-secrets: 4.2.2-dev

Logs

In the repo server:

time="2024-12-31T13:11:20Z" level=error msg="finished unary call with code Unknown" error="Manifest generation error (cached): `helm template . --name-template my-app --namespace app --kube-version 1.29 --values <path to cached source>/values.yaml --values secrets://secrets/prod.sops.yaml <api versions removed> --include-crds` failed exit status 1: Error: open secrets://secrets/prod.sops.yaml: no such file or directory" grpc.code=Unknown grpc.method=GenerateManifest grpc.service=repository.RepoServerService grpc.start_time="2024-12-31T13:11:20Z" grpc.time_ms=10.015 span.kind=server system=grpc

[agaudreault]

I don't think Argo is responsible for the failure and I am not sure what can be implemented in the Argo codebase to help with the issue. You could try to add additional logs and run a fork to investigate. I would suggest you to validate with https://github.com/jkroepke/helm-secrets/wiki/ArgoCD-Integration if it is installed properly and open an issue in https://github.com/jkroepke/helm-secrets/issues?q=sort%3Aupdated-desc+is%3Aissue+is%3Aopen.

Most people are using https://external-secrets.io/latest/ for Kubernetes secret management, so it does not have to be rendered by the deployment toolchain.