Intermediate Certificates are not trusted by fluentbit kubernetes filter

[firethestars]

Bug Report

Describe the bug
We are receiving TLS errors from kubernetes filters when an intermediate certificate is used as the ca.crt for kubernetes.

We have a kubernetes server that uses a certificate chain Offline Root CA -> Intermediate CA (functional Root) -> Intermediate Cert -> Kubernetes Cert. By default, kubernetes provides only the Intermediate Cert in any user token mounts (in the projected mount into a pod for the service, the CA is just the Intermediate Cert in ca.crt, and cannot be a chain or k8s controllers break). Our security team indicates that it is bad practice to share more certificates than necessary in our trust stores because it leads many services to trust more widely than they should, but also it should be true that if we indicate that you should trust an intermediate cert, it should not matter if you also trust the parent cert.

In this case, we see:

1. Just using default settings for the kubernetes filter causes a tls error till tls.verify is set to off.
2. Using the Kube_CA_File and a single file with all certificates concatenated in the chain works, but we cannot do this in production.
3. Using a Kube_CA_File pointing at the projected mount ca and Kube_CA_Path pointed to a folder with the rest of the chain also does not work (this does work with openssl's s_client, which is the only reason we tried it.)

To Reproduce
Run a kubernetes cluster with an intermediate certificate and use any kubernetes filter without disabling tls.verify.

example logs (repeated continuously):

[2024/11/01 17:21:34] [error] [tls] error: unexpected EOF with reason: certificate verify failed                                                                                                                                                                                          
[2024/11/01 17:21:34] [error] [filter:kubernetes:log_filter_kubernetes_k8s_container_k8s_container_custom_tag] kube api upstream connection error

Expected behavior
No error a successful connection to k8s to get pod details to annotate the logs.

Screenshots
none

Your Environment

• Version used: 3.1 and 3.2
• Configuration: Any kubernetes filter configured as above.
• Environment name and version: Kubernetes v1.30.5
• Server type and version:
• Operating System and version: standard fluentbit container image
• Filters and plugins: Kubernetes filter

Additional context

I think this is likely related to #3449 and appears to be happening in AKS as well Azure/AKS#3282 but both issues seem to have no traction and be dead.