Integrating Pinniped-Concierge with Capsule-Proxy for Tenant Authentication

[aleixam-klarrio]

We are currently working on a project to integrate Pinniped-Concierge with Capsule-Proxy to enable tenant authentication using Pinniped. Our goal is to allow users to authenticate using OIDC. Once authenticated, Pinniped-Concierge will impersonate them and forward their requests to Capsule-Proxy.

During our PoC we encountered two main issues that required changes to Pinniped:

1. RBAC conflict with Capsule-Proxy: Pinniped's impersonation proxy performs RBAC checks before reaching the server API. This conflicts with Capsule-Proxy, which allows tenants to view their assigned namespaces even if their RBAC denies namespace listing. (Basically, Capsule-Proxy intercepts these cluster-wide requests, performs the operations on behalf of the tenants, and patches the responses accordingly.)
2. Configure the Pinniped impersonation proxy endpoint: We want to point the Pinniped impersonation proxy at the Capsule-Proxy endpoint. Currently, Pinniped uses the in-cluster config associated with the service account, which does not allow us to configure the impersonation proxy on our Capsule-Proxy API endpoint.

Given the complexity of the project, we may be missing some general knowledge or alternative approaches to work around these issues. We would appreciate your advice on this:

• Possible solutions or workarounds to the RBAC conflict with Capsule-Proxy.
• How to configure Pinniped's impersonation proxy to point to a different endpoint.

If modifications to Pinniped are required, we are willing to contribute to the project to address these issues. Please let us know if these changes are within the scope of the Pinniped project.

Thank you very much!

[cfryanr]

Hi @aleixam-klarrio, very interesting. I'm not familiar with Capsule-Proxy so I just took a look at the documentation. It looks like a cool project.

One of the reasons that the Concierge impersonation proxy performs those RBAC checks is because the request that comes in to the proxy might be a request authenticated as user A for an API call to impersonate another user B. Only the impersonation proxy is in a position to confirm via k8s API server authorization checks that user A is allowed to impersonate user B, because once the request is allowed by the impersonation proxy, it will be sent to the k8s API server as a new request from the proxy's service account impersonating user B. The k8s API server does not see that the original requester was user A (although that is annotated onto the request for audit logging purposes, the k8s API server will not look at that annotation to make any authorization decisions). By the way, the configuration of the authorizer used to make these checks is also done via in-cluster config.

As you noticed, currently there is no way to configure the Concierge impersonation proxy to use any other backend aside from the actual k8s API server. Making this configurable could cause two potential concerns. Perhaps there are ways to overcome these concerns. We generally try to keep the configuration surface area of Pinniped as small and focused as possible, because that makes it easier to use, and I worry that allowing the configuration of the backend might be quite complex. I also worry that the impersonation proxy makes important assumptions about the detailed behavior of its backend, which are safe because its backend is always the k8s API server. Allowing the backend to be another type of server which is dynamically configured could invalidate those assumptions, making it hard to know if any security properties of the system have changed. Curious if you have any thoughts about how configuration of the backend would work, how complex it would be, and how security properties of the system could be maintained.

In terms of workarounds, I haven't been able to think of any yet. One idea that I had was maybe Capsule-Proxy could be configured to trust the CA of the impersonation proxy to validate mTLS client certs. Then in the kubeconfig, the user could replace the impersonation proxy endpoint with the Capsule-Proxy endpoint. But that doesn't seem great because although that gives the Capsule-Proxy an easy way to authenticate all incoming requests, it does not give it an easy way to forward those requests to the k8s API Server as the same authenticated user (unless Capsule-Proxy implemented impersonation similar to how the Concierge impersonation proxy does it). I'll keep thinking about it to see if I can come up with any better ideas.