Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue certifcates for etcd-operator #9

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

ArkaSaha30
Copy link

This PR will add a basic capability to issue self-signed certificates for etcd-operator.

  • Prerequisite: cert-manager needs to be installed before deploying the etcd-operator
    kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.16.2/cert-manager.yaml

  • Once etcd-operator is deployed it requests 3 certificates from an issuer in the etcd-operator-system namespace

    • peer certificate
    • client certificate
    • server certificate
deployment.apps/etcd-operator-controller-manager created
certificate.cert-manager.io/etcd-operator-etcd-client-certificate created
certificate.cert-manager.io/etcd-operator-etcd-peer-certificate created
certificate.cert-manager.io/etcd-operator-etcd-server-certificate created
issuer.cert-manager.io/etcd-operator-selfsigned created

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.22.7 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ kubectl get secrets -n etcd-operator-system
NAME              TYPE                DATA   AGE
etcd-client-tls   kubernetes.io/tls   3      33s
etcd-peer-tls     kubernetes.io/tls   3      33s
etcd-server-tls   kubernetes.io/tls   3      34s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.22.7 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ kubectl -n etcd-operator-system get secret etcd-client-tls -o yaml | grep ca.crt
  ca.crt: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUMxRENDQWJ5Z0F3SUJBZ0lSQU41cTB2b2tNREsrT3haankvQmZMcE13RFFZSktvWklodmNOQVFFTEJRQXcKQURBZUZ3MHlOREV5TURZeE1USXpNalJhRncweU5UQXpNRFl4TVRJek1qUmFNQUF3Z2dFaU1BMEdDU3FHU0liMwpEUUVCQVFVQUE0SUJEd0F3Z2dFS0FvSUJBUURDVHFVcjZNdnJXLzRYSjJna0Y4cWhhMnFWSnl1L1ozcmhuNENKCkZjQUdPOUlCUmxmbGpiajUvbU5iMW5SUkFJOXNraCtFdWlYd2Y1SFVQbTBNcGpnM3Y0MlRUbzd4MkIyTHloZmUKcDFrQVdBVlFPallOeFZHZjk1ZEFzK3U5MW5Xbi8zM2hHajlodlFTallJT1pORllFOS9xSHJQTlk3SEs4RFRCWQp4ZjZEaVIxSlFUZ1Jmc1FvRkpoZDRCMVhUeDV4eGlMdi85NzRLcmZDbmgrckMyeVVzZWRCMHlEaGlVTm9LVlM3CnU3RUdETC9SOFZUalJGWnphWEM4YXl6OUY1eTlwL1pmbjIzMElrOThqdVBzL0VURUZYUXBuUFFDaGFOTU9JaE0KcEs0ZWIvSWw3ek5lV3p0bnFvUjFJUSttYW1zTWd6bmk0RS9PMGxHbEYxekluMTBwQWdNQkFBR2pTVEJITUE0RwpBMVVkRHdFQi93UUVBd0lGb0RBTUJnTlZIUk1CQWY4RUFqQUFNQ2NHQTFVZEVRRUIvd1FkTUJ1Q0dXVjBZMlF1ClpYUmpaQzF2Y0dWeVlYUnZjaTF6ZVhOMFpXMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBQlhhbGJiR1VIbUwKbURCK0piK3pMTzlYT2ZwR21JMGhZRmdPakVtZmxqem4wa0ZRRHo0YmQyc1FlVi95U1BIZC84RXN2Y0NlME1ucwpaRXF1ejNTWUp5cmg2a2g0MjhqVTJPRGI2MXBUMmwreTUvdkU4bi8yVzgxTERYQzlOUnZNYkR2MmlYOVpORFhuCnZndVhnOUJPUXByd3craXlBUVFHdFk4NWYvc2pOdVZnQW8vU1ZQR2RHVUFWME5RbllSNFN1ZmpwOTh5aS9EK1IKRGM5WDNiSUtYVDNNWmcyKzcyU0JYV3YzSjVBZEFyU3lnVlRMUWVqdk5LZzIvbDMyUkN2cFgzR01jYk1IbFBOQgp3NUpkSmU5QWc2QS9mdTZJS1lDREowUlRlQVkwcDlCaEgvUmFncmhsZkdvcVA4U3V5YjZXbjhoekNLT3FPZDVGClJGRlhzTC9Qd1N3PQotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==

@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

@ArkaSaha30 please rebase this PR. I just merged #8

go.mod Outdated Show resolved Hide resolved
config/certmanager/cert-manager_issuer.yaml Outdated Show resolved Hide resolved
config/certmanager/etcd-client-cert.yaml Outdated Show resolved Hide resolved
config/crd/bases/operator.etcd.io_etcdclusters.yaml Outdated Show resolved Hide resolved
@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

A couple of high level thoughts:

  • For etcd-cluster's certificate,
    • add the cluster name as the prefix, i.e. cluster1-etcd-client-tls, etc. so that different clusters won't conflict.
    • expose an API for the reconciliation workflow to call to generate certificates for the etcd clusters
      • each member in one cluster should have different certificate

@ahrtr
Copy link
Member

ahrtr commented Dec 6, 2024

cc @hakman

@k8s-ci-robot k8s-ci-robot added size/M and removed size/L labels Dec 7, 2024
@ArkaSaha30 ArkaSaha30 force-pushed the issue-cert branch 2 times, most recently from 3685a85 to f5456d0 Compare December 7, 2024 18:33
@ArkaSaha30
Copy link
Author

A couple of high level thoughts:

  • For etcd-cluster's certificate,

    • add the cluster name as the prefix, i.e. cluster1-etcd-client-tls, etc. so that different clusters won't conflict.

    • expose an API for the reconciliation workflow to call to generate certificates for the etcd clusters

      • each member in one cluster should have different certificate

Sure, thank you!
Should I update it in this PR or raise a separate as an incremental development?

@ahrtr ahrtr mentioned this pull request Dec 9, 2024
11 tasks
@ahrtr
Copy link
Member

ahrtr commented Dec 9, 2024

Please read #10

@ArkaSaha30 ArkaSaha30 changed the title Issue certifcates for etcd-operator [WIP]Issue certifcates for etcd-operator Jan 12, 2025
@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ArkaSaha30
Once this PR has been reviewed and has the lgtm label, please assign justinsb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ArkaSaha30
Copy link
Author

Updating the current progress here, making changes to this PR to accommodate certificates prefixed with etcd cluster names as per mentioned in the issue.
I'll remove [WIP] once it's completed and ready for review.

@gdasson
Copy link
Contributor

gdasson commented Jan 13, 2025

@ArkaSaha30 : Please let me know if you need help in any tasks. Just to avoid overlap, I'll wait for you to confirm if I can pick up some of the tasks. Thanks.

@ArkaSaha30
Copy link
Author

ArkaSaha30 commented Jan 13, 2025

@ArkaSaha30 : Please let me know if you need help in any tasks. Just to avoid overlap, I'll wait for you to confirm if I can pick up some of the tasks. Thanks.

Sure, thank you!
I will quickly finish this part following which it will be nice if you can help with the next task - Update reconciliation workflow to mount the secrets into etcd POD.

@gdasson
Copy link
Contributor

gdasson commented Jan 13, 2025

Sure, once you complete this, I will add that the code for secrets mounting in the same PR.

@ArkaSaha30
Copy link
Author

ArkaSaha30 commented Jan 14, 2025

Updated the PR to align with the certificate management design,

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k apply -k config/samples/                                              
etcdcluster.operator.etcd.io/etcdcluster-sample created

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get etcdcluster -A
NAMESPACE   NAME                 AGE
default     etcdcluster-sample   12s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get certificates
NAME                                             READY   SECRET                               AGE
etcdcluster-sample-etcd-client-tls-certificate   True    etcdcluster-sample-etcd-client-tls   24s
etcdcluster-sample-etcd-peer-tls-certificate     True    etcdcluster-sample-etcd-peer-tls     24s
etcdcluster-sample-etcd-server-tls-certificate   True    etcdcluster-sample-etcd-server-tls   24s

etcd-operator on  issue-cert via 🐳 colima via 🐹 v1.23.4 on ☁️  (us-east-1) on ☁️  [email protected] 
❯ k get secrets
NAME                                 TYPE                DATA   AGE
etcdcluster-sample-etcd-client-tls   kubernetes.io/tls   3      31s
etcdcluster-sample-etcd-peer-tls     kubernetes.io/tls   3      32s
etcdcluster-sample-etcd-server-tls   kubernetes.io/tls   3      32s

etcd-operator-controller-manager logs:

2025-01-14T23:12:16Z    INFO    Reconciling EtcdCluster {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "spec": {"size":4,"version":"3.5.17","tls":{"member":{"peerSecret":"etcd-peer-tls","serverSecret":"etcd-server-tls"},"operatorSecret":"etcd-client-tls","provider":"cert-manager"}}}
2025-01-14T23:12:16Z    INFO    Reconciling EtcdCluster certificates    {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "tls": {"member":{"peerSecret":"etcd-peer-tls","serverSecret":"etcd-server-tls"},"operatorSecret":"etcd-client-tls","provider":"cert-manager"}}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Client Certificate   {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-client-tls-certificate": "etcdcluster-sample-etcd-client-tls"}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Peer Certificate     {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-peer-tls-certificate": "etcdcluster-sample-etcd-peer-tls"}
2025-01-14T23:12:16Z    INFO    Starting reconciliation of Server Certificate   {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "etcdcluster-sample-etcd-server-tls-certificate": "etcdcluster-sample-etcd-server-tls"}
2025-01-14T23:12:16Z    INFO    Successfully reconciled EtcdCluster certificates        {"controller": "etcdcluster", "controllerGroup": "operator.etcd.io", "controllerKind": "EtcdCluster", "EtcdCluster": {"name":"etcdcluster-sample","namespace":"default"}, "namespace": "default", "name": "etcdcluster-sample", "reconcileID": "0784e8f6-e3ab-413b-bf0b-ed754eb3b66c", "tls": [{"metadata":{"name":"etcdcluster-sample-etcd-client-tls-certificate","namespace":"default","uid":"c0037b03-1ae2-4bc6-b9e0-f72ef03e5fbc","resourceVersion":"808","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-client-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}},{"metadata":{"name":"etcdcluster-sample-etcd-peer-tls-certificate","namespace":"default","uid":"7fc98c31-548b-415c-a582-a8edaa14b9e0","resourceVersion":"809","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-peer-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}},{"metadata":{"name":"etcdcluster-sample-etcd-server-tls-certificate","namespace":"default","uid":"a64b484a-7f3a-4552-b5db-d2b34f23ebfa","resourceVersion":"810","generation":1,"creationTimestamp":"2025-01-14T23:12:16Z","ownerReferences":[{"apiVersion":"operator.etcd.io/v1alpha1","kind":"EtcdCluster","name":"etcdcluster-sample","uid":"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"manager","operation":"Update","apiVersion":"cert-manager.io/v1","time":"2025-01-14T23:12:16Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"5e05eee5-bf2d-4bc5-aec8-030c2ee9708b\"}":{}}},"f:spec":{".":{},"f:dnsNames":{},"f:issuerRef":{".":{},"f:kind":{},"f:name":{}},"f:secretName":{}}}}]},"spec":{"dnsNames":["etcd.etcd-operator-system"],"secretName":"etcdcluster-sample-etcd-server-tls","issuerRef":{"name":"etcd-operator-selfsigned","kind":"ClusterIssuer"}},"status":{}}]}

@ArkaSaha30 ArkaSaha30 changed the title [WIP]Issue certifcates for etcd-operator Issue certifcates for etcd-operator Jan 14, 2025
@ArkaSaha30 ArkaSaha30 requested a review from ahrtr January 15, 2025 07:26
@ArkaSaha30 ArkaSaha30 force-pushed the issue-cert branch 2 times, most recently from 35ca541 to 3be61a1 Compare January 15, 2025 09:52
@ArkaSaha30
Copy link
Author

/retest-required

internal/controller/utils.go Outdated Show resolved Hide resolved
internal/controller/utils.go Outdated Show resolved Hide resolved
internal/controller/constants.go Outdated Show resolved Hide resolved
config/manager/kustomization.yaml Outdated Show resolved Hide resolved
@ahrtr
Copy link
Member

ahrtr commented Jan 15, 2025

It seems that you only create one secret (certificate & key pair) for each etcd cluster? We need to different certificate & key for each member.

@ahrtr
Copy link
Member

ahrtr commented Jan 15, 2025

It seems that you only create one secret (certificate & key pair) for each etcd cluster? We need to different certificate & key for each member.

Also have you tested this PR in a K8s cluster?

@ArkaSaha30
Copy link
Author

It seems that you only create one secret (certificate & key pair) for each etcd cluster? We need to different certificate & key for each member.

Also have you tested this PR in a K8s cluster?

Yes, I have been testing this on a kind cluster. The logs mentioned in comment are from a kind cluster.

@ArkaSaha30
Copy link
Author

It seems that you only create one secret (certificate & key pair) for each etcd cluster? We need to different certificate & key for each member.

Yes, currently I am creating client, server and peer certificates for each etcd cluster.
So, do we need to create peer and server certificate for each member? and just one client certificate for the etcdcluster?
In that case, should we append the member name with the peer and server certs/secrets?

@ahrtr
Copy link
Member

ahrtr commented Jan 15, 2025

So, do we need to create peer and server certificate for each member? and just one client certificate for the etcdcluster?
In that case, should we append the member name with the peer and server certs/secrets?

Different members should have different certificates. Specifically, each member should have at least 2 certificates, one used for etcdserver to validate client connections, the others also used for etcdserver to validate peer's connection.

The etcd-operator just needs one certificate to connect to etcd PODs.

@ArkaSaha30
Copy link
Author

So, do we need to create peer and server certificate for each member? and just one client certificate for the etcdcluster?
In that case, should we append the member name with the peer and server certs/secrets?

Different members should have different certificates. Specifically, each member should have at least 2 certificates, one used for etcdserver to validate client connections, the others also used for etcdserver to validate peer's connection.

The etcd-operator just needs one certificate to connect to etcd PODs.

Got it! At what stage does it make sense to create the member specific certificates? Once the members are ready or once the member creation request is submitted?

@ahrtr
Copy link
Member

ahrtr commented Jan 19, 2025

At what stage does it make sense to create the member specific certificates?

We need to ensure the certificate is ready before creating each etcd member/POD.

@ArkaSaha30
Copy link
Author

At what stage does it make sense to create the member specific certificates?

We need to ensure the certificate is ready before creating each etcd member/POD.

Just to clarify, the client and peer cert names should have the member/POD name prefixed/suffixed to it?

This commit will add the capability to issue selfsigned certificates for etcd-operator.
These certs are issued with the etcdcluster resouce name prepended and also reconciles these certs.

Signed-off-by: ArkaSaha30 <[email protected]>
@ArkaSaha30
Copy link
Author

As per the discussion in etcd-operator meeting, we need to create a certificate package to manage the certificates so that it is generic and future changes can be isolated to the package itself.
cc @ahrtr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants