diff --git a/developer/config/etcdclient.yaml b/developer/config/etcdclient.yaml new file mode 100644 index 00000000000..0cc06e41a85 --- /dev/null +++ b/developer/config/etcdclient.yaml @@ -0,0 +1,48 @@ +apiVersion: v1 +kind: Pod +metadata: + creationTimestamp: null + labels: + component: etcdclient + tier: debug + name: etcdclient + namespace: kube-system +spec: + containers: + - command: + - sleep + - 9999d + image: apecloud/etcd:v3.5.6 + imagePullPolicy: IfNotPresent + name: etcdclient + resources: {} + volumeMounts: + - mountPath: /etc/kubernetes/pki/etcd + name: etcd-certs + readOnly: true + - mountPath: /var/lib/etcd + name: etcd-data + readOnly: false + env: + - name: ETCDCTL_API + value: "3" + - name: ETCDCTL_CACERT + value: /etc/kubernetes/pki/etcd/ca.crt + - name: ETCDCTL_CERT + value: /etc/kubernetes/pki/etcd/healthcheck-client.crt + - name: ETCDCTL_KEY + value: /etc/kubernetes/pki/etcd/healthcheck-client.key + - name: ETCDCTL_ENDPOINTS + value: "https://127.0.0.1:2379" + # - name: ETCDCTL_CLUSTER + # value: "true" + hostNetwork: true + volumes: + - hostPath: + path: /etc/kubernetes/pki/etcd + type: DirectoryOrCreate + name: etcd-certs + - hostPath: + path: /var/lib/etcd + type: DirectoryOrCreate + name: etcd-data diff --git a/docs/developer.md b/docs/developer.md index 3e524bc81aa..93836e61d5c 100644 --- a/docs/developer.md +++ b/docs/developer.md @@ -47,6 +47,7 @@ Please see these documents for more information about Armadas Design: * [Using OIDC with Armada](./developer/oidc.md) * [Building the Website](./developer/website.md) * [Using Localdev Manually](./developer/manual-localdev.md) +* [Inspecting and Debugging etcd in Localdev setup](./developer/etc-localdev.md) ## Pre-requisites diff --git a/docs/developer/etcd-localdev.md b/docs/developer/etcd-localdev.md new file mode 100644 index 00000000000..fe8c2e309a8 --- /dev/null +++ b/docs/developer/etcd-localdev.md @@ -0,0 +1,74 @@ +# Inspect and Debugging etcd in Localdev setup + +When developing or testing Armada in the Localdev setup, it's sometimes helpful +to directly query the etcd database to gather various statistics. However, by +default, the `kind` tool (for running Kubernetes clusters inside a local Docker +side) does not expose the etcd interface for direct querying. The following +steps give a solution for querying an etcd instance running inside of `kind`. + + +First, verify the running nodes and podes. +```bash +$ kubectl get nodes -A +NAME STATUS ROLES AGE VERSION +armada-test-control-plane Ready control-plane 78m v1.24.7 +armada-test-worker Ready 77m v1.24.7 + +$ kubectl get pods -A +NAMESPACE NAME READY STATUS RESTARTS AGE +ingress-nginx ingress-nginx-admission-create-9xnpn 0/1 Completed 0 78m +ingress-nginx ingress-nginx-admission-patch-phkgm 0/1 Completed 1 78m +ingress-nginx ingress-nginx-controller-646df5f698-zbgqz 1/1 Running 0 78m +kube-system coredns-6d4b75cb6d-9z87w 1/1 Running 0 79m +kube-system coredns-6d4b75cb6d-flz4r 1/1 Running 0 79m +kube-system etcd-armada-test-control-plane 1/1 Running 0 79m +kube-system kindnet-nx952 1/1 Running 0 79m +kube-system kindnet-rtqkc 1/1 Running 0 79m +kube-system kube-apiserver-armada-test-control-plane 1/1 Running 0 79m +kube-system kube-controller-manager-armada-test-control-plane 1/1 Running 0 79m +kube-system kube-proxy-cwl2r 1/1 Running 0 79m +kube-system kube-proxy-wjqft 1/1 Running 0 79m +kube-system kube-scheduler-armada-test-control-plane 1/1 Running 0 79m +local-path-storage local-path-provisioner-6b84c5c67f-22m8m 1/1 Running 0 79m +``` +You should see an etcd control plane pod in the list of pods. + +Copy the etcdclient deployment YAML into the cluster control plane node: + +```bash +$ docker cp developer/config/etcdclient.yaml armada-test-control-plane:/ +``` + +Then, open a shell in the control plane node: +```bash +$ docker exec -it -u 0 --privileged armada-test-control-plane /bin/bash +``` + +In the container shell, move the deployment YAML file to the Kubernetes deployments source +directory. Kubernetes (Kind) will notice the file's appearance and will deploy +the new pod. +```bash +root@armada-test-control-plane:/# mv etcdclient.yaml /etc/kubernetes/manifests/ +root@armada-test-control-plane:/# exit +$ kubectl get pods -A +``` +You should see an etcdclient pod running. + +Open a shell in the new etcdclient utility pod, and start using `etcdctl` to query etcd. +```bash +$ kubectl exec -n kube-system -it etcdclient-armada-test-control-plane -- sh +/ # etcdctl endpoint status -w table ++-------------------------+------------------+---------+---------+-----------+-----------+------------+ +| ENDPOINT | ID | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX | ++-------------------------+------------------+---------+---------+-----------+-----------+------------+ +| https://172.19.0.2:2379 | d7380397c3ec4b90 | 3.5.3 | 3.9 MB | true | 2 | 16727 | ++-------------------------+------------------+---------+---------+-----------+-----------+------------+ +/ # +/ # exit +``` +At this point, you can use `etcdctl` to query etcd for key-value pairs, get the health and/or metrics +of the etcd server. + +## References + +https://mauilion.dev/posts/etcdclient/ diff --git a/docs/developer/manual-localdev.md b/docs/developer/manual-localdev.md index 06c6c4dfea7..b360548de2a 100644 --- a/docs/developer/manual-localdev.md +++ b/docs/developer/manual-localdev.md @@ -4,7 +4,7 @@ Here, we give an overview of a development setup for Armada that gives users ful Before starting, please ensure you have installed [Go](https://go.dev/doc/install) (version 1.20 or later), gcc (for Windows, see, e.g., [tdm-gcc](https://jmeubank.github.io/tdm-gcc/)), [mage](https://magefile.org/), [docker](https://docs.docker.com/get-docker/), [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), and, if you need to compile `.proto` files, [protoc](https://github.com/protocolbuffers/protobuf/releases). -For a full lust of mage commands, run `mage -l`. +For a full list of mage commands, run `mage -l`. ## Setup