Add localdev documentation on accessing etcd (armadaproject#3210) (ar…

…madaproject#3227) Add documentation on accessing the etcd instance in a localev setup (running in a Kind cluster), so developers can use `etcdctl` (or other tools) to access etcd directly, for test/debugging purposes. Ship local copy of etcdclient deployment YAML, with a useful container image reference. Signed-off-by: Rich Scott <[email protected]>
richscott · Jan 4, 2024 · b2c30e6 · b2c30e6
1 parent 2047cd1
commit b2c30e6
Show file tree

Hide file tree

Showing 4 changed files with 124 additions and 1 deletion.
diff --git a/developer/config/etcdclient.yaml b/developer/config/etcdclient.yaml
@@ -0,0 +1,48 @@
+apiVersion: v1
+kind: Pod
+metadata:
+  creationTimestamp: null
+  labels:
+    component: etcdclient
+    tier: debug
+  name: etcdclient
+  namespace: kube-system
+spec:
+  containers:
+  - command:
+    - sleep
+    - 9999d
+    image: apecloud/etcd:v3.5.6
+    imagePullPolicy: IfNotPresent
+    name: etcdclient
+    resources: {}
+    volumeMounts:
+    - mountPath: /etc/kubernetes/pki/etcd
+      name: etcd-certs
+      readOnly: true
+    - mountPath: /var/lib/etcd
+      name: etcd-data
+      readOnly: false
+    env:
+    - name: ETCDCTL_API
+      value: "3"
+    - name: ETCDCTL_CACERT
+      value: /etc/kubernetes/pki/etcd/ca.crt
+    - name: ETCDCTL_CERT
+      value: /etc/kubernetes/pki/etcd/healthcheck-client.crt
+    - name: ETCDCTL_KEY
+      value: /etc/kubernetes/pki/etcd/healthcheck-client.key
+    - name: ETCDCTL_ENDPOINTS
+      value: "https://127.0.0.1:2379"
+    # - name: ETCDCTL_CLUSTER
+    #  value: "true"
+  hostNetwork: true
+  volumes:
+  - hostPath:
+      path: /etc/kubernetes/pki/etcd
+      type: DirectoryOrCreate
+    name: etcd-certs
+  - hostPath:
+      path: /var/lib/etcd
+      type: DirectoryOrCreate
+    name: etcd-data
diff --git a/docs/developer.md b/docs/developer.md
@@ -47,6 +47,7 @@ Please see these documents for more information about Armadas Design:
 * [Using OIDC with Armada](./developer/oidc.md)
 * [Building the Website](./developer/website.md)
 * [Using Localdev Manually](./developer/manual-localdev.md)
+* [Inspecting and Debugging etcd in Localdev setup](./developer/etc-localdev.md)
 
 ## Pre-requisites
 

diff --git a/docs/developer/etcd-localdev.md b/docs/developer/etcd-localdev.md
@@ -0,0 +1,74 @@
+# Inspect and Debugging etcd in Localdev setup
+
+When developing or testing Armada in the Localdev setup, it's sometimes helpful
+to directly query the etcd database to gather various statistics. However, by
+default, the `kind` tool (for running Kubernetes clusters inside a local Docker
+side) does not expose the etcd interface for direct querying. The following
+steps give a solution for querying an etcd instance running inside of `kind`.
+
+
+First, verify the running nodes and podes.
+```bash
+$ kubectl get nodes -A
+NAME                        STATUS   ROLES           AGE   VERSION
+armada-test-control-plane   Ready    control-plane   78m   v1.24.7
+armada-test-worker          Ready    <none>          77m   v1.24.7
+
+$ kubectl get pods -A
+NAMESPACE            NAME                                                READY   STATUS      RESTARTS   AGE
+ingress-nginx        ingress-nginx-admission-create-9xnpn                0/1     Completed   0          78m
+ingress-nginx        ingress-nginx-admission-patch-phkgm                 0/1     Completed   1          78m
+ingress-nginx        ingress-nginx-controller-646df5f698-zbgqz           1/1     Running     0          78m
+kube-system          coredns-6d4b75cb6d-9z87w                            1/1     Running     0          79m
+kube-system          coredns-6d4b75cb6d-flz4r                            1/1     Running     0          79m
+kube-system          etcd-armada-test-control-plane                      1/1     Running     0          79m
+kube-system          kindnet-nx952                                       1/1     Running     0          79m
+kube-system          kindnet-rtqkc                                       1/1     Running     0          79m
+kube-system          kube-apiserver-armada-test-control-plane            1/1     Running     0          79m
+kube-system          kube-controller-manager-armada-test-control-plane   1/1     Running     0          79m
+kube-system          kube-proxy-cwl2r                                    1/1     Running     0          79m
+kube-system          kube-proxy-wjqft                                    1/1     Running     0          79m
+kube-system          kube-scheduler-armada-test-control-plane            1/1     Running     0          79m
+local-path-storage   local-path-provisioner-6b84c5c67f-22m8m             1/1     Running     0          79m
+```
+You should see an etcd control plane pod in the list of pods.
+
+Copy the etcdclient deployment YAML into the cluster control plane node:
+
+```bash
+$ docker cp developer/config/etcdclient.yaml armada-test-control-plane:/
+```
+
+Then, open a shell in the control plane node:
+```bash
+$ docker exec -it -u 0 --privileged armada-test-control-plane  /bin/bash
+```
+
+In the container shell, move the deployment YAML file to the Kubernetes deployments source
+directory. Kubernetes (Kind) will notice the file's appearance and will deploy
+the new pod.
+```bash
+root@armada-test-control-plane:/# mv etcdclient.yaml /etc/kubernetes/manifests/
+root@armada-test-control-plane:/# exit
+$ kubectl get pods -A
+```
+You should see an etcdclient pod running.
+
+Open a shell in the new etcdclient utility pod, and start using `etcdctl` to query etcd.
+```bash
+$ kubectl exec -n kube-system -it etcdclient-armada-test-control-plane -- sh
+/ # etcdctl endpoint status -w table
++-------------------------+------------------+---------+---------+-----------+-----------+------------+
+|        ENDPOINT         |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM | RAFT INDEX |
++-------------------------+------------------+---------+---------+-----------+-----------+------------+
+| https://172.19.0.2:2379 | d7380397c3ec4b90 |   3.5.3 |  3.9 MB |      true |         2 |      16727 |
++-------------------------+------------------+---------+---------+-----------+-----------+------------+
+/ #
+/ # exit
+```
+At this point, you can use `etcdctl` to query etcd for key-value pairs, get the health and/or metrics
+of the etcd server.
+
+## References
+
+https://mauilion.dev/posts/etcdclient/
diff --git a/docs/developer/manual-localdev.md b/docs/developer/manual-localdev.md
@@ -4,7 +4,7 @@ Here, we give an overview of a development setup for Armada that gives users ful
 
 Before starting, please ensure you have installed [Go](https://go.dev/doc/install) (version 1.20 or later), gcc (for Windows, see, e.g., [tdm-gcc](https://jmeubank.github.io/tdm-gcc/)), [mage](https://magefile.org/), [docker](https://docs.docker.com/get-docker/), [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl), and, if you need to compile `.proto` files, [protoc](https://github.com/protocolbuffers/protobuf/releases).
 
-For a full lust of mage commands, run `mage -l`.
+For a full list of mage commands, run `mage -l`.
 
 ## Setup