WIP: Kubernetes Deployment

.github/workflows/main.yml

@@ -29,7 +29,7 @@ jobs:
     - name: Cache node modules
       uses: actions/cache@v1
       with:
-        path: ~/.cache/yarn 
+        path: ~/.cache/yarn
         key: yarn-${{ hashFiles('**/yarn.lock') }}
         restore-keys: |
           yarn-${{ hashFiles('**/yarn.lock') }}

deploy/hcloud-k8s/Makefile

@@ -0,0 +1,16 @@
+install: install-ansible-dependencies install-hcloud-servers install-k8s-cluster
+
+install-ansible-dependencies:
+	ansible-galaxy collection install community.kubernetes
+	ansible-galaxy collection install hetzner.hcloud
+
+install-hcloud-servers:
+	ansible-playbook create-hcloud-infrastructure.yaml  -e "state=present"
+	sleep 300
+
+install-k8s-cluster:
+	ansible-playbook create-kubernetes-cluster.yaml -i env/inventory --private-key ~/.ssh/talexdev_rsa
+	ansible-playbook deploy-db-prometheus.yaml
+	ansible-playbook deploy-db-pg.yaml
+	ansible-playbook deploy-app-termine.yaml
+	ansible-playbook deploy-ingress.yaml

deploy/hcloud-k8s/README.md

@@ -0,0 +1,103 @@
+# Covid19 Teststation - Deployment on Kubernetes
+
+This directory provides an easy-to-use scripting to deploy the Covid19 Teststation on a bare-metal [Kubernetes](https://kubernetes.io) cluster.
+As infrastructure several virtual servers and other resources will be provisioned on [Hetzner Cloud](https://www.hetzner.com/de/cloud) using [Terraform](https://www.terraform.io):
+
+- **Servers**: at least one virtual server (CX11, 1 vCPU, 2 GB RAM) as Master Node and another one (CX21, 2 vCPUs, 4 GB RAM) as Worker Node.
+  We recommend using two Worker Nodes, although a single machine will do the job.
+- **Volumes**: Kubernetes will provision just some small persistent volumes using the [HCloud CSI provider](https://github.com/hetznercloud/csi-driver). 
+- **Load Balancers**: Traefik will provision an [HCloud Load Balancer](https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/docs/load_balancers.md)
+  that forwards HTTPS traffic to the Kubernetes cluster. TLS termination is done by this load balancer, DNS records will point to its ip-address.
+- **Networks**: [Cilium](https://cilium.io) is being used as Kubernetes pod network. The [HCloud Controller Manager](https://github.com/hetznercloud/hcloud-cloud-controller-manager/blob/master/docs/deploy_with_networks.md)   
+  is configured to use a private host network in the Hetzner Cloud.
+
+An HCloud setup like this will cost about 15-20 € per month. Therefore, the costs for running a test system for a just few days will be about 1-2 €.
+The [Hetzner DNS Console](https://dns.hetzner.com) service is for free.
+
+To run the application we use the following open-source services:
+
+- **Traefik**: an easy-to-use Ingress Controller, deployed via Helm chart. The installation integrates [External-DNS](https://github.com/kubernetes-sigs/external-dns)
+  which provisions DNS records in Hetzner DNS Console as annotated in Traefik Kubernetes Service object. Also Instrumentation using Prometheus and Grafana is configured.
+- **Prometheus**: collects metrics from infrastructure and services. Deployed using Helm chart from Prometheus-Operator Kube Stack.
+- **Grafana**: also installed by Prometheus-Operator Kube Stack package.
+- **Postgres**: full-featured SQL database, the Covid19 Teststation stores appointments information here.
+
+## Prerequisites
+
+Before you can start the deployment please follow these steps to create your Hetzner Cloud account:
+
+### Hetzner Cloud Registration
+
+![](img/01-hcloud-registration.png)
+
+Register with your email and select a password. New accounts must confirm their emails before first login.
+Two-Factor authentication has to be enabled later on and is recommended for security reasons.
+New HCloud accounts also have a resource limit but it's enough for your first deployments and can be increased later on.
+
+### Create new HCloud Project
+
+![](img/02-hcloud-add-project.png)
+
+On HCloud access to resources is scoped by projects. Create a new project in the HCloud user interface and select your project from the dashboard.
+
+### Create new HCloud API Token
+
+![](img/03-hcloud-api-tokens.png)
+
+Navigate to the Security section within HCloud user interface. There's no need to add your SSH key here as this information will be provided by Terraform scripting.
+
+![](img/04-hcloud-generate-token.png)
+
+The API token needs to be created with read/write permissions. You can use any name/description but the API token will only be shown at creation time.
+Copy your API token and store it somewhere on your computer e.g. using a safe password store.
+
+### Add Payment Method to Hetzner Cloud Account
+
+![](img/05-hcloud-add-payment.png)
+
+Before you can continue and use your HCloud account you need to add a payment method.
+
+### Add DNS Zone to Hetzner DNS Console
+
+![](img/06-dns-console-add-zone.png)
+
+To get this deployment up and running you need to configure a DNS zone for use with Hetzner DNS Console. This service provides a free
+primary and secondary name server which can be controlled programmatically. The domain needs to be switch completely as HCloud does not support DNS sub zones.
+Existing DNS entries may be migrated automatically by Hetzner DNS Console. Please follow the instructions on the Hetzner website.
+
+![](img/07-dns-console.png)
+
+If the migration to Hetzner DNS Console was successful you can use your domain for the deployment of Covid19 Teststation.
+
+### Create new Hetzner DNS Console API Token
+
+![](img/08-dns-console-generate-token.png)
+
+Create and save a new Hetzner DNS Console token. You will need this secret to enable External-DNS to create automatically DNS records.
+
+## Configure Infrastructure ENV
+
+### HCloud Credentials for Ansible
+
+Copy `env/credentials.yaml.sample` as `env/credentials.yaml` and add the missing information:
+
+...
+...
+...
+
+### Application Setup
+
+
+```bash
+ansible-galaxy collection install community.kubernetes
+ansible-galaxy collection install hetzner.hcloud
+
+ansible-playbook create-hcloud-infrastructure.yaml  -e "state=present"
+ansible-playbook create-kubernetes-cluster.yaml -i env/inventory --private-key ~/.ssh/talexdev_rsa
+ansible-playbook deploy-db-prometheus.yaml
+ansible-playbook deploy-db-pg.yaml
+ansible-playbook deploy-app-termine.yaml
+ansible-playbook deploy-ingress.yaml
+```
+
+install: htpasswd, openssl

deploy/hcloud-k8s/ansible.cfg

@@ -0,0 +1,9 @@
+[defaults]
+vault_password_file = ~/.ansible-vault-pass
+host_key_checking = False
+inventory = inventory
+deprecation_warnings = False
+interpreter_python = auto_legacy_silent
+
+[ssh_connection]
+pipelining = True

deploy/hcloud-k8s/create-hcloud-infrastructure.yaml

@@ -0,0 +1,12 @@
+---
+- name: create hosts and network infrastructure on Hetzner Cloud
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/credentials.yaml"
+    - "env/setup.yaml"
+    - "env/hcloud.yaml"
+    - "env/app.yaml"
+  roles:
+    - { role: tf-infrastructure, tags: tf-infrastructure }

deploy/hcloud-k8s/create-kubernetes-cluster.yaml

@@ -0,0 +1,42 @@
+---
+- name: install Kubernetes control plane on first master node
+  hosts: master[0]
+  gather_facts: yes
+  remote_user: k8s
+  become: yes
+  vars_files:
+    - "env/credentials.yaml"
+    - "env/setup.yaml"
+    - "env/hcloud.yaml"
+    - "env/app.yaml"
+    - "env/ips.yaml"
+  vars:
+    kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+  roles:
+    - { role: kube-prepare, tags: kube-prepare }
+    - { role: kube-master, tags: kube-master }
+    - { role: kube-config, tags: kube-config }
+
+- name: join Kubernetes control plane with other master nodes
+  hosts: master[1:]
+  gather_facts: yes
+  remote_user: k8s
+  become: yes
+  vars_files:
+    - "env/setup.yaml"
+    - "env/hcloud.yaml"
+  roles:
+    - { role: kube-prepare, tags: kube-prepare }
+    - { role: kube-join-master, tags: kube-join-master }
+
+- name: join Kubernetes cluster with worker nodes
+  hosts: worker
+  gather_facts: yes
+  remote_user: k8s
+  become: yes
+  vars_files:
+    - "env/setup.yaml"
+    - "env/hcloud.yaml"
+  roles:
+    - { role: kube-prepare, tags: kube-prepare }
+    - { role: kube-join-worker, tags: kube-join-worker }

deploy/hcloud-k8s/deploy-app-termine.yaml

@@ -0,0 +1,12 @@
+---
+- name: deploy Termine application
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/setup.yaml"
+    - "env/app.yaml"
+  vars:
+    kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+  roles:
+    - { role: termine, tags: termine }

deploy/hcloud-k8s/deploy-db-ldap.yaml

@@ -0,0 +1,16 @@
+---
+- name: deploy 389DS LDAP server
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/setup.yaml"
+    - "env/app.yaml"
+  vars:
+    kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+  roles:
+    - { role: ldap, tags: ldap }
+  pre_tasks:
+    - name: create k8s namespace
+      shell: |
+        kubectl --kubeconfig {{ kubeconfig }} create namespace {{ namespace.ldap }} || true

deploy/hcloud-k8s/deploy-db-pg.yaml

@@ -0,0 +1,24 @@
+---
+- name: deploy Postgres SQL database
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/setup.yaml"
+    - "env/app.yaml"
+  vars:
+    kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+  roles:
+    - { role: postgres, tags: postgres }
+  pre_tasks:
+    - name: install Ansible module dependencies
+      pip:
+        name: openshift
+        state: present
+    - name: "create k8s namespace - {{ namespace.postgres }}"
+      community.kubernetes.k8s:
+        kubeconfig: "{{ kubeconfig }}"
+        name: "{{ namespace.postgres }}"
+        api_version: v1
+        kind: Namespace
+        state: present

deploy/hcloud-k8s/deploy-db-prometheus.yaml

@@ -0,0 +1,24 @@
+---
+- name: deploy Prometheus Operator and Grafana
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/setup.yaml"
+    - "env/app.yaml"
+  vars:
+    kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+  roles:
+    - { role: prometheus-operator, tags: prometheus-operator }
+  pre_tasks:
+    - name: install Ansible module dependencies
+      pip:
+        name: openshift
+        state: present
+    - name: "create k8s namespace - {{ namespace.prometheusOperator }}"
+      community.kubernetes.k8s:
+        kubeconfig: "{{ kubeconfig }}"
+        name: "{{ namespace.prometheusOperator }}"
+        api_version: v1
+        kind: Namespace
+        state: present

deploy/hcloud-k8s/deploy-ingress.yaml

@@ -0,0 +1,39 @@
+---
+- name: deploy Traefik as ingress controller
+  hosts: localhost
+  gather_facts: false
+  become: false
+  vars_files:
+    - "env/credentials.yaml"
+    - "env/setup.yaml"
+    - "env/app.yaml"
+    - "env/hcloud.yaml"
+    - "env/ips.yaml"
+  vars:
+    - kubeconfig: "{{ kube_dir }}/{{ kube_config }}"
+    - domain_filters: [ "{{ subdomain }}.{{ domain }}" ]
+    - hostnames:
+      - "{{ hostname.traefik_dashboard }}.{{ subdomain }}.{{ domain }}"
+      - "{{ hostname.application }}.{{ subdomain }}.{{ domain }}"
+  roles:
+    - { role: external-dns, tags: external-dns }
+    - { role: traefik, tags: traefik }
+  pre_tasks:
+    - name: install Ansible module dependencies
+      pip:
+        name: openshift
+        state: present
+    - name: "create k8s namespace - {{ namespace.externalDNS }}"
+      community.kubernetes.k8s:
+        kubeconfig: "{{ kubeconfig }}"
+        name: "{{ namespace.externalDNS }}"
+        api_version: v1
+        kind: Namespace
+        state: present
+    - name: "create k8s namespace - {{ namespace.traefik }}"
+      community.kubernetes.k8s:
+        kubeconfig: "{{ kubeconfig }}"
+        name: "{{ namespace.traefik }}"
+        api_version: v1
+        kind: Namespace
+        state: present

deploy/hcloud-k8s/env/app.yaml

@@ -0,0 +1,60 @@
+#
+# PRESETS: DO NOT CHANGE UNLESS YOU KNOW WHY :)
+#
+
+# Kubernetes cluster cert
+kube_dir: ~/.kube
+kube_config: kubernetes-admin-covid-test-station
+
+# Time-to-live for DNS entries
+dns_ttl: 300
+
+# Managed SSL certs
+managed_cert_name: k8s-tls-managed-cert-covidts
+
+# Kubernetes Namespaces
+namespace:
+  prometheusOperator: monitoring
+  postgres: postgres
+  ldap: ldap
+  externalDNS: traefik
+  traefik: traefik
+
+# Helm chart version pinning
+helm_chart_version:
+  prometheusOperator: 16.14.1
+  postgres: 10.5.3
+  externalDNS: 5.2.0
+  traefik: 10.0.2
+
+docker_image_version:
+  hcloudCCM: v1.11.1
+
+# Parameters for Traefik setup
+traefik:
+  dashboard:
+    user: "admin"
+    password: "DKL-3K2-Ld2-Dhg"
+
+# Helm chart values for Prometheus-Operator
+prometheusOperator:
+  deploy: true
+  grafana:
+    user: "admin"
+    password: "DKL-3K2-Ld2-Dhg"
+    persistentConfig: true
+
+# Helm chart values for Postgres database
+postgres:
+  port: 5432
+  volumeSize: 10Gi
+  database: "termine"
+  password: "termine_pw!"
+  metrics: true
+
+# Docker ENV parameters for 389DS LDAP server
+ldap:
+  port: 389
+  tlsPort: 636
+  volumeSize: 10Gi
+  password: "secret"