ci: check documentation (#308)

Avoid breaking documentation without noticing !

Use docker to build, to enable local build & checks.

.github/workflows/generate-docs.yml

@@ -22,16 +22,12 @@ jobs:
     steps:
       - name: Check out the repo
         uses: actions/checkout@v4
-      - uses: actions/setup-python@v2
-        with:
-          python-version: 3.x
-      - name: Install mkdocs requirements
+      - name: Checks
         run: |
-          pip install mkdocs-material mdx_truly_sane_lists mkdocs-glightbox mdx-breakless-lists
+          make check_docs
       - name: Generate documentation
         run: |
-          bash scripts/build_mkdocs.sh
-          mkdocs build
+          make build_docs
       - name: Deploy documentation to Github Pages 🚀
         # we only deploy on push to main
         if: |

Makefile

@@ -0,0 +1,28 @@
+#!/usr/bin/env make
+
+# use bash !
+SHELL := /bin/bash
+
+.DEFAULT_GOAL := checks
+# avoid target corresponding to file names, to depends on them
+.PHONY: *
+
+#------------#
+# Checks     #
+#------------#
+
+checks: check_docs check_build_docs
+
+check_docs:
+	@echo "🥫 Checking documentation …"
+	@WRONG_EXT=$$(find docs docs/reports -maxdepth 1 -type f|grep -v ".md$$"); \
+	if [[ -n "$${WRONG_EXT}" ]] ; then echo >&2 "File with wrong extensions $${WRONG_EXT}"; exit 1; fi
+
+check_build_docs:
+	@echo "🥫 Building documentation to check it …"
+	@./scripts/build_mkdocs.sh --check
+
+
+build_docs:
+	@echo "🥫 Building documentation …"
+	@./scripts/build_mkdocs.sh

docs/logs-off1.md

@@ -16,4 +16,4 @@ cd /var/log/exim4; mv paniclog paniclog.1; systemctl reload exim4
 
 ## 2023-07-25 Disk full on /rpool/off/products
 
-See [2023-07-26 rpool/off/products dataset full on off1](reports/2023-07-26-rpool-off-products-full.md)
+See [2023-07-26 rpool/off/products dataset full on off1](reports/2023-07-26-off1-rpool-products-full.md)

docs/logs-off3.md

@@ -15,12 +15,12 @@ sudo ip route add 10.1.0.0/16 dev ens19 proto kernel scope link src 10.0.0.3
 
 ## 2023-05-19 Mongodb down
 
-see [2023-05-18 Mongodb down](reports/2023-05-19-overload-of-osm-machine.md)
+see [2023-05-18 Mongodb down](./reports/2023-05-19-overload-of-osm-machine.md)
 
 
 ## 2023-05-05 Mongodb access from off2
 
-see [*Mongodb access* in 2023-03 OFF2 reinstall - opff migration](docs/reports/2023-03-14-off2-opff-reinstall.md#mongodb-access)
+see [*Mongodb access* in 2023-03 OFF2 reinstall - opff migration](./reports/2023-03-14-off2-opff-reinstall.md#mongodb-access)
 
 ## 2023-12-17 Mongodb down
 

docs/prod-architecture.md

@@ -184,7 +184,7 @@ ZFS datastest are replicated snapshoted (backups) and synced.
 
 We use [sanoid](./sanoid.md) to manage snapshot (create them and handle retention policy).
 
-We use [syncoid](./syncoid.md), a companion tool of sanoid to synchronize snapshots on other servers.
+We use [syncoid](./sanoid.md#syncoid-service-and-configuration), a companion tool of sanoid to synchronize snapshots on other servers.
 
 Also some snapshot are used through clones to provide data for staging. (see `/opt/openfoodfacts-infrastructure/scripts/ovh3/maj-clones-nfs-VM-dockers.sh`).
 

docs/producers_sftp.md

@@ -17,7 +17,7 @@ If a producer want's to connect with a key, put the public key in a file named `
 
 ## Adding a new sftp user
 
-Use the script [`add_sftp_user.pl`](../scripts/off1/add_sftp_user.pl) (present in `script/off-proxy`) with user root in the reverse proxy container.
+Use the script `add_sftp_user.pl` (present in `script/off-proxy`) with user root in the reverse proxy container.
 
 **:fire: IMPORTANT :fire::** every user **must be in `sftponly` group** and only in this one.
 

docs/reports/2023-03-14-off2-opff-reinstall.md

@@ -151,7 +151,7 @@ Using:
 * GET/PUT/POST/DELETE /domain/zone/openfoodfacts.org/*
 
 ![token creation at form at OVH](../img/2023-05-ovh-create-token-openfoodfacts.org-form.png "token creation at form at OVH"){width=50%}
-![token creation result](img/2023-05-ovh-create-token-openfoodfacts.org-result.png "token creation result"){width=50%}
+![token creation result](../img/2023-05-ovh-create-token-openfoodfacts.org-result.png "token creation result"){width=50%}
 
 and we put config file in `/root/.ovhapi/openpetfoodfacts.org`
 ```bash

docs/reports/2023-07-off2-off-reinstall.md

@@ -2,7 +2,7 @@
 
 We will follow closely what we did for [OPFF reinstall on off2](./2023-03-14-off2-opff-reinstall.md).
 Refer to it if you need more explanation on a step.
-Also following [OPF and $SERVICE reinstall on off2](./2023-06-07-off2-opf-$SERVICE-reinstall.md).
+Also following [OPF and OBF reinstall on off2](./2023-06-07-off2-opf-obf-reinstall.md).
 
 
 ## switching to right branch of off2 and ovh3
@@ -1789,7 +1789,7 @@ To test my installation I added this to `/etc/hosts` on my computer:
 
 - **DONE:** migrate ip tables rules
   - on reverse proxy
-  - (done) use fail2ban instead of iptables - see [How to use fail2ban to ban bots](./how-to-fail2ban-ban-bots.md)
+  - (done) use fail2ban instead of iptables - see [How to use fail2ban to ban bots](../how-to-fail2ban-ban-bots.md)
   - (wontfix) we dont continue with the cron tail -n 10000 /srv/off/logs/access_log | grep search | /srv/off/logs/ban_abusive_ip.pl > /dev/null 2>&1 for now
   - NOTE: in parallel we are setting up rate limiting with nginx which could then be combined with fail2ban on 409 errors (easy to add to auth error bans)
 

scripts/build_mkdocs.sh

@@ -2,5 +2,41 @@
 
 # Renders markdown doc in docs to html in gh_pages
 
+# --check just checks for errors and warnings
+echo "OPTION IS $1"
+if [[ "$1" == "--check" ]]
+then
+  TMP_BUILD_DIR=$(mktemp -d)
+  DOCKER_ARGS="-v $TMP_BUILD_DIR:/app/gh_pages"
+  # ensure dir exists however to avoid having it created with root:root permissions
+  mkdir -p gh_pages
+fi
+
+# we need to install one more dependency to minidocs/mkdocs
+PIP_INSTALL=$(mktemp)
+cat >$PIP_INSTALL <<EOF
+#!/bin/sh
+echo "installing mdx_truly_sane_lists and mdx-breakless-lists"
+pip3 install mdx_truly_sane_lists mkdocs-glightbox mdx-breakless-lists
+EOF
+# get group id to use it in the docker
+GID=$(id -g)
+
 # copy README.md as the index but change links starting with ./docs/ to ./
-sed -e 's|(\./docs/|(./|g' README.md > docs/index.md
+sed -e 's|(\./docs/|(./|g' README.md > docs/index.md
+
+# we use minidocks capability to add entrypoint to install some pip package
+# we use also it's capability to change user and group id to avoid permissions problems
+docker run --rm \
+  -v $PIP_INSTALL:/docker-entrypoint.d/60-pip-install.sh \
+  -e USER_ID=$UID -e GROUP_ID=$GID \
+  $DOCKER_ARGS \
+  -v $(pwd):/app -w /app \
+  minidocks/mkdocs build --strict
+# get exit code !
+ERROR=$?
+# cleanup
+rm $PIP_INSTALL docs/index.md
+if [[ -n $TMP_BUILD_DIR ]]; then rm -rf $TMP_BUILD_DIR; fi
+
+exit $ERROR