docs: matomo for performance

confs/common/systemd/system/email-failures@.service

@@ -4,6 +4,6 @@ Description=%i failure email notification
 [Service]
 Type=oneshot
 # replace __ by @ in instance name to be able to call status on a specific instance
-# then use mailx to email status
+# then use mailx to email status (apt install bsd-mailx if needed)
 ExecStart=/bin/bash -c 'INSTANCE_NAME=%i; SERVICE_NAME=$${INSTANCE_NAME//__/@}; /bin/systemctl status $$SERVICE_NAME | /usr/bin/mailx -s "[$$SERVICE_NAME] failure notification" root'
 

docs/matomo.md

@@ -7,7 +7,19 @@ Available at: https://analytics.openfoodfacts.org/
 You must have a user account to access it (hopefully !). Ask for an admin to create you an account if you need it (Beware, there are personal information in the sense of GDPR like ip addresses).
 Ask for it to *contact* email.
 
-See also [Install log](./reports/2021-02-22-matomo-install.md)
+## Main running services
+
+* nginx is used as a HTTP frontend
+* php7.3-fpm run the matomo software
+* mariadb is the main database for matomo
+* redis is used to fast track  matomo requests (see [Matomo setup for performance, below](#matomo-setup-for-performance))
+* Two systemd timer takes care of putting tracking from redis to mariadb and to consolidate archive reports (see [Matomo setup for performance, below](#matomo-setup-for-performance))
+* prometheus exporters are installed for nginx and mysql (see [Prometheus exporters, below](#prometheus-exporters))
+
+Most of those systemd services should email on failures.
+
+Important configuration files are linked to this git repository (see [using git, below](#using-git))
+
 
 ## Site setup
 
@@ -28,4 +40,36 @@ To be GDPR compliant (and user friendly) [^gdpr_ref]:
 
 ### In productopener
 
-We use the `$google_analytics` variable in config to add the javascript snippet for Matomo.
+We use the `$google_analytics` variable in config to add the javascript snippet for Matomo.
+
+## Matomo install
+
+See also [Install log](./reports/2021-02-22-matomo-install.md) and [2023-12-11 Matomo down](./reports/2023-12-11-matomo-down.md)
+
+### using git
+
+We use git to track some important configurations files in this repository.
+
+The repository is checked out in /opt/openfoodfacts-infrastructure and specific `/etc` configurations files are symlinked there (see `confs/matomo`).
+
+### Matomo setup for performance
+
+We setup matomo for performance (our websites requires it) with two main points:
+
+* it does not process archives on incoming requests but instead on a cron job (see `confs/matomo/cront.d/matomo-archive`).
+  See also [official doc](https://matomo.org/faq/on-premise/how-to-set-up-auto-archiving-of-your-reports/).
+* on incoming update request (on a tracked website being visited), it does not immediately updates the database but goes in redis instead,
+  then a cron job process redis entries every minute (see `confs/matomo/cront.d/matomo-tracking`).
+  See also [official doc](https://matomo.org/faq/on-premise/how-to-configure-matomo-to-handle-unexpected-peak-in-traffic/)
+* `MariaDB` has been tuned a bit toward performance (using more memory)
+
+Both cronjob logs to `/var/log/matomo`.
+
+See also [2023-12-11 Matomo down](./reports/2023-12-11-matomo-down.md)
+
+### Prometheus exporters
+
+Nginx prometheus exporter is installed and needs the stub_status site, which exposes nginx metrics.
+
+Mysql server prometheus exporter is installed (for MariaDB), its `/etc/default` configuration file sets the connection string.
+The corresponding mysql user had to be manually created (instrutions in the config file).

docs/prod-architecture.md

@@ -18,7 +18,7 @@ Of course we should never commit any secret (but few files contains secrets).
 The server has the [infrastructure git repository](https://github.com/openfoodfacts/openfoodfacts-infrastructure)
 cloned in `/opt/openfoodfacts-infrastructure` by root user.
 
-We symlink files in `/etc` to this directory, this enables updating the server with a simple `git pull` (but this still may needs services reload or restart).
+We symlink files in `/etc` to this directory, this enables updating the server with a simple `git pull` (but this still may needs services reload or restart, or a systemctl daemon-reload, etc.).
 
 `/etc/pve` is a bit particular for it is only an image of Proxmox configuration (and not a real filesystem),
 thus we cannot use symlink there. Instead, we copy the file contents to the git repository after each modifications.

docs/reports/2023-12-11-matomo-down.md

@@ -0,0 +1,203 @@
+# 2023-12-11 Matomo down
+
+Matomo is not responding on 2023-12-11 morning.
+We either got "gateway timeout" or "An error occured"
+
+The incident went unnoticed, which shows that Matomo is not monitored.
+
+
+## Resolution
+
+### Monitoring matomo
+
+I added blackbox monitoring for matomo, following https://matomo.org/faq/how-to/faq_20278/
+
+* monitoring https://analytics.openfoodfacts.org/index.php?module=Login for admin access
+* https://analytics.openfoodfacts.org/matomo.php for trackers access
+
+see [openfoodfacts-monitoring commit:db8f911ce](https://github.com/openfoodfacts/openfoodfacts-monitoring/commit/db8f911ce48de278752232229b63b85e780e1a4a)
+
+
+### Restarting fpm-php
+
+A `systemctl restart php7.3-fpm.service` do put the service back online but for a short time only… after some minutes it's unresponsive again.
+
+### Looking at processes
+
+`ps -elf|grep php` shows us that all php thread seems occupied at doing `"archive:core"` it correspond to the task launched by `/etc/cron.d/matomo-archive`.
+
+This is something really needed to get statistics, matomo rely on that.
+
+
+### Adding more child processes to fpm-php
+
+I edit `/etc/php/7.3/fpm/pool.d/www.conf` to add more child processes:  (`pm.max_children` scaled up from 5 to `30`).
+
+This does not resolve the problem. The bottleneck seems to be the database !
+
+All fpm-php processes are stucked polling while lot of processes are still doing archive:core
+
+### Zombies
+
+At some point I realize that may php processes are indeed zombies ! (parent pid is 1).
+
+So I decide to reboot the container.
+
+I also decided to avoid archive:core running multiple time in parallel.
+So I modified `/etc/cron.d/matomo-archive` to be:
+```
+MAILTO="[email protected]"
+# only start if no process is already running
+5 * * * * www-data ps -elf|grep "core:archive" || /usr/bin/php /var/www/html/matomo/console core:archive --url=http://analytics.openfoodfacts.org/ >> /var/log/matomo/matomo-archive.log 2>>/var/log/matomo/matomo-archive-err.log
+```
+
+### Configured infrastructure git on the matomo container
+
+One good way of keeping track of specific work done in an install is to track files the infrastructure git repository.
+
+To clone the openfoodfacts-infrastructure project on matomo container:
+
+#. I generated a key `ssh-keygen -t ed25519 -C "[email protected]"`
+#. I added the public key top openfoodfacts-infrastructure as a deploy key
+#. cloned it in /opt/openfoodfacts-infrastructure (as on other servers)
+
+### Configure with git
+
+After moving the files to the repository,
+```bash
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/mysql/mariadb.conf.d/50-server.cnf /etc/mysql/mariadb.conf.d/50-server.cnf
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/cron.d/matomo-archive /etc/cron.d/matomo-archive
+```
+
+I also added systemd email on failure for various services.
+
+```bash
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/systemd/[email protected] /etc/systemd/system
+systemctl daemon-reload
+# test
+systemctl start [email protected]
+```
+
+
+### Re-configure email
+
+Email was not well configured, did see that by doing a `systemctl start [email protected]` to test it.
+
+* `apt install bsd-mailx`
+* `dpkg-reconfigure postfix` following instructions on [how to setup email](../mail.md#postfix-configuration)
+
+
+### Optimize Matomo for performances
+
+Going to [parameters, system, general parameter, archival parameters](https://analytics.openfoodfacts.org/index.php?module=CoreAdminHome&action=generalSettings&idSite=1&period=day&date=yesterday&activated=&mode=admin&type=&show=#), I verified that Matomo is configured that we do not archive reports from the browser, and that we only do it once an hour.
+
+(see https://matomo.org/faq/on-premise/how-to-set-up-auto-archiving-of-your-reports/)
+
+But finally after I observed untrack failures (process hung so cron not relaunching…), I decided to use systemd timer + service.
+
+```bash
+unlink  /etc/cron.d/matomo-archive
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/systemd/matomo-archive.* /etc/systemd/system
+systemctl daemon-reload
+systemctl enable matomo-archive.timer
+systemctl start  matomo-archive.timer
+```
+
+### Optimize MariaDB performances
+
+Added some RAM and CPUs
+
+https://vitux.com/tune-and-optimize-mysql-mariadb/
+https://mariadb.com/kb/en/configuring-mariadb-for-optimal-performance/
+
+Relevant files are in this repository and linked in `/etc`
+
+### Using Redis
+
+I first installed redis and phpredis on the server:
+`sudo apt install redis php-redis`
+with a restart of php ``
+
+Following https://matomo.org/faq/on-premise/how-to-configure-matomo-to-handle-unexpected-peak-in-traffic/
+
+- Got QueuedTracking in the [Platform / market place in our Matomo instance](https://analytics.openfoodfacts.org/index.php?module=Marketplace&action=overview&idSite=1&period=day&date=yesterday&activated=&mode=admin&type=&show=)
+- Activate the QueuedTracking plugin in “Matomo Administration > Plugins “
+- Under “Matomo Administration > System > General Settings > QueuedTracking”
+- Select Backend = Redis, Select Number of Queue workers = 1
+- Select Number of requests that are processed in one batch = 100
+- Disable the setting Process during tracking request
+
+Then I setup a cronjob that executes the command ./console queuedtracking:process every minute:
+
+```bash
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/cron.d/matomo-tracking /etc/cron.d/matomo-tracking
+```
+
+But finally after I observed untrack failures (process hung so cron not relaunching…), I decided to use systemd timer + service.
+
+
+```bash
+unlink  /etc/cron.d/matomo-tracking
+ln -s /opt/openfoodfacts-infrastructure/confs/matomo/systemd/matomo-tracking.* /etc/systemd/system
+systemctl daemon-reload
+systemctl enable matomo-tracking.timer
+systemctl start  matomo-tracking.timer
+```
+
+
+### Install Prometheus nginx exporter
+
+While at it, it may help.
+
+`sudo apt install prometheus-nginx-exporter`
+
+For nginx exporter we need to expose /stub_status on port 8080 (see [doc](https://github.com/nginxinc/nginx-prometheus-exporter)) https://nginx.org/en/docs/http/ngx_http_stub_status_module.html#stub_status
+
+I created the `stub_status` conf for that (linked from this repository).
+
+Check status: `systemctl status prometheus-nginx-exporter`
+
+Test it locally:
+```bash
+curl http://127.0.0.1:9113/metrics
+curl http://10.1.0.107:9113/
+```
+
+Tested it from ovh2, it does not work but from ovh1 it works !
+
+
+### Install Prometheus mysqld exporter
+
+`sudo apt install prometheus-mysqld-exporter`
+
+I then had to edit `default/prometheus-mysqld-exporter` (linked from this repository) to add:
+```conf
+DATA_SOURCE_NAME="prometheus:nopassword@unix(/run/mysqld/mysqld.sock)/"
+```
+
+And create the user in the database using `mysql` command:
+```sql
+CREATE USER IF NOT EXISTS 'prometheus'@'localhost' IDENTIFIED VIA unix_socket;
+GRANT PROCESS, REPLICATION CLIENT, SELECT ON *.* TO 'prometheus'@'localhost';
+```
+
+Then restart the service:
+```bash
+systemctl start prometheus-mysqld-exporter
+```
+
+Testing:
+
+```bash
+systemctl status prometheus-mysqld-exporter
+curl "http://127.0.0.1:9104/metrics"
+curl "http://10.1.0.107:9104/metrics"
+```
+
+### Install Prometheus redis exporter
+
+It's not present in current distribution so I didn't install it !
+
+### Monitoring
+
+Now to be able to monitor those, I added them to monitoring configuration.