RaspberryMatic 3.59.6.20210911

[jens-maus]

This is release 3.59.6.20210911 of RaspberryMatic which is a maintenance release with bugfixes and feature changes including the following changes:

_{Hier klicken für deutschsprachige 🇩🇪🇦🇹🇨🇭 Version des ChangeLogs/Diskussionsbeitrag}

🚧 Changes:

_{For all changes, see the full commit log.}

CCU/HomeMatic service changes:

• partly reintroduced the use of an own 64bit version of libNRJavaSerial because the hmip-copro-update.jar tool does not come with an on aarch64 version yet. So until eQ3 provides an updated hmip-copro-update.jar tool we ship RaspberryMatic with an own copy of libNRJavaSerialv8.so again. This fixes an issue where RaspberryMatic was not able to update the firmware of a RPI-RF-MOD/HM-MOD-RPI-PCB rf module (Error bei RPI-RF-MOD - Firmware Update, Firmware wird nicht aktualisiert #1387).
• reworked /bin/triggerAlarm.tcl to allow to generate alarm variables as pure internal alarm variables which will hide them from the standard WebUI, but make it still usable as full-fledged alarm variables. This finally allowed to rework all our triggerAlarm.tcl uses throughout different places (monitrc, cronBackup.sh, updateDCVars.tcl, etc.) so that for every WatchDog alarm a dedicated alarm variable with the name scheme "WatchDog: XXXXX" is generated where XXXXX is specific to the individual alarm. This should make the whole WatchDog alarm mechanisms more flexible (Kategorisierte WatchDog-Alarme #1321).
• implemented a new monit-based watchdog check which in case of using a RaspberryPi4 regularly checks if an onboard USB3 port (the blue ones) is used together with a rf module (RPI-RF-MOD/HM-MOD-RPI-PCB) directly connected to the GPIO bus and outputs a WebUI alarm if identified. This is motivated by the common problem that a RaspberryPi4 is known to produce significant rf interference which could negatively affect the rf communication and thus a user have to be warned that the combination of a Pi4+USB+GPIO is not a good idea. However, for use cases where a user wants to disable that regular watchdog check a file /etc/config/rpi4usb3CheckDisabled can be generated (cf. https://homematic-forum.de/forum/viewtopic.php?f=65&t=69421).
• integrated a fix for /bin/ha-proxy.js which moves the \u0027 string out of the character class (where it was wrong). This should integrate a fix for getting the general file upload via fileupload.ccc running correctly in an HA add-on environment (Fehler bei Installation von Zusatzsoftware im Home Assistant Raspberrymatic Add On #1368).
• updated generic_raw_uart package to latest 1.20 version.
• removed invalid https -> http redirection in NEOserver addon config page index.html which is anyway not required since our addon patches make it fully https compatible already (NEO Addon Konfigurationsseite wird permanent "reloaded" bei HTTPS Umleitung #1382).
• instead of checking for the process id file of ha-proxy we check for HM_HAPROXY_SRC in /var/hm_mode so that the firewall is setup correctly in the HA addon use-case also in the early phase. In addition, we use NET_ADMIN in config.json of the HA addon so that iptables can be used within the HA add-on docker container.
• moved the CustomStoragePath initialization to the S62HMServer startup script so that using a network path should also work when specified in the /etc/config/CustomStoragePath config option (CustomStoragePath funktioniert nicht (für Diagramme) wenn der Pfad über rc.local gemountet wird #1377).

WebUI changes:

• added new 0134-WebUI-Add-ShowInternalSysvars WebUI patch which adds functionality that internal system variables can be listed like normal system variables, but will carry a grey background color to allow to more easily distinguish them from the normal ones (add 0134-WebUI-Add-ShowInternalSysvars patch #1400, @jp112sdl).
• optimized the 0072-WebUI-Fix-hm_autoconf WebUI patch to use ID_GW_SYSALARM and ID_GW_SYSSERVICE rather than fixed integer IDs.
• added a new 0133-WebUI-Fix-AlarmSysVarInternal WebUI patch which combines the EnumIDs() and EnumEnabledInternalIDs() of the oUser.UserTempViewIDs() usage when displaying alarm variables. This is necessary because the UserTempViewIDs are IseIdArrays for which EnumIDs() does unfortunately not return sysvars which are flagged as "internal". This prevented the alarm display from showing internal alarm variables.
• added a new 0132-WebUI-Fix-UnitDisplay WebUI bugfix patch which will make sure that between a value and its unit a whitespace is enforced by using the URL code  . This should make the value+unit display in the WebUI more consistent.
• added another improvement for the 0119-WebUI-Fix-ProofAndSetValue WebUI patch which fixes broken floating point / ProofAndSetValue use on the COND_TX_THRESHOLD_LO and COND_TX_THRESHOLD_HI values for HmIP-based powermeter devices like the HmIP-PSM. (cf. https://homematic-forum.de/forum/viewtopic.php?f=65&p=677137#p677136).
• added new 0131-WebUI-ShowAddonUpdatesOnStartpage WebUI patch which will inform a user on the startpage if there are any pending add-on updates available ("Zusatzsoftware-Update verfügbar" auf Startseite anzeigen #1384, add patch for #1384 #1385, @jp112sdl).
• improved 0119-WebUI-Fix-ProofAndSetValue WebUI patch which should fix another issue with the ProofAndSetValue() function use for certain device types accepting floating point numbers with a certain precision (ProofAndSetValue - Anpassung Funktionsaufruf in ic_deviceparameters.cgi #1383, @Baxxy13).
• added new 0130-WebUI-Fix-ChannelSysVarPrefix WebUI bugfix patch which fixes the incorrect prefix of "Acoustic/Optic signal" for system variables associated with such device channels (Kanalzugeordnete SysVars erhalten einen unnötigen Präfix in WebUI-Programmen (im DANN - Teil) #1376).
• reworked 0124-WebUI-Fix-ShowMultipleSpacesInSysVar to use a CSS style statements to preserve each white space when showing the content of a system variable rather than replacing them via   statements. Using CSS style methods should prevent inline html code from getting mixed up by the   statements (0124-WebUI-Fix-ShowMultipleSpacesInSysVar break inline html use in system variables #1375).
• reworked 0042-WebUI-HeatingControl-BatteryValveState WebUI patch to only display a single valve level ui element for HmIP heating devices supporting the HEATING_CONTROL_HMIP.LEVEL datapoint (Ventil-Öffnungsgrad zweimal beim HmIP-eTRV-C-2 #1374).
• adding new 0127-WebUI-Sort-FirmwareOverview.patch WebUI patch which sorts the list of device firmware updates by device-type and device-address (adding new fix 0127 for sort the list of device firmware updates #1370, @Maik2208).
• adding new 0129-WebUI-delete-debuggingoutput.patch WebUI patch which removes invalid debug output in a heating group causing a empty popup to be displayed (adding new fix 0129 to delete an debuggingoutput by changing a heatgroup #1371, @Maik2208).
• added new 0128-WebUI-Fixing-Label-ControlPanel.patch WebUI patch which fixes the formatting of the english text on updating access points (added new fix 0128 for correcting "Update Access Point"-Label in system control #1369, @Maik2208).

Operating system changes:

• Added a patch to fix a u-boot issue with RaspberryPi hardware causing troubles when enumerating USB devices (required when booting from USB) (cf. Fix U-Boot USB initialization failure (#1203, #1490) home-assistant/operating-system#1529).
• increased vm.min_free_kbytes to 16384 for the RaspberryPi platform to increase stability in memory critical situations (cf. Increase minimal free memory for Raspberry Pis home-assistant/operating-system#1525).
• modified all ln -sf XXX command uses with ln -sfn XXX to make sure symlinks are properly deferred so that they can be replaced rather than the link target. This should fix certain issues in replacing existing symlink (CustomStoragePath funktioniert nicht (für Diagramme) wenn der Pfad über rc.local gemountet wird #1377).
• added a check and execution of /usr/local/etc/rc.prelocal right in the startup phase of S55InitAddons which will be executed right before all major homematic services are started but after the network should be up. This should solved the need for having a user definable startup script where users could mount external drives or executed utilities right before the major homematic services are started (CustomStoragePath funktioniert nicht (für Diagramme) wenn der Pfad über rc.local gemountet wird #1377, https://github.com/jens-maus/RaspberryMatic/wiki/Experten-Features#eigene-aktionen-w%C3%A4hrend-des-bootvorgangs).
• use CONFIG_OF_BOARD for all RaspberryPi platforms so that the board's own device tree overlay is used instead. We had to, however, add CONFIG_REQUIRE_SERIAL_CONSOLE=n to the uboot_defconfig files as otherwise u-boot ends up in an endless loop in case the rpi-rf-mod.dtbo changes the compatible entry of the uart0 entry and thus would leave no serial console for u-boot (Nightly RaspberryMatic-3.59.6.20210811-7ac604 --> kein Booten eines Pi3B / Pi3B+ #1378).
• fixed xe-daemon startup which was not putting the daemon to background and thus stalled the bootup (RaspberryMatic VM startet mit 3.59.6.20210807 nicht mehr (XCP-ng / Xen Server) #1381).
• in case no HTTP_CONTENT_LENGTH env variable exists we using CONTENT_LENGTH instead in the file upload cgi scripts we use in the recovery system as well as in the main system. This fixes a problem where in case of running RaspberryMatic as a HA add-on the ingress routed WebUI did not allow to upload files, thus resulted in not being able to install CCU addons (Fehler bei Installation von Zusatzsoftware im Home Assistant Raspberrymatic Add On #1368).
• added more Intel WiFi firmware (IWLWIFI) files to support more different intelNUC hardware models.
• integrated a potential u-boot fix when a USB keyboard is connected and issues are observed upon bootup. (cf. Fix buffer overflow when initialize USB keyboard in U-Boot (#1538) home-assistant/operating-system#1541)
• reduce the -t XX timeout option when calling udhcpc for obtaining an IP address so that it only waits up to ~1 min for an IP address and then continue in background. This should significantly increase the bootup times in cases where no DHCP is present or does not have a free IP address left.
• updated u-boot for rpi0 and rpi2 to latest 2021.04 as boot works now with the latest version due to the recent fixes with CONFIG_BOARD support in uboot_defconfig.
• updated Mediola NEO Server to latest 2.7.1
• updated RaspberryPi kernel to 1.20210831 (5.10.60) and firmware/bootloader to 1.20210805.
• updated upstream Linux kernel to 5.10.62 for ova, intelnuc, odroid and tinkerboard platform.
• updated buildroot to latest stable 2021.08 version.

👪 Contributors (alphabetically):

• @Baxxy13
• @Maik2208
• @jens-maus
• @jp112sdl

📝 Support:

For support on installation and help please visit the following web pages:

RaspberryMatic - Documentation 🇺🇸
RaspberryMatic - Discussions 🇺🇸
RaspberryMatic - Dokumentation 🇩🇪
RaspberryMatic - Forum 🇩🇪

📦 Download:

The following installation archives can be downloaded for different hardware platforms. To verify their integrity a sha256 checksum is provided as well. You can either upload these files using the WebUI-based update mechanism or unarchive them to e.g. flash the included *.img files on a fresh installation media (e.g. microSD card):

• RaspberryPi4 Model B, RaspberryPi Compute Module 4, RaspberryPi 400 – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-rpi4.zip
  SHA256: eacd40beef2715b53adc81662dccb349313df04ff1daf9bd5d645e1a335eb360}

• CCU3, ELV-Charly, RaspberryPi3 Model B+, RaspberryPi3 Model B, RaspberryPi3 Model A+, RaspberryPi Compute Module 3, RaspberryPi Compute Module 3 lite – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-rpi3.zip
  SHA256: 5c00e586610774a5b7bf22e1f71bcee37d96ae333e596b758f34c2c777c54ff3
  📦 RaspberryMatic-3.59.6.20210911-ccu3.tgz (only for initial CCU3 Firmware -> RaspberryMatic Upgrade)
  SHA256: 40255fa990d0d05dc33fe3ab599e6c25651461c2df9fb7491b3d496daffb7db2}

• RaspberryPi2 Model B – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-rpi2.zip
  SHA256: c9780a593db75d787ca8e037b3d648ab2f7279413aa1e24707878c854cd1f10e}

• RaspberryPi Zero W, RaspberryPi Zero, RaspberryPi Compute Module 1, RaspberryPi1 (A+/B+) – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-rpi0.zip
  SHA256: 2e05f1b677388c3dc58b66ccf452db97554a3c17dc83218a713e99393a5a3a74}

• TinkerBoard S, TinkerBoard – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-tinkerboard.zip
  SHA256: e34baeca56186981b6d2f51b9432325059e3d82e379c437f26a6319ae895c374}

• ODROID-N2/N2+/C4/C2 – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-odroid-n2.zip
  SHA256: 9384eed5733ffde8f1f7e845ca8c3dd55217c7709c48e5f86685f0d481ff0db3
  📦 RaspberryMatic-3.59.6.20210911-odroid-c4.zip
  SHA256: 16202b13dfc0accbe298e7bc5c175d94de1f62dda16ce70871ec9fe4de26d4b8
  📦 RaspberryMatic-3.59.6.20210911-odroid-c2.zip
  SHA256: e72b6f54df8b5312c2c79acc8332d32ba36f77089fd9fdc34278edca5927e9c3}

• Intel NUC – (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-intelnuc.zip
  SHA256: 99592e6e164848d6b58bce2c678fd35748f0ed2d0a10e09bcda671fea58de209}

• Open Virtual Appliance (OVA) – (ESXi, Proxmox, VirtualBox, Synology, QNAP, Workstation Player, QEmu, UNRAID, HyperV):
  _{📦 RaspberryMatic-3.59.6.20210911-ova.zip
  SHA256: b8e36a5a0b627bf29d1f82466a4178437586401c3e674557a9e9abd9043e8e33
  📦 RaspberryMatic-3.59.6.20210911.ova (only for initial OVA installation)
  SHA256: e981f1eccee062f5d05b8f93c0381f877f35ad1f276ac7606912cee57d73307f}

• Docker / Open Container Initiative (OCI) – virtual appliance (installation):
  _{📦 RaspberryMatic-3.59.6.20210911-oci_amd64.tgz (amd64/x86_64)
  SHA256: 58f846fea3462a60ba82242d98a9952b75689505e50140a92711a6e45a27944b
  📦 RaspberryMatic-3.59.6.20210911-oci_arm64.tgz (arm64/aarch64)
  SHA256: 67cb2bc066007886b2bea53e2a1c989c84d0d4b7b89be5b5bfe38d34be45b2b4
  📦 RaspberryMatic-3.59.6.20210911-oci_arm.tgz (arm/armhf)
  SHA256: f7d477a09da40f9fb4a369fe0011ed933d2894b2a5d704cba96bcf543c7487e0}

• Kubernetes / K8s – virtual appliance:
  _{see install documentation}

• Home Assistant Add-on – virtual appliance:
  _{see install documentation}

---

This discussion was created from the release RaspberryMatic 3.59.6.20210911.