From 9e6c2ee09a1bc63dd1bf660f29704b8fd5c6599c Mon Sep 17 00:00:00 2001 From: Michael Fellinger Date: Thu, 25 Jul 2024 20:42:57 +0200 Subject: [PATCH] SRE update 2024-07-25 --- blog/2024-07-25-sre.md | 114 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 114 insertions(+) create mode 100644 blog/2024-07-25-sre.md diff --git a/blog/2024-07-25-sre.md b/blog/2024-07-25-sre.md new file mode 100644 index 0000000000..279276dc72 --- /dev/null +++ b/blog/2024-07-25-sre.md @@ -0,0 +1,114 @@ +--- +title: SRE Team Update +slug: 2024-07-25-sre +authors: manveru +tags: [sre] +hide_table_of_contents: false +--- + +## High level summary + +The SRE team continues work on cardano environment improvements and general maintenance. + +Some notable recent changes, updates or improvements include: + +* Our new baseline version of Cardano Node is `9.1.0` and all environments have been updated. + The main change from node `9.0.0` is that node `9.1.0` requires a Conway genesis file at startup, where the genesis file was optional in node `9.0.0`. + +* Sanchonet had another respin for node `9.0.0` with new Conway genesis parameters and has since been upgraded to node `9.1.0`. + +* The `cardano-monitoring` cluster received a lot of documentation and improvements and now also serves as the home for `devx-ci` metrics. + +* Hydra CI performance was improved once again by changes to our custom Nix evaluator. + We also found further ways to improve the resource usage of waiting for IFDs. + +### Cardano Airgap + +A new project that provides a completely airgapped environment for constitution members to sign proposals and transactions. +It consists of an image for USB sticks and ensures all private data is stored securely with strong encryption. + +We'll keep it updated as required with the latest Cardano versions. + +[cardano-airgap](https://github.com/Intersectmbo/cardano-airgap) + +### Cardano Parts + +* cardano-node and cardano-node-ng are now at version `9.0.0` +* cardano-db-sync and cardano-db-sync-ng are now at version `13.3.0.0` +* Several NixOS modules and recipes have been fixed and improved. +* Bump dependency of `capkgs` for node, db-sync, mithril, and cardano-wallet updates. +* Update profile-cardano-db-sync-snapshots for schema 13.3 docs and with script edge case fixes +* Update profile-cardano-node-group to use a SIGINT instead of SIGTERM for systemd stop +* Update profile-common to deploy `atd` service +* Update template recipe `dbsync-prep` to match faucet script defaults +* Update template recipe `update-ips` to fix a nushell breaking change +* Update `.envrc` with a newer direnv version and allows for symlinks on `.envrc.local` and `~/.age/credentials` + +[PR#44](https://github.com/input-output-hk/cardano-parts/pull/44) + +### Cardano Playground + +* All networks are now running cardano-node `9.1.0` in preparation of the Chang hard-fork. +* Also upgraded db-sync to `13.3.0.0` +* Added the `cardano-ipfs` module and a derivation for `pinata-go-cli` that is used to store and distribute documents that can be referenced on chain. +* Some updates to the [Cardano Operations Book](https://book.play.dev.cardano.org/) about: + * UseLedgerPeerAfter updates + * Sanchonet respins configs + * Dbsync `EnableFutureGenesis` flag +* Add a block header block producer readiness test +* Respin of sanchonet for node `9.0.0`, then upgraded to `9.1.0` +* Tune webserver size and Varnish RAM to improve caching efficiency +* Updates `govtool` module for multi-nginx module compatibility +* Updates `update-ips` recipe for nushell breaking change in nixpkgs 24.05 +* Updates `direnv` version, allow symlinks config files used by `direnv` +* Update `.envrc` with a newer direnv version, allow symlinks on direnv used config files + +[PR#28](https://github.com/input-output-hk/cardano-playground/pull/28) + +### Cardano Mainnet + +* Upgraded Cardano Node to `9.0.0` +* Upgraded Cardano DB Sync to `13.3.0.0` +* Bump `capkgs` dependency +* Investigate bootstrap missingBlock error and deploy fixes for it. +* Update scripts to be compatible with latest `nushell` version + +[PR#17](https://github.com/input-output-hk/cardano-mainnet/pull/17) + +### Cardano Monitoring + +* Write comprehensive documentation for all the Nix code, as well as detailed instructions for usage and deployment. +* Overhaul most Just tasks to bring them more in sync with the other repositories +* Upgrade all machines to NixOS 24.05 +* Upgrade `auth-keys-hub` to prevent lockout in case `SOPS` is unable to decrypt +* Fix `SOPS` decryption failure on boot because of missing network. +* Limit bootloader entries to 5 since the `/boot` partition is tiny +* Additionally add fallback SSH keys for emergency use + +[PR#1](https://github.com/input-output-hk/cardano-monitoring/pull/1) + +### IOHK Nix + +* Update ledger peers to be after a more recent epoch boundary to improve bootstrapping and fix a private chain p2p delayed sync config issue. +* Update sanchonet conway-genesis for respin + * DRep voting thresholds both need to be 65% + * Set `govActionLifetime` above the guardrail because of the short epochs. + * Set `minCommitteeSize` to 5 (from recommended 7) because only 5 ICC members were able to provide keys for the respin. + +[PR#584](https://github.com/input-output-hk/iohk-nix/pull/584) +[PR#585](https://github.com/input-output-hk/iohk-nix/pull/585) + +### Cardano Ops + +* tweak the stop timeout and change Cardano Node killsignal to `SIGINT` for clean restarts. + +[Diff](https://github.com/input-output-hk/cardano-ops/compare/7dd0502...07a4721) + +### CAPkgs + +Added following packages: + +* cardano-node `9.0.0` and `9.1.0` +* For cardano-db-sync releases of `sancho-5.1.0` and `13.3.0.0` +* For mithril `2428.0` and `mithril-unstable` +* For cardano-wallet `v2024-07-19`