SRE update 2024-07-25

blog/2024-07-25-sre.md

@@ -0,0 +1,114 @@
+---
+title: SRE Team Update
+slug: 2024-07-25-sre
+authors: manveru
+tags: [sre]
+hide_table_of_contents: false
+---
+
+## High level summary
+
+The SRE team continues work on cardano environment improvements and general maintenance.
+
+Some notable recent changes, updates or improvements include:
+
+* Our new baseline version of Cardano Node is `9.1.0` and all environments have been updated.
+  The main change from node `9.0.0` is that node `9.1.0` requires a Conway genesis file at startup, where the genesis file was optional in node `9.0.0`.
+
+* Sanchonet had another respin for node `9.0.0` with new Conway genesis parameters and has since been upgraded to node `9.1.0`.
+
+* The `cardano-monitoring` cluster received a lot of documentation and improvements and now also serves as the home for `devx-ci` metrics.
+
+* Hydra CI performance was improved once again by changes to our custom Nix evaluator.
+  We also found further ways to improve the resource usage of waiting for IFDs.
+
+### Cardano Airgap
+
+A new project that provides a completely airgapped environment for constitution members to sign proposals and transactions.
+It consists of an image for USB sticks and ensures all private data is stored securely with strong encryption.
+
+We'll keep it updated as required with the latest Cardano versions.
+
+[cardano-airgap](https://github.com/Intersectmbo/cardano-airgap)
+
+### Cardano Parts
+
+* cardano-node and cardano-node-ng are now at version `9.0.0`
+* cardano-db-sync and cardano-db-sync-ng are now at version `13.3.0.0`
+* Several NixOS modules and recipes have been fixed and improved.
+* Bump dependency of `capkgs` for node, db-sync, mithril, and cardano-wallet updates.
+* Update profile-cardano-db-sync-snapshots for schema 13.3 docs and with script edge case fixes
+* Update profile-cardano-node-group to use a SIGINT instead of SIGTERM for systemd stop
+* Update profile-common to deploy `atd` service
+* Update template recipe `dbsync-prep` to match faucet script defaults
+* Update template recipe `update-ips` to fix a nushell breaking change
+* Update `.envrc` with a newer direnv version and allows for symlinks on `.envrc.local` and `~/.age/credentials`
+
+[PR#44](https://github.com/input-output-hk/cardano-parts/pull/44)
+
+### Cardano Playground
+
+* All networks are now running cardano-node `9.1.0` in preparation of the Chang hard-fork.
+* Also upgraded db-sync to `13.3.0.0`
+* Added the `cardano-ipfs` module and a derivation for `pinata-go-cli` that is used to store and distribute documents that can be referenced on chain.
+* Some updates to the [Cardano Operations Book](https://book.play.dev.cardano.org/) about:
+  * UseLedgerPeerAfter updates
+  * Sanchonet respins configs
+  * Dbsync `EnableFutureGenesis` flag
+* Add a block header block producer readiness test
+* Respin of sanchonet for node `9.0.0`, then upgraded to `9.1.0`
+* Tune webserver size and Varnish RAM to improve caching efficiency
+* Updates `govtool` module for multi-nginx module compatibility
+* Updates `update-ips` recipe for nushell breaking change in nixpkgs 24.05
+* Updates `direnv` version, allow symlinks config files used by `direnv`
+* Update `.envrc` with a newer direnv version, allow symlinks on direnv used config files
+
+[PR#28](https://github.com/input-output-hk/cardano-playground/pull/28)
+
+### Cardano Mainnet
+
+* Upgraded Cardano Node to `9.0.0`
+* Upgraded Cardano DB Sync to `13.3.0.0`
+* Bump `capkgs` dependency
+* Investigate bootstrap missingBlock error and deploy fixes for it.
+* Update scripts to be compatible with latest `nushell` version
+
+[PR#17](https://github.com/input-output-hk/cardano-mainnet/pull/17)
+
+### Cardano Monitoring
+
+* Write comprehensive documentation for all the Nix code, as well as detailed instructions for usage and deployment.
+* Overhaul most Just tasks to bring them more in sync with the other repositories
+* Upgrade all machines to NixOS 24.05
+* Upgrade `auth-keys-hub` to prevent lockout in case `SOPS` is unable to decrypt
+* Fix `SOPS` decryption failure on boot because of missing network.
+* Limit bootloader entries to 5 since the `/boot` partition is tiny
+* Additionally add fallback SSH keys for emergency use
+
+[PR#1](https://github.com/input-output-hk/cardano-monitoring/pull/1)
+
+### IOHK Nix
+
+* Update ledger peers to be after a more recent epoch boundary to improve bootstrapping and fix a private chain p2p delayed sync config issue.
+* Update sanchonet conway-genesis for respin
+  * DRep voting thresholds both need to be 65%
+  * Set `govActionLifetime` above the guardrail because of the short epochs.
+  * Set `minCommitteeSize` to 5 (from recommended 7) because only 5 ICC members were able to provide keys for the respin.
+
+[PR#584](https://github.com/input-output-hk/iohk-nix/pull/584)
+[PR#585](https://github.com/input-output-hk/iohk-nix/pull/585)
+
+### Cardano Ops
+
+* tweak the stop timeout and change Cardano Node killsignal to `SIGINT` for clean restarts.
+
+[Diff](https://github.com/input-output-hk/cardano-ops/compare/7dd0502...07a4721)
+
+### CAPkgs
+
+Added following packages:
+
+* cardano-node `9.0.0` and `9.1.0`
+* For cardano-db-sync releases of `sancho-5.1.0` and `13.3.0.0`
+* For mithril `2428.0` and `mithril-unstable`
+* For cardano-wallet `v2024-07-19`