Docs and changelogs

docs/website/contents/for-developers/utxo-hd/Overview.md

@@ -1,3 +1,28 @@
-# Overview
+# High level overview of UTxO-HD
 
-TODO
+UTxO-HD is an internal rework of the Consensus layer that features a hybrid
+database for Ledger State data. UTxOs are stored in a separate database that
+can be backed by an on-disk database or with an in-memory implementation.
+
+Each of those backends have specific behaviors and implications, so we will
+refer to them individually by `InMemory` and `OnDisk`.
+
+End-users of the `InMemory` backend (the default one) should not appreciate any
+major difference in behavior and performance with respects to a pre-UTxO-HD
+node.
+
+End-users of the `OnDisk` backend will observe a regression in performance. For
+now the `OnDisk` backend is implemented via LMDB and not optimal in terms of
+performance, but we plan on making use of the LSM trees library that Well-Typed
+is developing for a much better performance. In particular operations that need
+UTxOs (applying blocks/transactions) will have the overhead of a trip to the
+disk storage plus some calculations to bring the disk values up to date to the
+tip of the chain.
+
+In exchange for that performance regression, a Cardano node using the `OnDisk`
+backend can run with much more modest memory requirements than a pre-UTxO-HD
+node.
+
+In terms of functionality, both backends are fully functional.
+
+For a more extensive description of UTxO-HD, see [the full documentation](./utxo-hd-in-depth).

docs/website/sidebars.js

@@ -23,7 +23,6 @@ const sidebars = {
       label: 'About Ouroboros',
       items: [
         'about-ouroboros/index',
-        'about-ouroboros/utxo-hd',
         'about-ouroboros/References'
       ]
     }
@@ -60,8 +59,8 @@ const sidebars = {
       label: 'UTxO HD',
       items: [
         'for-developers/utxo-hd/Overview',
+        'for-developers/utxo-hd/utxo-hd-in-depth',
         'for-developers/utxo-hd/future-ledger-hd',
-        'for-developers/utxo-hd/utxo-hd',
       ]
     }
   ]

ouroboros-consensus-cardano/changelog.d/utxo-hd.md

@@ -0,0 +1,6 @@
+### Breaking
+
+- Implement the UTxO-HD feature. See the documentation in [the
+  webpage](https://ouroboros-consensus.cardano.intersectmbo.org/docs/for-developers/utxo-hd/Overview).
+  Fill in the UTxO-HD type families and instances for Shelley based blocks, the
+  Byron block and the Cardano block.

ouroboros-consensus-diffusion/changelog.d/utxo-hd.md

@@ -0,0 +1,12 @@
+### Breaking
+
+- Implement the UTxO-HD feature. See the documentation in [the
+  webpage](https://ouroboros-consensus.cardano.intersectmbo.org/docs/for-developers/utxo-hd/Overview).
+  - `hStateQueryServer` now needs a `ResourceRegistry` to allocate `Forker`s.
+  - `DiskPolicyArgs` was transformed into `SnapshotPolicyArgs`.
+  - `StdRunNodeArgs` got two new fields:
+    - `srnQueryBatchSize` to specify how many entries read on each batch when
+      answering queries.
+    - `srnLdbFlavorArgs` to select the LedgerDB backend.
+  - The forging loop uses a `ReadOnlyForker` to get the ledger state of the
+    block on top of which it should forge and the mempool snapshot.

ouroboros-consensus/changelog.d/utxo-hd.md

@@ -0,0 +1,23 @@
+### Breaking
+
+- Implement the UTxO-HD feature. See the documentation in [the
+  webpage](https://ouroboros-consensus.cardano.intersectmbo.org/docs/for-developers/utxo-hd/Overview).
+  - The `LedgerState` type family changed its kind to `Type -> (Type -> Type) ->
+    Type` to account for `MapKind`s.
+  - `LedgerTables` store a `MapKind` of `TxIn l` to `TxOut l`.
+  - The `HardFork` block's `TxIn` is `CanonicalTxIn` which is stable across
+    eras.
+  - Applying blocks and ticking consume `ValuesMK` and produce `DiffMK`.
+  - The LedgerDB has three implementations: `V1InMemory`, `V1LMDB` and
+    `V2InMemory`. The first one is not intended to be used in production.
+  - The LedgerDB keeps track of differences from blocks and flushes them to the
+    backend.
+  - Snapshots have changed in format, now there are two files in each snapshot
+    instead of one.
+  - To evaluate forks, the LedgerDB exposes the `Forker` and the
+    `ReadOnlyForker` interfaces.
+  - `BlockQuery` is now parametrized by a `QueryFootprint` type.
+  - `HardFork` `BlockQuery` are now processed slightly different than single era
+    `BlockQuery`, see `BlockSupportsHFLedgerQuery`.
+  - The mempool state is now held in a `TMVar` instead of a `TVar` to be able to
+    acquire it, read values from the backend and update it.

ouroboros-consensus/src/ouroboros-consensus/Ouroboros/Consensus/Ledger/Extended.hs

@@ -35,7 +35,9 @@ import           Codec.CBOR.Decoding (Decoder, decodeListLenOf)
 import           Codec.CBOR.Encoding (Encoding, encodeListLen)
 import           Control.Monad.Except
 import           Data.Functor ((<&>))
+#if __GLASGOW_HASKELL__ >= 906
 import           Data.MemPack
+#endif
 import           Data.Proxy
 import           Data.Typeable
 import           GHC.Generics (Generic)

sop-extras/changelog.d/utxo-hd.md

@@ -0,0 +1,4 @@
+### Breaking
+
+- Delete the `K2` combinator.
+- Define `Fn2`.