High-level summary
During the past two weeks, the consensus team started documenting the
+implementation of the UTxO HD feature and continued developing tests for it. As
+part of our work on UTxO HD, we improved the Haskell support for LMDB. We also
+spent time working on the LSM tree prototype, and designed a parameter tuning
+algorithm for it. Regarding our work on Genesis, our investigation of the
+"plateaus" pointed at the TICKF
slowdown on era boundaries as culprit. This
+led us to developing a caching strategy that will not only remove the
+aforementioned "plateaus", but can help alleviating the growing block production
+delay on epoch switch. We also helped reviewing the block forge credential
+hotswap feature, which is intended for use in the adoption of P2P.
We also worked on paying technical debt and fostering collaboration. In
+particular, we improved the io-sim
framework, which is crucial for testing and
+simulating Cardano components. We also removed thunks that appeared on era
+translations, and improved our diffusion pipelining feature. We are working on a
+presentation for explaining Praos and Genesis.
High-level status report
- Finish the UTxO HD prototype: in progress.
- We added documentation for this feature.
- We developed the second version of the mempool tests.
- We fixed benchmarks that were inflating the speedup we observed in the
+anti-diff implementation of sequences of differences. Speedups are now in the
+range of [3.33, 4.75], which remain significant.
- We continued improving Haskell LMDB support.
- We finished implementing a "parameter tuning algorithm" for the LSM tree
+prototype. This enables us to run experiments to check the correctness of
+the algorithm.
- Genesis: in progress.
- Work investigating the "plateaus" in the
ChainSync
jumping prototype
+pointed to the TICKF
slowdown on era boundaries as culprit.
- Tech debt:
- We improved the capabilities of our
io-sim
library, which is crucial for
+testing and simulating Cardano components. - We removed thunks from epoch translations in the ledger.
- We added Linux CI support for
lmdb-simple
. - We got pending diffusion pipelining improvements merged.
- Fostering collaboration:
- We are working on a explanation of Praos and Genesis protocols.
- Support:
- Investigation of CSJ "plateaus" led us to developing a caching strategy for
+
TICKF
that will not only remove these "plateaus", but can help alleviating
+the growing block production delay on epoch switch. - We reviewed the block forge credential hotswapping feature which is intended
+for use in the adoption of P2P.
Workstreams
Finish the UTxO HD prototype
We merged PR #4060, which adds a report documenting the UTxO HD
+feature, and puts emphasis in explaining how the mempool works in combination
+with UTxO HD.
We opened a draft PR with the second iteration of the property tests for the
+mempool (#4076).
We fixed the Arbitrary
instances for keys and values in DiffSeq
benchmarks
+(#4143). The problem was that we were testing with mostly small
+values, which artificially boosted the performance gains we saw on benhcmarks.
+Speedups are now in the range of [3.33, 4.75] across the different
+configurations.
Backing store property tests
We focused on incorporating feedback on the monadic cursor API PR (#1).
+This required us to make small tweaks to quickcheck-lockstep
to test the new
+API. We also updated the backing store property tests to use the new version of
+the monadic cursor API.
LSM tree implementation
We worked on the LSM tree prototype. In particular: finished implementing a
+"parameter tuning algorithm" that adapts the LSM tree design based on factors
+like:
- workload
- machine specs,
- and characteristics of the data being stored.
We are now running experiments to gather results and cross-reference them with
+existing experimental results from the LSM tree paper to see if the algorithm is
+working correctly.
Benchmarking the CSJ prototype
We focused on investigating the "plateaus" in the ChainSync
tip, which turned
+out to be due to the TICKF
bug which we previously were only aware of in the
+context of the long forging times near epoch boundaries. For the most drastic
+patch by @nfrisby to speed up TICKF
, full sync is speeding up by 7%.
The following plot shows that by caching the TICKF
the ChainSync
tip and the
+VolatileDB
tip progress at the same rate.
The plot below shows the speedup observed by caching the TICKF
rule wrt the
+baseline.
Technical debt
After addressing the PR comments, we merged PR #16, which implements
+the MonadCatch
instance for STM
. This extends the capability of our io-sim
+library, which is crucial for testing and simulating Cardano components PR #16
+closed #1461. This new feature was published as version 0.4.0.0
+of io-sim
.
We continued with our work fixing the NoThunk
errors required for enabling
+nightly tests, with the help of TVarInvariant
checks in strict-stm
and
+nothunks
libraries. We proposed fixes in cardano-ledger
that took care of
+thunks that appeared in era translations (#3143). The fixes will be
+integrated back into consensus when cardano-ledger
approves and publish the
+changes introduced in #3143
.
We added CI support for lmdb-simple
(#2). We currently test the build on
+a Linux environment only.
We got pending diffusion pipelining PRs (#3857, #3860,
+#3856) merged, after rebasing and addressing feedback.
Fostering collaboration
@nfrisby finished a visualisation tool and outlined scripts for the Praos and
+Genesis explanation presentations. The idea is to produce a video that gives an
+overview of these protocols.
Support
We started working on caching the computation of the TICKF
rule
+(#4054), since this was blocking our benchmarking work for
+Genesis. In addition, this issue has the Cardano community quite
+concerned, so we are hoping the work done in caching the
+computation of the TICKF
rule can help alleviating the growing block
+production delay on epoch switch.
We reviewed the block forge credential hotswapping PR #3800 from the
+networking team, which is intended for use in the adoption of P2P.