Include base -> overlay key-values migration logic #199

jsign · 2023-04-21T21:27:31Z

This PR contains a decent implementation of the key-value migration leveraging the new optimized APIs from a tentative go-verkle PR.

This code might not be final, and there're some ideas to optimize a bit further... but I'm opening this first version which should be pretty decent. I want first to land a complete version that does the work in a non-naive way.

After we have the prototype working and have some numbers, we can consider some tweaks I've in mind in case they're needed.

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign · 2023-04-21T21:28:19Z

core/state_processor.go

@@ -109,17 +115,21 @@ func (p *StateProcessor) Process(block *types.Block, statedb *state.StateDB, cfg
 				return nil, nil, 0, err
 			}

-			// move N=500 accounts into the verkle tree, starting with the
+			const maxMovedCount = 500


Only this offtopic nit.

jsign · 2023-04-21T21:29:58Z

core/state_processor.go

+			// mkv will be assiting in the collection of up to maxMovedCount key values to be migrated to the VKT.
+			// It has internal caches to do efficient MPT->VKT key calculations, which will be discarded after
+			// this function.
+			mkv := &keyValueMigrator{}


This keyValueMigrator is a component that encapsulates a bit capturing the relevant walked key-values. I did this to keep the walk clean and let the component do the work. I'll comment on it later.

jsign · 2023-04-21T21:30:50Z

core/state_processor.go

+			if err := mkv.migrateCollectedKeyValues(tt.Overlay()); err != nil {
+				return nil, nil, 0, fmt.Errorf("could not migrate key values: %w", err)
+			}


After we finish collecting them, we let the component do the go-verkle use of new APIS: batch create leaf nodes, and insert into the tree. More about this below.

jsign · 2023-04-21T21:34:02Z

core/state_processor.go

+func (kvm *keyValueMigrator) migrateCollectedKeyValues(tree *trie.VerkleTrie) error {
+	// Transform the map into a slice.
+	nodeValues := make([]verkle.BatchNewLeafNodeData, 0, len(kvm.vktLeafData))
+	for _, vld := range kvm.vktLeafData {
+		nodeValues = append(nodeValues, *vld)
+	}
+
+	// Create all leaves in batch mode so we can optimize cryptography operations.
+	newLeaves := verkle.BatchNewLeafNode(nodeValues)
+
+	// Insert into the tree.
+	if err := tree.InsertMigratedLeaves(newLeaves); err != nil {
+		return fmt.Errorf("failed to insert migrated leaves: %w", err)
+	}
+
+	return nil
+}


This is what we called in L173. Basically, we use the new APIs to efficiently create leaves and insert into the tree.

jsign · 2023-04-21T21:34:33Z

go.mod

-	github.com/gballet/go-verkle v0.0.0-20230414192453-2838510d5ee0
+	github.com/gballet/go-verkle v0.0.0-20230420192103-fe912821ad59


Pointing to last commit of ethereum/go-verkle#343

is there any reason that PR is still in draft, btw?

Mostly to have some extra information after seeing how would fit "this side" of the story.
Opened for review; not sure if you want to have an extra look, but from my side it's fine to review & merge.

After that, I can update go.mod to the latest master.

jsign · 2023-04-21T21:35:05Z

trie/transition.go

+func (t *TransitionTrie) Overlay() *VerkleTrie {
+	return t.overlay
+}


Needed for core/state_processor.go L173. Double check!

is it needed? We could also just implement all the required functions on top of TransitionTrie? Not sure which one is the best yet, I'm just not a big fan of exposing too much of the internal structure.

I guess it's simpler to do for now, could you just please add a TODO in a comment so that we remember to check that when we merge the final PR ?

The "problem" with making it an API in TransitionTrie is that the methods are only Verkle related, and not conceptually generic. Still, it might be a good idea.. it's also true that "opening" the box is indirectly giving access to all the APIs.

I'll add the TODO to remember about this, yes.

jsign · 2023-04-21T21:36:30Z

trie/verkle.go

+func (trie *VerkleTrie) InsertMigratedLeaves(leaves []verkle.LeafNode) error {
+	return trie.root.(*verkle.InternalNode).InsertMigratedLeaves(leaves, func(hash []byte) ([]byte, error) {
+		return trie.db.diskdb.Get(hash)
+	})
+}


Just a trampoline to use the new API to avoid magic castings and skipping layers.

For now should be fine... whenever we reconsider ethereum/go-verkle#314 maybe this method will go away if it can be as fast as the new API.

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign · 2023-04-22T22:54:26Z

core/state_processor.go

-					// @jsign: do your magic here adding the account at `addr
+					mkv.addAccount(addr, acc)

 					// Store the account code if present
 					if !bytes.Equal(acc.CodeHash, emptyCode) {
 						code := rawdb.ReadCode(statedb.Database().DiskDB(), common.BytesToHash(acc.CodeHash))
 						chunks := trie.ChunkifyCode(code)

-						// @jsign: do your magic here with the code chunks
+						mkv.addAccountCode(addr, uint64(len(code)), chunks)
 					}

 					if !bytes.Equal(acc.Root, emptyRoot[:]) {
-						for ; stIt.Next() && count < 500; count++ {
+						for ; stIt.Next() && count < maxMovedCount; count++ {
 							slotnr := rawdb.ReadPreimage(statedb.Database().DiskDB(), stIt.Hash())

-							// @jsign do your magic here with extra slots
+							mkv.addStorageSlot(addr, slotnr, stIt.Slot())
 						}


TL;DR: we simply "collect" the key values and let mkv capture them.

jsign · 2023-04-22T22:54:57Z

core/state_processor.go

+// keyValueMigrator is a helper struct that collects key-values from the base tree.
+// The walk is done in account order, so **we assume** the APIs hold this invariant. This is
+// useful to be smart about caching banderwagon.Points to make VKT key calculations faster.
+type keyValueMigrator struct {
+	currAddr      []byte
+	currAddrPoint *verkle.Point
+
+	vktLeafData map[string]*verkle.BatchNewLeafNodeData
+}


There shouldn't be much surprises here.
The TL;DR is that since we walk the tree by address, we keep the current address point cached and only update if a new account is detected. This makes sense considering how the caller uses this component.

We keep collecting stuff in the vktLeafData which we use at the end to batch create leaf nodes and insert into the tree.

The method implementations shouldn't be surprising. They're doing something similar to what we did in the full tree conversion. Note that I'm sharing as soon as I finished writing the code, so might be something missing, so please double check.

I tried to use as many defined constants as possible to avoid magic indexes or sizes, etc.

(As mentioned in the PR description, there're some potential changes to make this faster. But I think this is reasonable and quite clear/clean to have a decent starting point; we can see later if we need to push limits a bit more that can justify a more complex version).

gballet

Nice, I have left a few comments but I'm happy to merge whenever you are happy with it.

gballet · 2023-04-24T11:28:13Z

go.mod

-	github.com/gballet/go-verkle v0.0.0-20230414192453-2838510d5ee0
+	github.com/gballet/go-verkle v0.0.0-20230420192103-fe912821ad59


is there any reason that PR is still in draft, btw?

gballet · 2023-04-24T11:34:50Z

trie/transition.go

+func (t *TransitionTrie) Overlay() *VerkleTrie {
+	return t.overlay
+}


is it needed? We could also just implement all the required functions on top of TransitionTrie? Not sure which one is the best yet, I'm just not a big fan of exposing too much of the internal structure.

gballet · 2023-04-24T11:35:36Z

trie/transition.go

+func (t *TransitionTrie) Overlay() *VerkleTrie {
+	return t.overlay
+}


I guess it's simpler to do for now, could you just please add a TODO in a comment so that we remember to check that when we merge the final PR ?

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign · 2023-04-24T12:33:53Z

Nice, I have left a few comments but I'm happy to merge whenever you are happy with it.

@gballet, opened for review ethereum/go-verkle#343 if you want to give it another look, and we can merge it. After that, I can update go.mod to latest master, and from my side it's also fine to merge this if sounds good to you.

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign · 2023-04-24T15:19:34Z

@gballet, updated go.mod with the merged PR; all ready.

Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification

Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB

* overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]>

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code move UpdateContractCode to updateStateObject fix: generated rlp decoders fix: activate Shanghai in test add withdrawals to witness fix the first half of verkle test fix: open an empty storage trie after transition activate verkle on IsVerkle, not IsCancun deactivate snapshot for tests enable proof of stake in verkle test save last MPT root for transition remove unnecessary snapshot Cap in flush fix test: include EIP-3860 implement missing odrDatabase function fix incorrect equality condition is HasAccount fixes to replay ~500 blocks fix to replay more blocks fix preimage issue in conversion code cleanup: remove a lot of TODOs & friends more code cleanup fix: alignment of values whose len is < 32 New access-witness module (#235) Use plain addresses/slot numbers instead of hashing them in the witness remove unused map param in MakeVerkleMultiProof (#261) core/state: rewrite a new optimized keyValueMigrator (#256) * trie/utils: add helper to calculate code tree indices * core/state: rewrite optimized version of keyValueMigrator Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: remove uint256 allocs (#257) Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> core: move overlay conversion code to its own file (#266) add pre-pbss rebase branches to CI (#270) * add post-pbss rebase branches to CI * fix go version in CI * fix linter issues * upgrade go version to 1.21.1 to avoid github deploy error quell zero-tree message port kaustinen code on top of shapella-rebased branch activate proof generation on fork + remove code dups use go-verkle's post-state API to verify proofs (#262) use prague as the verkle activation fork (#263) upgrade to latest go-ipa activate verkle transition in "miner" (#265) fix: do not force cancunTime upon verkle activation workaround: do not use root translation in replay workaround: deactivate overlay transition for now fixes from trying to get the devnet to work (#267) this line was left out from the previous commit upgrade to go-verkle with fixed newvalue serialization fix: ensure point cache isn't nil in copy (#268) fix: dependency cycle in tests (#269) upgrade to latest go-verkle fix: write trie preimage data to db (#274) fix: zero-root in produced block + sync (#275) upgrade go-ipa fix build fix typo include review feedback add switch to add proofs to blocks (#278) add fee recipient to witness (#279) touch all fields in withdrawal account header (#277) fix: add ProofInBlocks to chain config (#280) remove StateDB as an extra param to Prepare (#281) fix: need commitment in tests (#282) fix linter message fix: a couple of CI issues fix more CI complaints achieving perfection workaround: disable check for root presence (#283) fix: activate verkle at genesis (#284) fix: ensure read-only values are resolved in post trie (#285) upgrade deps to get proof generation absence/presence bugfix fix: make sure AccessWitness isn't nil when calling RPC methods (#287) fix: incorrect access copy (#288) fix: return serialized root in state root, not its mapping to a scalar field (#289) add workaround to use --override.prague (#292) Shanghai time workaround (#294) * add workaround to use --override.prague * fix typo in comments * fix: enable overrides at genesis time remove pre-state tree warmup (#293) * remove pre-state tree warmup Signed-off-by: Ignacio Hagopian <[email protected]> * update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]>

Squash the main verkle PR ahead of rebase don't call Bytes() in GetTreeKey (#137) trie: avoid endianness conversion in GetTreeKey (#140) * trie/utils: add concrete expected value in trie key generation test Signed-off-by: Ignacio Hagopian <[email protected]> * mod: update to latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: avoid endianness conversions Signed-off-by: Ignacio Hagopian <[email protected]> * apply review changes & update to official go-verkle version Signed-off-by: Ignacio Hagopian <[email protected]> Signed-off-by: Ignacio Hagopian <[email protected]> upgrade go-verkle to CoW version and get TestProcessVerkle to build (#138) updating ci to use self-hosted machine (#143) fix: storage offset in non-header group + reuse of value buffer (#145) dedup call to ChunkifyCode, same as replay branch (#156) * dedup call to ChunkifyCode, same as replay branch * fix some linter issues fix code offset in tree update (#157) fix REVERT in state processor test execution (#158) * fix code offset in tree update * fix REVERT in test execution save on key hashing: lump code size update with first code chunk group (#159) fix code chunk key calculation and storage key calculation (#161) * fix codeKey calculation * Remove * fix storageOffset * fix the fix to the fix to the offset fix * Remove copy/pasted, unused code in test * fix linter --------- Co-authored-by: Guillaume Ballet <[email protected]> fix: infinite loop when calling extcodecopy on empty code (#151) upgrade to latest go-verkle fix: only update code in the tree if it's dirty (#174) fix: read-touch the code size and Keccak of the origin (#175) List of changes for converting a sepolia database (#182) * naive conversion rebased on top of beverly hills * changes for the sepolia shadow fork conversion * fixes to please the linter * fixes to please the linter Unified point cache (#180) * Unified point cache * Use cache for Try*Account * alter Trie interface to use caching for slots (#181) * alter Trie interface to use caching for slots * fix: use a lock to protect the point cache (#185) * use fastest non-master go-verkle version & pull trie/Verkle.go changes to use new api (#184) * mod: update to fastest go-verkle version today Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: use new batch serialization api Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: TryDelete signature in unit tests --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> trie/utils: fix potential overflow (#191) * trie/utils: fix potential overflow Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: receive storage key as a byte slice Signed-off-by: Ignacio Hagopian <[email protected]> * revert formatter changes Signed-off-by: Ignacio Hagopian <[email protected]> * trie/utils: fix mod 256 Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> trie/utils: fix incorrect bigint assignment (#193) Signed-off-by: Ignacio Hagopian <[email protected]> upgrade precomp link to fix CI fix: add missing code size&keccak leaves in empty accounts (#192) fixes to use the latest go-verkle@master (#197) * fixes to use the latest go-verkle@master * linter fixes * linter fixes for tests * fix: use jsign's go-verkle fix refactor: remove unused (*StateDB).GetXLittleEndian methods (#204) fix gas accounting issue in state_processor_test.go (#207) update go-verkle not to use StatelessNode anymore (#206) * update go-verkle not to use StatelessNode anymore * update go-verkle to latest refactor: move verkle gas accounting to its own block in TransitionDB (#208) fix a panic in deserializeVerkleProof if GetProofItems returns a nil ProofElements use the cachingDB instead of a custom VerkleDB (#209) * use the cachingDB instead of a custom VerkleDB * fix stack trace in LES remove holiman from CODEOWNERS as he gets too many emails read from tree in state object if the snapshot is nil (#205) add missing error checks for the root node type (#214) implement OpenStorageTrie for verkle trees (#210) * implement OpenStorageTrie for verkle trees * add a few comments for future maintenance * fix linter issue fix: copy balance leaf to new buffer in TryGetAccount (#217) implement some heretofore unimplemented iterator methods (#219) params: move verkle params to their own file (#228) fix: proper number of chunk evals (#215) overlay transition (#244) * overlay transition Fix some bugs identified in the code review Co-authored-by: Ignacio Hagopian <[email protected]> Include base -> overlay key-values migration logic (#199) * mod: add go-verkle version with key-value migration new apis Signed-off-by: Ignacio Hagopian <[email protected]> * core/stateprocessor: use constant for max number of migrated key-values Signed-off-by: Ignacio Hagopian <[email protected]> * core: add base->overlay key-values migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * core: fix some compiler errors Signed-off-by: Ignacio Hagopian <[email protected]> * trie: consider removing transition trie api in the future Signed-off-by: Ignacio Hagopian <[email protected]> * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> fix some unit tests errors get convresion block from file fix compilation issues fix initialization issue in migrator fix: changes needed to run the first 28 blocks important sutff: fix the banner fix: use nonce instead of balance in nonce leaf (#202) fixes for performing the overlay transition (#203) * fixes for performing the overlay transition * fixes for the full replay * fix: deletion-and-recreation of EoA * fixes to replay 2M+ blocks * upgrade to go-verkle@master * fix: proper number of chunk evals * rewrite conversion loop to fix known issues changes to make replay work with the overlay method (#216) * fixes for performing the overlay transition fixes for the full replay fix: deletion-and-recreation of EoA fixes to replay 2M+ blocks upgrade to go-verkle@master fix: proper number of chunk evals rewrite conversion loop to fix known issues changes to make replay work with the overlay method fixes to replay 2M+ blocks update to latest go-verkle@master * use a PBSS-like scheme for internal nodes (#221) * use a PBSS-like scheme for internal nodes * a couple of fixes coming from debugging replay * fix: use an error to notify the transition tree that a deleted account was found in the overlay tree (#222) * fixes for pbss replay (#227) * fixes for pbss replay * trie/verkle: use capped batch size (#229) * trie/verkle: use capped batch size Signed-off-by: Ignacio Hagopian <[email protected]> * trie/verkle: avoid path variable allocation per db.Put Signed-off-by: Ignacio Hagopian <[email protected]> * don't keep more than 32 state root conversions in RAM (#230) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Guillaume Ballet <[email protected]> * cleanup some code * mod: update go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> * re-enable snapshot (#231) * re-enable cancun block / snapshot (#226) * clear storage conversion key upon translating account (#234) * clear storage conversion key upon translating account * mod: use latest go-verkle Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> * fix: self-deadlock with translated root map mutex (#236) * return compressed commitment as root commitment (#237) --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> fix first panic in *TransitionTrie.Copy() upgrade go-verkle to latest master mod: update go-verkle (#239) Signed-off-by: Ignacio Hagopian <[email protected]> core: print state root every 100 blocks (#240) Signed-off-by: Ignacio Hagopian <[email protected]> fix: only Commit the account trie (#242) fixes to get TestProcessVerkle to work with the overlay branch (#238) * fixes to get TestProcessVerkle to work with the overlay branch * fix all panics in verkle state processor test * fix proof verification move transition management to cachingDB * fix: mark the verkle transition as started if it's ended without being started * fix the verkle state processing test * fix linter errors * Add a function to clear verkle params for replay * fix: handle TransitionTrie in OpenStorageTrie * fix linter issue * fix the deleted account error (#247) * code cleanup (#248) * fix: don't error on a missing conversion.txt (#249) * Overlay Tree preimages exporting and usage (#246) * export overlay preimages tool Signed-off-by: Ignacio Hagopian <[email protected]> * use preimages flat file in overlay tree migration logic Signed-off-by: Ignacio Hagopian <[email protected]> * cmd/geth: add --roothash to overlay tree preimage exporting command Signed-off-by: Ignacio Hagopian <[email protected]> * cleanup Signed-off-by: Ignacio Hagopian <[email protected]> * review feedback Signed-off-by: Ignacio Hagopian <[email protected]> --------- Signed-off-by: Ignacio Hagopian <[email protected]> * fix: reduce the PR footprint (#250) * fix: don't fail when preimages.bin is missing (#251) * fix: don't fail when preimages.bin is missing * fix: don't open the preimages file when outside of transition --------- Signed-off-by: Ignacio Hagopian <[email protected]> Co-authored-by: Ignacio Hagopian <[email protected]> review changes remove replay-specific code

jsign added 3 commits April 21, 2023 13:51

mod: add go-verkle version with key-value migration new apis

5aea81b

Signed-off-by: Ignacio Hagopian <[email protected]>

core/stateprocessor: use constant for max number of migrated key-values

4896b69

Signed-off-by: Ignacio Hagopian <[email protected]>

core: add base->overlay key-values migration logic

db97f14

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign commented Apr 21, 2023

View reviewed changes

jsign marked this pull request as ready for review April 21, 2023 21:36

jsign requested a review from holiman as a code owner April 21, 2023 21:36

core: fix some compiler errors

c9a85f6

Signed-off-by: Ignacio Hagopian <[email protected]>

jsign commented Apr 22, 2023

View reviewed changes

gballet reviewed Apr 24, 2023

View reviewed changes

trie: consider removing transition trie api in the future

b505401

Signed-off-by: Ignacio Hagopian <[email protected]>

mod: use latest go-verkle

55ca4c3

Signed-off-by: Ignacio Hagopian <[email protected]>

gballet merged commit ce6c94a into gballet:overlay-implementation Apr 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include base -> overlay key-values migration logic #199

Include base -> overlay key-values migration logic #199

jsign commented Apr 21, 2023

jsign Apr 21, 2023

jsign Apr 21, 2023

jsign Apr 21, 2023

jsign Apr 21, 2023

jsign Apr 21, 2023

gballet Apr 24, 2023

jsign Apr 24, 2023

jsign Apr 21, 2023

gballet Apr 24, 2023

gballet Apr 24, 2023

jsign Apr 24, 2023

jsign Apr 24, 2023

jsign Apr 21, 2023

jsign Apr 22, 2023

jsign Apr 22, 2023

gballet left a comment

gballet Apr 24, 2023

gballet Apr 24, 2023

gballet Apr 24, 2023

jsign commented Apr 24, 2023

jsign commented Apr 24, 2023

		github.com/gballet/go-verkle v0.0.0-20230414192453-2838510d5ee0
		github.com/gballet/go-verkle v0.0.0-20230420192103-fe912821ad59

Include base -> overlay key-values migration logic #199

Include base -> overlay key-values migration logic #199

Conversation

jsign commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gballet left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsign commented Apr 24, 2023

jsign commented Apr 24, 2023