core/tracing: state journal wrapper #30441

s1na · 2024-09-16T08:46:58Z

Here we add some more changes for the live tracing API:

OnSystemCallStartV2 is introduced with VMContext as parameter.
GetCodeHash is added to the state interface
The new WrapWithJournal construction helps with tracking EVM reverts in the tracer.

s1na · 2024-09-16T11:05:26Z

core/tracing/hooks.go

+	NonceReadHook = func(addr common.Address, nonce uint64)
+
+	// CodeReadHook is called when EVM reads the code of an account.
+	CodeReadHook = func(addr common.Address, code []byte)


Open question: should we add codeHash here to be consistent with OnCodeChange?

core/tracing/CHANGELOG.md

s1na · 2024-10-08T16:21:14Z

Ah seems like the journal has a crasher:

revisions: [{0 2} {1 4} {2 4} {3 4} {4 6} {5 6} {6 9} {7 11} {8 12} {9 18} {10 18} {11 20} {12 22} {13 24} {14 24} {18 27}]
panic: revision id 17 cannot be reverted

goroutine 10470 [running]:
github.com/ethereum/go-ethereum/core/tracing.(*journal).revertToSnapshot(0xc050c41c70, 0x11, 0xc0570360e0)
        github.com/ethereum/go-ethereum/core/tracing/journal.go:170 +0x185
github.com/ethereum/go-ethereum/core/tracing.(*journal).OnExit(0xc050c41c70, 0x0, {0xc13a87fe30, 0x64, 0x64}, 0x48dc9, {0x203f680, 0xc018bcc978}, 0x1)
        github.com/ethereum/go-ethereum/core/tracing/journal.go:206 +0x6f
github.com/ethereum/go-ethereum/core/vm.(*EVM).captureEnd(0xc13a9e0780?, 0x0, 0x12e208, 0xe543f, {0xc13a87fe30, 0x64, 0x64}, {0x203da40, 0x2e05070})

core/tracing/journal_test.go

karalabe · 2024-10-10T09:11:27Z

core/tracing/CHANGELOG.md

+
+### New methods
+
+- `OnReorg(reverted []*types.Block)`: This hook is called when a reorg is detected. The `reverted` slice contains the blocks that are no longer part of the canonical chain.


Here types block is very very heavy. You should at most pass headers and allow chain access to pull the blocks on demand (chain access in someconstructor, ha)

On second thought what is the issue? it is a slice so passed by reference and the memory can be freed as soon as OnReorg processing is done.

Ugh, this is annoying. So reorg in the blockchain at some point in the past used to collect blocks. Turned out that sometimes it became insanely heavy and we've switched so it operates on headers. I guess later someone refactored it back to operate on blocks again. This is an issue when you do setHead or any similar operation; of even if finality fails for a while and you have blocks reorging back and forth. It's very very bad to pull all the block in from disk IMO.

CC @holiman @rjl493456442 ?

I agree. I don't particularly recall switching from headers to blocks....

core/tracing/hooks.go

s1na · 2024-10-10T09:23:23Z

core/tracing/hooks.go

@@ -194,6 +221,30 @@ type Hooks struct {
 	OnCodeChange    CodeChangeHook
 	OnStorageChange StorageChangeHook
 	OnLog           LogHook
+	// State reads
+	OnBalanceRead  BalanceReadHook
+	OnNonceRead    NonceReadHook


Question from triage: how exactly is OnNonceRead used?

holiman · 2024-12-10T11:52:41Z

core/tracing/hooks.go

+}
+
+// Copy creates a new Hooks instance with all implemented hooks copied from the original.
+func (h *Hooks) Copy() *Hooks {


Why is this method needed? It's not obvious to me what the side-effects are. Typically, the hooks might be closures, and the closures are still referenced as if they were not copied.

func TestHooks(t *testing.T) { counter := 0 a := &Hooks{ OnClose: func() { counter++ }, } a.OnClose() t.Logf("counter is %d", counter) a.Copy().OnClose() t.Logf("counter is %d", counter) }

outputs:

hooks_test.go:13: counter is 1 hooks_test.go:15: counter is 2

I'm curious why you'd ever need to use this Copy method.

Is it because you want to copy all, but not have to specify all manually?

Typically, the hooks might be closures, and the closures are still referenced as if they were not copied.

Right you are correct I had not foreseen that. After thinking a bit, it feels like for my use-case that is fine. Essentially the clone will replace some of the methods to add some pre-processing logic. The rest are supposed to execute as in the original tracer.

I have un-exported the Copy method to avoid people to shoot themselves in the foot and added a comment to clarify this point.

Edit: right what I want is to add the journal in front of the tracer and process some of the hooks first before proxying back to tracer. And this without having to iterate the list of all hooks which I find very error-prone. I have already had to fix bugs because of missing some hook in there.

holiman · 2024-12-10T11:52:50Z

core/tracing/hooks.go

@@ -172,6 +173,9 @@ type (

 	// LogHook is called when a log is emitted.
 	LogHook = func(log *types.Log)
+
+	// BlockHashReadHook is called when EVM reads the blockhash of a block.
+	BlockHashReadHook = func(blockNumber uint64, hash common.Hash)


Why is this needed?

The use-case is to have access to the headers of hashes that are accessed by the EVM. Alternative would be if we added a GetHeaderByHash method somewhere. But getting the hash from OnOpcode is also tricky since the hash will be put on the stack after OnOpcode is invoked.

holiman · 2024-12-10T11:57:46Z

core/tracing/journal.go

+type journal struct {
+	entries     []entry
+	hooks       *Hooks
+	lastCreator *common.Address // Account that initiated the last contract creation
+
+	validRevisions []revision
+	nextRevisionId int
+	revIds         []int
+}


the linearJournal in my PR #30660 is IMO a better base to start from. It does away with validrevisions and revIds, instead it just as a list of indexes, revisions, which point to an entry.

type linearJournal struct { entries []journalEntry // Current changes tracked by the linearJournal dirties map[common.Address]int // Dirty accounts and the number of changes revisions []int // sequence of indexes to points in time designating snapshots }

I have looked at #30660 and agree it is a better way to do journaling for tracers. The key point for me there is that there will be only 1 revert hook emitted as opposed to one for each change to a state element.

Given that #30660 seems to be still in flux I like to wait on it to be merged and implement it for tracers in a future PR as an improvement.

These points were discussed at standup:

It looks like we will need to change the model a bit with the set-based journal as it operates on accounts, requiring also a new hook like OnAccountReverted and the inconsistencies there around emitting state changes on the field level and the reverts being on the account level.

The point was raised by @holiman that we are exposing a behaviour to tracers that will be hard to revert. The behaviour in question is the reverse of every state change (i.e. if Balance: A->B->C, we emit Balance: C->B->A on revert instead of just Balance: C->A).

On the second point I'd like to add my perspective: This journal is simply a wrapper around the tracers. We are not changing tracing interface semantics at all. Users can copy this file and run it themselves right now. And we are exposing every state modification, and I believe the reverse of it is the same.

It looks like we will need to change the model a bit with the set-based journal

I don't get it. The two PRs, the two journals are unrelated. You have opted to copy-paste the legacy linear journal-implementation. I think the new linear journal-implementation is better/simpler.

You could also have chosen to copy-paste the set-based journal-implementation. I don't care which you choose, really, but I don't see any point in picking one now and switching later. If you want another, pick that one from the get-go ?

Users can copy this file and run it themselves right now.

They can't perform WrapWithJournal from "user-space," can they? Isn't that what makes this a big "blessed" ?

They can't perform WrapWithJournal from "user-space," can they? Isn't that what makes this a big "blessed" ?

Right exact copy wouldn't work. They'd have to implement WrapWithJournal locally, but it's totally possible.

I don't care which you choose, really, but I don't see any point in picking one now and switching later.

I think the current journal is good, it is consistent with the existing interface, and it has been running in geth for quite a while.

holiman · 2024-12-19T18:30:11Z

core/tracing/journal.go

+	validRevisions []revision
+	nextRevisionId int
+	revIds         []int


I don't see why you need to track this. Why not just maintain a list of entries, and you hand out the id which is the current length of the entries?

core/tracing/journal.go

holiman · 2024-12-19T18:34:34Z

core/tracing/journal.go

+type journal struct {
+	entries     []entry
+	hooks       *Hooks
+	lastCreator *common.Address // Account that initiated the last contract creation


This looks like a hack. I don't see how this can be accurately updated going forward and backward along the entries. I mean, an inner scope will overwrite the outer lastCreator, and when the inner scope is reverted, the lastCreator will not be set back correctly.

Or if we're inside a creation, and inside the constructor we call ripemd to calculate a signature: we lost lastCreator.

fjl · 2025-01-28T13:43:45Z

core/tracing/hooks.go

+			dstValue.Field(i).Set(field)
+		}
+	}
+	return copied


Is this not equivalent to

copied := *h return &copied

gballet · 2025-02-05T09:55:44Z

core/tracing/hooks.go

@@ -163,6 +171,9 @@ type (
 	// NonceChangeHook is called when the nonce of an account changes.
 	NonceChangeHook = func(addr common.Address, prev, new uint64)

+	// NonceChangeHookV2 is called when the nonce of an account changes.
+	NonceChangeHookV2 = func(addr common.Address, prev, new uint64, reason NonceChangeReason)


The V2 isn't helpful, we should use NonceChangeHokWithReason. Function names should be explict in what they do.

We have chosen this naming scheme explicitly. The intent is communicating which version of the hook is the newest one. When we introduce a new hook version, the old one becomes deprecated and will eventually be unsupported. Also, if we were to introduce another revision of this hook, would it be called NonceChangeHookWithReasonAndBellsAndWhistles? The *Vx naming scheme avoids this problem.

When the old one is removed, you can remove the WithReason part, but in the meantime, you know at a glance why the two methods differ. I don't think there's a big risk of that function being rewritten many times and, therefore, having the names getting longer and longer. But sure, just giving my opinion on what could make the code readable, not going to hold the release for that one 🤷

We cannot rename the function because it is a stable, user-exposed API. If we could just change it, we wouldn't go through the trouble of having multiple versions of the hook.

core/tracing/journal.go

core/tracing/journal_test.go

s1na added 7 commits August 26, 2024 15:45

core/tracing: add vm context to system call hook

8659e68

core/tracing: add GetCodeHash to statedb interface

b4e0174

core/tracing: emit state change events for journal reverts

f670a7f

core/tracing: add hook for reverted out blocks

cf873c3

log selfdestructs balance revert

365b715

Add state read hooks

aac4024

add tracing journal

dbe5f83

s1na requested review from karalabe, holiman and rjl493456442 as code owners September 16, 2024 08:46

s1na commented Sep 16, 2024

View reviewed changes

s1na added 8 commits September 16, 2024 13:26

update changelog

b87c4fe

fix indent

702a42f

add block hash read hook

c915bed

resolve merge conflict

838fc25

fix code and nonce param order

1cc58cf

update test

3c58155

pass-through non-journaled hooks

501f302

missed two hooks

1a64297

maoueh reviewed Oct 5, 2024

View reviewed changes

core/tracing/CHANGELOG.md Show resolved Hide resolved

s1na added 2 commits October 8, 2024 20:09

fix journal cur rev Id

1862333

add note on balanceChangeRevert reason

6650000

s1na added the status:triage label Oct 9, 2024

refactor WrapWithJournal to use reflection

d9de74e

karalabe reviewed Oct 10, 2024

View reviewed changes

core/tracing/journal_test.go Show resolved Hide resolved

karalabe reviewed Oct 10, 2024

View reviewed changes

holiman reviewed Oct 10, 2024

View reviewed changes

core/tracing/hooks.go Show resolved Hide resolved

s1na commented Oct 10, 2024

View reviewed changes

core/tracing/hooks.go Show resolved Hide resolved

s1na commented Oct 10, 2024

View reviewed changes

holiman reviewed Dec 10, 2024

View reviewed changes

s1na added 2 commits December 10, 2024 18:12

un-expose hooks copy

9cae376

Merge branch 'master' into tracing/v1.1

bf51dde

holiman reviewed Dec 19, 2024

View reviewed changes

s1na added the status:marinating label Dec 19, 2024

fjl removed the status:marinating label Dec 20, 2024

fjl self-assigned this Jan 21, 2025

fjl modified the milestones: 1.14.13, 1.15.0 Jan 23, 2025

fjl reviewed Jan 28, 2025

View reviewed changes

s1na and others added 4 commits February 4, 2025 16:30

refactor copy

459c50f

Merge branch 'master' into tracing/v1.1

831524a

Use nonce reason in journal

6f5e74b

resolve conflict

bca2e2c

s1na requested review from lightclient and MariusVanDerWijden as code owners February 4, 2025 17:21

s1na and others added 8 commits February 4, 2025 18:23

resolve conflict

8a2230e

fix test

59a5022

core/tracing: add logging in journal test

4ba05e9

core/tracing: simplify journal implementation

2795c0e

core/tracing: further improve journal tests

51720dc

core/tracing: remove Hooks.copy

4787f31

core/tracing: add note about WrapWithJournal in comments

8a44029

core/tracing: add a package-level doc comment

eaacae4

gballet reviewed Feb 5, 2025

View reviewed changes

core/tracing/journal.go Outdated Show resolved Hide resolved

gballet reviewed Feb 5, 2025

View reviewed changes

core/tracing/journal_test.go Outdated Show resolved Hide resolved

license year

93432fc

fjl approved these changes Feb 5, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

core/tracing: state journal wrapper #30441

core/tracing: state journal wrapper #30441

s1na commented Sep 16, 2024 •

edited by fjl

Loading

s1na Sep 16, 2024

s1na commented Oct 8, 2024

karalabe Oct 10, 2024

s1na Oct 14, 2024

karalabe Oct 14, 2024

holiman Oct 14, 2024

s1na Oct 10, 2024

holiman Dec 10, 2024

holiman Dec 10, 2024

s1na Dec 10, 2024 •

edited

Loading

holiman Dec 10, 2024

s1na Dec 10, 2024

holiman Dec 10, 2024

s1na Dec 16, 2024

s1na Dec 19, 2024

holiman Dec 19, 2024

holiman Dec 19, 2024

s1na Dec 19, 2024

holiman Dec 19, 2024

holiman Dec 19, 2024

holiman Dec 19, 2024

fjl Jan 28, 2025

gballet Feb 5, 2025

fjl Feb 5, 2025 •

edited

Loading

gballet Feb 5, 2025

fjl Feb 5, 2025


		### New methods

		- `OnReorg(reverted []*types.Block)`: This hook is called when a reorg is detected. The `reverted` slice contains the blocks that are no longer part of the canonical chain.

core/tracing: state journal wrapper #30441

Are you sure you want to change the base?

core/tracing: state journal wrapper #30441

Conversation

s1na commented Sep 16, 2024 • edited by fjl Loading

Choose a reason for hiding this comment

s1na commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1na Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fjl Feb 5, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

s1na commented Sep 16, 2024 •

edited by fjl

Loading

s1na Dec 10, 2024 •

edited

Loading

fjl Feb 5, 2025 •

edited

Loading