Starting sequencer after fork upgrade transaction, the starting block is still on the prior fork ID

[Sharonbc01]

On a fork ID upgrade, once the upgrade transaction is finalized you can restart the sequencer. Once started, check logs and ensure new blocks are generated with new fork ID. During our upgrade we have observed the new block showing the prior fork ID. As a work around we have then taken one of the RPC datadirs (that were synced till the halt and are currently stopped) and replaced it to become new sequencer.

Why is the sequencer not seeing the new fork ID?

• Does it need more time after the upgrade transaction is finalized?
• Is it related to the fact that when the sequencer is halted the RPC does not sync

This issue needs investigation by Gateway FM and resolution along with clear guidance on the steps to follow when re starting the sequencer after the upgrade transaction is finalized

cc: @ToniRamirezM @rafael-genes

[hexoscott]

The fix for this is available in zkevm branch now. Before we promote it to the stable branches it would be good to verify that all is working as expected in an integration network. Maybe Thiago/Xavi can help here?

[xavier-romero]

I'm working on that, will update here at some point !

[xavier-romero]

As for the sequencer, it looks like certainly it started ok and processed the forkid upgrade, immediately generating new blocks for the new forkid.

However, when starting the RPC, it didn't like what it was receiving through DS.

I must admit I didn't pay much attention to CDKnode status, which for some reason wasn't sequencing (and ofc neither verifiyng to the tip), so that may be related. Tomorrow I'll repeat the whole process but the right way, so not upgrading forkid until everything is verified.

[xavier-romero]

Tried yesterday with zkevm branch and today again with image v2.61.3-RC1, same result.

I Halt Sequencer on batch 21, and it halts ok.

I keep waiting for virtual and verified batch number endpoints to return 20, but that NEVER happens.

cast rpc --rpc-url $(kurtosis port print cdk sequencer001 sequencer8123) zkevm_virtualBatchNumber
cast rpc --rpc-url $(kurtosis port print cdk sequencer001 sequencer8123) zkevm_verifiedBatchNumber
cast rpc --rpc-url $(kurtosis port print cdk rpc001 rpc8123) zkevm_virtualBatchNumber
cast rpc --rpc-url $(kurtosis port print cdk rpc001 rpc8123) zkevm_verifiedBatchNumber

Both Sequencer and RPC report 0x10 as virtual and 0x9 as verified, which turns to be WRONG.

I check on L1 contracts:
cast call $ROLLUP_MAN "rollupIDToRollupData(uint32)(address,uint64,address,uint64,bytes32,uint64,uint64,uint64,uint64,uint64,uint64,uint8)" 1
That reports 20 as last sequenced and verified.
With kurtosis L1 finality is just matter of seconds, however, even after hours Erigon keeps reporting the wrong numbers.

If I ignore these numbers and proceed with the upgrade, when I resume sequencer+rpc, the sequencer produces blocks in the new forkid, but RPC reports a state root missmatch, as reported in the previous comment.

[hexoscott]

Interesting, is the RPC going through the stage loop at all? Or does it keep reporting that it's waiting for data from the stream? This is when the network is halted.