Unable to sync Lisk Mainnet with the latest version (v1.1.1)

[matjazv]

Describe the bug

After updating the Reth client to the latest version (v1.1.1), the op-node v1.9.5 (as well as the previous version v1.9.4) generates the following logs: Failed to get payload and Cannot seal block, payload ID is unknown. Additionally, Reth stalls at New payload job created id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f.

PR for updating the Reth client for the Lisk network:
LiskHQ/lisk-node#45

Steps to reproduce

CLIENT=reth docker compose up --build --detach

Node logs

OP-NODE:
2024-11-12 14:00:17 t=2024-11-12T13:00:17+0000 lvl=info msg="Received signed execution payload from p2p" id=0x26ed278d4e18b61a3eaca5460b2e3b6b020e143ee376aa08079ae3c3c83d9826:8343813 peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb txs=1
2024-11-12 14:00:17 t=2024-11-12T13:00:17+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x26ed278d4e18b61a3eaca5460b2e3b6b020e143ee376aa08079ae3c3c83d9826:8343813
2024-11-12 14:00:18 t=2024-11-12T13:00:18+0000 lvl=warn msg="Failed to get payload" payload_id=0x039f75b9b517fb98 payload_id=0x039f75b9b517fb98 err="Unknown payload"
2024-11-12 14:00:18 t=2024-11-12T13:00:18+0000 lvl=warn msg="Cannot seal block, payload ID is unknown" payloadID=0x039f75b9b517fb98 payload_time=1714730555 started_time=2024-11-12T13:00:18+0000
2024-11-12 14:00:18 t=2024-11-12T13:00:18+0000 lvl=warn msg="Block sealing job of derived attributes expired, job will be re-attempted." build_id=0x039f75b9b517fb98 timestamp=1714730555 err="failed to seal execution payload (ID: 0x039f75b9b517fb98): Unknown payload"

Reth:
2024-11-12 13:54:25 ts=2024-11-12T12:54:25.172012844Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:54:30 ts=2024-11-12T12:54:30.679607305Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-12 13:54:35 ts=2024-11-12T12:54:35.299266085Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:54:45 ts=2024-11-12T12:54:45.825349548Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:54:55 ts=2024-11-12T12:54:55.681237011Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-12 13:54:57 ts=2024-11-12T12:54:57.207042054Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:55:07 ts=2024-11-12T12:55:07.616718211Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:55:19 ts=2024-11-12T12:55:19.296872925Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:55:20 ts=2024-11-12T12:55:20.684855675Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-12 13:55:31 ts=2024-11-12T12:55:31.401555708Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:55:41 ts=2024-11-12T12:55:41.97687438Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 13:55:45 ts=2024-11-12T12:55:45.685097798Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-12 13:55:53 ts=2024-11-12T12:55:53.08140001Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f

Platform(s)

Mac (Apple Silicon)

What version/commit are you on?

v1.1.1

What database version are you on?

N/A

Which chain / network are you on?

Lisk Mainnet

What type of node are you running?

Full via --full flag

What prune config do you use, if any?

No response

If you've built Reth from source, provide the full command you used

No response

Code of Conduct

• I agree to follow the Code of Conduct

[mattsse]

there's a 5min diff between those logs

could you provide matching logs when this error first appears?
ideally with the reth_payload_builder target

[mattsse]

perhaps you can also try to increase the --builder.deadline

[matjazv]

I've executed Reth with --builder.deadline=30. Still the same issue.

New logs:
OP-NODE:

2024-11-12 15:06:37 t=2024-11-12T14:06:37+0000 lvl=info msg="Received signed execution payload from p2p" id=0x661b21ddd8f4586e84283e807b2b5f2c88092c18ebf7dab09f8329a182980d79:8345803 peer=16Uiu2HAmBuVpfZwoRodETmKQPiiE7awhs53vGaBzJCuZnSuU2zyb txs=1
2024-11-12 15:06:37 t=2024-11-12T14:06:37+0000 lvl=info msg="Optimistically queueing unsafe L2 execution payload" id=0x661b21ddd8f4586e84283e807b2b5f2c88092c18ebf7dab09f8329a182980d79:8345803
2024-11-12 15:06:38 t=2024-11-12T14:06:38+0000 lvl=warn msg="Failed to get payload" payload_id=0x039f75b9b517fb98 payload_id=0x039f75b9b517fb98 err="Unknown payload"
2024-11-12 15:06:38 t=2024-11-12T14:06:38+0000 lvl=warn msg="Cannot seal block, payload ID is unknown" payloadID=0x039f75b9b517fb98 payload_time=1714730555 started_time=2024-11-12T14:06:38+0000
2024-11-12 15:06:38 t=2024-11-12T14:06:38+0000 lvl=warn msg="Block sealing job of derived attributes expired, job will be re-attempted." build_id=0x039f75b9b517fb98 timestamp=1714730555 err="failed to seal execution payload (ID: 0x039f75b9b517fb98): Unknown payload"

Reth:

2024-11-12 15:06:15 ts=2024-11-12T14:06:15.517993841Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 15:06:24 ts=2024-11-12T14:06:24.775990179Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-12 15:06:26 ts=2024-11-12T14:06:26.156497929Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-12 15:06:38 ts=2024-11-12T14:06:38.124267254Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f

[mattsse]

could you provide trace level for payload_builder?

[sameersubudhi]

Hi @mattsse,

Could you please guide me on how do we build the op-reth client with reth_payload_builder target?

We currently run the following command to build the client:
RUSTFLAGS="-C target-cpu=native" cargo build --bin op-reth --locked --features jemalloc,asm-keccak,optimism --profile maxperf --manifest-path crates/optimism/bin/Cargo.toml

[mattsse]

you can configure this via RUST_LOG=payload_builder=trace,info env var

[MrFrogoz]

ethereum-optimism/optimism#12897 (comment)

[matjazv]

@mattsse
Logs for Reth client with enabled RUST_LOG=payload_builder=trace,info env var:

2024-11-13 10:09:53 ts=2024-11-13T09:09:53.578325757Z level=debug target=payload_builder message="awaiting in progress payload build job" id=0x039f75b9b517fb98
2024-11-13 10:09:53 ts=2024-11-13T09:09:53.57879709Z level=debug target=payload_builder message="sealed built block" sealed_block="SealedBlock { header: SealedHeader { hash: 0x9880f563da463b582bb504de91f2541bbf9a8e687766be8695505edb5ae53907, header: Header { parent_hash: 0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6df02d0e8b8da897f37b794634862bfba5ce097ac81f22c2bc45d6f51f784d1f, transactions_root: 0xdb895bf4bf911214338af3620dd84783cce9071a5d5bfd72d376d123def7afc8, receipts_root: 0x7601778118c1b03f5482f838bbf01a5d18faf7d8678d9488d12a632d538733b6, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0, number: 882, gas_limit: 30000000, gas_used: 52215, timestamp: 1714730555, mix_hash: 0x78b92d44efe68c1342d9714024b9955a0a9a8fb1d671b4718dd837cdabc1d304, nonce: 0x0000000000000000, base_fee_per_gas: Some(424921815), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x68b06e4beffbdb374d69a734e3a68fbfa26c36d5d56529c44f4b6f75917028f6), requests_hash: None, extra_data: 0x } }, body: BlockBody { transactions: [TransactionSigned { hash: 0x2f1f7e3f96563332f07405573bce2184365809f4b7dd118e4b9817e03c4403e6, signature: Signature { v: Parity(false), r: 0, s: 0 }, transaction: Deposit(TxDeposit { source_hash: 0xb620ae53f69ff7a3183a74b1be8f9354e17277ac4d0d029ca6be79cc9951023d, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0, gas_limit: 1000000, is_system_transaction: false, input: 0x440a5e20000f4240000000000000000000000000000000006634b63b00000000012df44000000000000000000000000000000000000000000000000000000001a22648260000000000000000000000000000000000000000000000000000000000000001ddccbc6008e855b86b6535a94157fd96d66eee56af0b61e7083d8fd0aa6e5495000000000000000000000000a6ea2f3299b63c53143c993d2d5e60a69cd6fe24 }) }], ommers: [], withdrawals: Some(Withdrawals([])) } }"
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.067858179Z level=trace target=payload_builder message="spawn new payload build task" id=0x039f75b9b517fb98
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.068245679Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.068707804Z level=debug target=payload_builder message="building new payload" id=0x039f75b9b517fb98 parent_header=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f parent_number=881
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.068943804Z level=debug target=payload_builder message="no best payload yet to resolve, building empty payload" id=0x039f75b9b517fb98
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.068971013Z level=debug target=payload_builder message="awaiting in progress payload build job" id=0x039f75b9b517fb98
2024-11-13 10:10:06 ts=2024-11-13T09:10:06.069855554Z level=debug target=payload_builder message="sealed built block" sealed_block="SealedBlock { header: SealedHeader { hash: 0x9880f563da463b582bb504de91f2541bbf9a8e687766be8695505edb5ae53907, header: Header { parent_hash: 0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6df02d0e8b8da897f37b794634862bfba5ce097ac81f22c2bc45d6f51f784d1f, transactions_root: 0xdb895bf4bf911214338af3620dd84783cce9071a5d5bfd72d376d123def7afc8, receipts_root: 0x7601778118c1b03f5482f838bbf01a5d18faf7d8678d9488d12a632d538733b6, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0, number: 882, gas_limit: 30000000, gas_used: 52215, timestamp: 1714730555, mix_hash: 0x78b92d44efe68c1342d9714024b9955a0a9a8fb1d671b4718dd837cdabc1d304, nonce: 0x0000000000000000, base_fee_per_gas: Some(424921815), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x68b06e4beffbdb374d69a734e3a68fbfa26c36d5d56529c44f4b6f75917028f6), requests_hash: None, extra_data: 0x } }, body: BlockBody { transactions: [TransactionSigned { hash: 0x2f1f7e3f96563332f07405573bce2184365809f4b7dd118e4b9817e03c4403e6, signature: Signature { v: Parity(false), r: 0, s: 0 }, transaction: Deposit(TxDeposit { source_hash: 0xb620ae53f69ff7a3183a74b1be8f9354e17277ac4d0d029ca6be79cc9951023d, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0, gas_limit: 1000000, is_system_transaction: false, input: 0x440a5e20000f4240000000000000000000000000000000006634b63b00000000012df44000000000000000000000000000000000000000000000000000000001a22648260000000000000000000000000000000000000000000000000000000000000001ddccbc6008e855b86b6535a94157fd96d66eee56af0b61e7083d8fd0aa6e5495000000000000000000000000a6ea2f3299b63c53143c993d2d5e60a69cd6fe24 }) }], ommers: [], withdrawals: Some(Withdrawals([])) } }"
2024-11-13 10:10:15 ts=2024-11-13T09:10:15.074794961Z level=info target=reth::cli message=Status connected_peers=0 latest_block=881
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.163064298Z level=trace target=payload_builder message="spawn new payload build task" id=0x039f75b9b517fb98
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.163326631Z level=info target=reth_payload_builder::service message="New payload job created" id=0x039f75b9b517fb98 parent=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.164509256Z level=debug target=payload_builder message="no best payload yet to resolve, building empty payload" id=0x039f75b9b517fb98
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.164188214Z level=debug target=payload_builder message="building new payload" id=0x039f75b9b517fb98 parent_header=0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f parent_number=881
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.164583548Z level=debug target=payload_builder message="awaiting in progress payload build job" id=0x039f75b9b517fb98
2024-11-13 10:10:22 ts=2024-11-13T09:10:22.166738173Z level=debug target=payload_builder message="sealed built block" sealed_block="SealedBlock { header: SealedHeader { hash: 0x9880f563da463b582bb504de91f2541bbf9a8e687766be8695505edb5ae53907, header: Header { parent_hash: 0xc1ed6db609d23ec3050258d409f4fcae537f7fd0ec6d29d54f0191f7ec8bc79f, ommers_hash: 0x1dcc4de8dec75d7aab85b567b6ccd41ad312451b948a7413f0a142fd40d49347, beneficiary: 0x4200000000000000000000000000000000000011, state_root: 0x6df02d0e8b8da897f37b794634862bfba5ce097ac81f22c2bc45d6f51f784d1f, transactions_root: 0xdb895bf4bf911214338af3620dd84783cce9071a5d5bfd72d376d123def7afc8, receipts_root: 0x7601778118c1b03f5482f838bbf01a5d18faf7d8678d9488d12a632d538733b6, withdrawals_root: Some(0x56e81f171bcc55a6ff8345e692c0f86e5b48e01b996cadc001622fb5e363b421), logs_bloom: 0x00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000, difficulty: 0, number: 882, gas_limit: 30000000, gas_used: 52215, timestamp: 1714730555, mix_hash: 0x78b92d44efe68c1342d9714024b9955a0a9a8fb1d671b4718dd837cdabc1d304, nonce: 0x0000000000000000, base_fee_per_gas: Some(424921815), blob_gas_used: Some(0), excess_blob_gas: Some(0), parent_beacon_block_root: Some(0x68b06e4beffbdb374d69a734e3a68fbfa26c36d5d56529c44f4b6f75917028f6), requests_hash: None, extra_data: 0x } }, body: BlockBody { transactions: [TransactionSigned { hash: 0x2f1f7e3f96563332f07405573bce2184365809f4b7dd118e4b9817e03c4403e6, signature: Signature { v: Parity(false), r: 0, s: 0 }, transaction: Deposit(TxDeposit { source_hash: 0xb620ae53f69ff7a3183a74b1be8f9354e17277ac4d0d029ca6be79cc9951023d, from: 0xdeaddeaddeaddeaddeaddeaddeaddeaddead0001, to: Call(0x4200000000000000000000000000000000000015), mint: None, value: 0, gas_limit: 1000000, is_system_transaction: false, input: 0x440a5e20000f4240000000000000000000000000000000006634b63b00000000012df44000000000000000000000000000000000000000000000000000000001a22648260000000000000000000000000000000000000000000000000000000000000001ddccbc6008e855b86b6535a94157fd96d66eee56af0b61e7083d8fd0aa6e5495000000000000000000000000a6ea2f3299b63c53143c993d2d5e60a69cd6fe24 }) }], ommers: [], withdrawals: Some(Withdrawals([])) } }"

[mattsse]

thanks for flagging,
somehow we never Hit this on our infra...
sorry about this, expediting new release with fix #12520 in the meantime please use 1.1.0