FM-363: State sync test

[aakoshh]

Closes consensus-shipyard/ipc#178

The snapshot-test now sets up 4 nodes:

• node-0: default created by cargo make setup, preserving full block history
• node-1 and node-2: using node-0 as P2P seed, preserving only 100 long block history, snapshotting every 10th block
• node-3: using node-1 as P2P seed, and both node-1 and node-2 as RPC for the light client, with syncstate enabled in cometbft

Once node-0, node-1 and node-2 are running, cargo make node-3-setup starts the last one. A successful sync looks as follows:

node-3 logs

❯ docker logs snapshot-cometbft-3
I[2023-12-12|23:16:15.021] service start                                module=proxy msg="Starting multiAppConn service" impl=multiAppConn
I[2023-12-12|23:16:15.021] service start                                module=abci-client connection=query msg="Starting socketClient service" impl=socketClient
I[2023-12-12|23:16:15.024] service start                                module=abci-client connection=snapshot msg="Starting socketClient service" impl=socketClient
I[2023-12-12|23:16:15.025] service start                                module=abci-client connection=mempool msg="Starting socketClient service" impl=socketClient
I[2023-12-12|23:16:15.029] service start                                module=abci-client connection=consensus msg="Starting socketClient service" impl=socketClient
I[2023-12-12|23:16:15.031] service start                                module=events msg="Starting EventBus service" impl=EventBus
I[2023-12-12|23:16:15.032] service start                                module=pubsub msg="Starting PubSub service" impl=PubSub
I[2023-12-12|23:16:15.048] service start                                module=txindex msg="Starting IndexerService service" impl=IndexerService
I[2023-12-12|23:16:15.048] Version info                                 module=main tendermint_version=0.37.4 abci=1.0.0 block=11 p2p=8 commit_hash=8b360e1
I[2023-12-12|23:16:15.048] This node is not a validator                 module=consensus addr=7ABE6A5C5DE9FC20B739E0BB3843FFCB500EBAF6 pubKey=PubKeyEd25519{646E4F05E5105D1B6A4CB672882B201A9C8644F5970D2C442FDE5AD8E58F0AA5}
I[2023-12-12|23:16:15.069] P2P Node ID                                  module=p2p ID=879c372c18c498e67fce9d83c65c62a8c986bfa0 file=/cometbft/config/node_key.json
I[2023-12-12|23:16:15.069] Adding persistent peers                      module=p2p addrs=[a06d6526be45c541183b358533dbaf4eef21a493@snapshot-cometbft-1:26656]
I[2023-12-12|23:16:15.070] Adding unconditional peer ids                module=p2p ids=[]
I[2023-12-12|23:16:15.071] Add our address to book                      module=p2p book=/cometbft/config/addrbook.json [email protected]:26656
I[2023-12-12|23:16:15.071] service start                                module=main msg="Starting Node service" impl=Node
I[2023-12-12|23:16:15.072] service start                                module=p2p msg="Starting P2P Switch service" impl="P2P Switch"
I[2023-12-12|23:16:15.072] serve                                        module=rpc-server msg="Starting RPC HTTP server on [::]:26657"
I[2023-12-12|23:16:15.072] service start                                module=blockchain msg="Starting Reactor service" impl=Reactor
I[2023-12-12|23:16:15.072] service start                                module=consensus msg="Starting Consensus service" impl=ConsensusReactor
I[2023-12-12|23:16:15.072] Reactor                                      module=consensus waitSync=true
I[2023-12-12|23:16:15.072] service start                                module=evidence msg="Starting Evidence service" impl=Evidence
I[2023-12-12|23:16:15.072] service start                                module=statesync msg="Starting StateSync service" impl=StateSync
I[2023-12-12|23:16:15.073] Saving AddrBook to file                      module=p2p book=/cometbft/config/addrbook.json size=1
I[2023-12-12|23:16:15.078] Starting state sync                          module=statesync 
I[2023-12-12|23:16:15.079] Downloading trusted light block using options module=light 
I[2023-12-12|23:16:15.164] Started node                                 module=main nodeInfo="{ProtocolVersion:{P2P:8 Block:11 App:0} DefaultNodeID:879c372c18c498e67fce9d83c65c62a8c986bfa0 ListenAddr:tcp://0.0.0.0:26656 Network:2618377826245745 Version:0.37.4 Channels:402021222330386061 Moniker:dockernode Other:{TxIndex:on RPCAddress:tcp://0.0.0.0:26657}}"
I[2023-12-12|23:16:15.164] sync any                                     module=statesync msg="Discovering snapshots for 15s"
I[2023-12-12|23:16:15.229] service start                                module=p2p [email protected]:26656 msg="Starting Peer service" impl="Peer{MConn{172.19.0.6:26656} a06d6526be45c541183b358533dbaf4eef21a493 out}"
I[2023-12-12|23:16:15.229] service start                                module=p2p [email protected]:26656 msg="Starting MConnection service" impl=MConn{172.19.0.6:26656}
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=160 format=1 hash=5A518C455112FFFDF42EEFDB066F85C103D7EFAC95FC59F013FB4FDF5A388BC3
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=150 format=1 hash=73BD71184C61719155B0AFBB74002463ABC9B0E50B5BEBD4F161C11887A7576D
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=140 format=1 hash=BE1B7CFEBC1D19042C132B9C0E978FE7A94C4D71A4030A6E6CE43540924C2C7E
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=130 format=1 hash=BA3BE923A0B8F40F2E48701F97D2CB4AB44C1CDAA83F5CB488DC3E0B5A2884E9
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=120 format=1 hash=34AF93255616F04C565A5E77138BD8C10ADBFACEFB8E2A1AE26717A25D781DD0
I[2023-12-12|23:16:15.433] Discovered new snapshot                      module=statesync height=110 format=1 hash=43122B386AD741D247225469FE53E6049DE228743881C0D8135B77BEC4134172
I[2023-12-12|23:16:15.434] Discovered new snapshot                      module=statesync height=100 format=1 hash=8BF46D63BAB4AA8BF10BB862886E88EE2462310E7C9F3760E955FFA95933B95C
I[2023-12-12|23:16:15.434] Discovered new snapshot                      module=statesync height=90 format=1 hash=964780A4EADDCDEADFA33C8C949DF6D9489B4689FD9D3B8C03EFF0FF999357F2
I[2023-12-12|23:16:15.434] Discovered new snapshot                      module=statesync height=80 format=1 hash=90ADA427EAFCEF1C5A102F8A53EE71F9A93A5CA015893DF674A30E33D28264B5
I[2023-12-12|23:16:15.434] Discovered new snapshot                      module=statesync height=70 format=1 hash=59316AB08667A06D6D4AE9B5545E4CBAFEBBB9C9B9119B231C10854A3FFA8E20
I[2023-12-12|23:16:30.171] VerifyHeader                                 module=light height=161 hash=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7
I[2023-12-12|23:16:30.181] VerifyHeader                                 module=light height=162 hash=C5DC4C09C938FE9D912304D8F6768DA1A909993453565C507E65C9D1A03DEA4E
I[2023-12-12|23:16:30.185] Offering snapshot to ABCI app                module=statesync height=160 format=1 hash=5A518C455112FFFDF42EEFDB066F85C103D7EFAC95FC59F013FB4FDF5A388BC3
I[2023-12-12|23:16:30.186] Snapshot accepted, restoring                 module=statesync height=160 format=1 hash=5A518C455112FFFDF42EEFDB066F85C103D7EFAC95FC59F013FB4FDF5A388BC3
I[2023-12-12|23:16:30.186] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=0 total=8
I[2023-12-12|23:16:30.186] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=1 total=8
I[2023-12-12|23:16:30.186] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=2 total=8
I[2023-12-12|23:16:30.187] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=3 total=8
I[2023-12-12|23:16:30.190] VerifyHeader                                 module=light height=160 hash=BB559F62D37FCB92C7A72716F79EEB4BD43751D39BB7690FA241241E465EDF52
I[2023-12-12|23:16:30.196] Header has already been verified             module=light height=161 hash=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7
I[2023-12-12|23:16:30.196] Header has already been verified             module=light height=162 hash=C5DC4C09C938FE9D912304D8F6768DA1A909993453565C507E65C9D1A03DEA4E
I[2023-12-12|23:16:30.199] Header has already been verified             module=light height=161 hash=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7
I[2023-12-12|23:16:30.199] Header has already been verified             module=light height=160 hash=BB559F62D37FCB92C7A72716F79EEB4BD43751D39BB7690FA241241E465EDF52
I[2023-12-12|23:16:30.531] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=4 total=8
I[2023-12-12|23:16:30.531] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=5 total=8
I[2023-12-12|23:16:30.532] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=0 total=8
I[2023-12-12|23:16:30.547] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=1 total=8
I[2023-12-12|23:16:30.747] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=6 total=8
I[2023-12-12|23:16:30.759] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=2 total=8
I[2023-12-12|23:16:31.242] Fetching snapshot chunk                      module=statesync height=160 format=1 chunk=7 total=8
I[2023-12-12|23:16:31.263] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=3 total=8
I[2023-12-12|23:16:31.445] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=4 total=8
I[2023-12-12|23:16:31.762] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=5 total=8
I[2023-12-12|23:16:32.037] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=6 total=8
I[2023-12-12|23:16:32.338] Applied snapshot chunk to ABCI app           module=statesync height=160 format=1 chunk=7 total=8
I[2023-12-12|23:16:32.338] Verified ABCI app                            module=statesync height=160 appHash=0171A0E40220F5606000816CD4F26001D8E36A8EF77C2D4E3A96E8B5E2F4D787AE09997D2AED
I[2023-12-12|23:16:32.338] Snapshot restored                            module=statesync height=160 format=1 hash=5A518C455112FFFDF42EEFDB066F85C103D7EFAC95FC59F013FB4FDF5A388BC3
I[2023-12-12|23:16:32.343] service start                                module=blockchain msg="Starting BlockPool service" impl=BlockPool
I[2023-12-12|23:16:33.344] Time to switch to consensus reactor!         module=blockchain height=161
I[2023-12-12|23:16:33.344] service stop                                 module=blockchain msg="Stopping BlockPool service" impl=BlockPool
I[2023-12-12|23:16:33.344] SwitchToConsensus                            module=consensus 
I[2023-12-12|23:16:33.344] service start                                module=consensus msg="Starting State service" impl=ConsensusState
I[2023-12-12|23:16:33.344] service start                                module=consensus wal=/cometbft/data/cs.wal/wal msg="Starting baseWAL service" impl=baseWAL
I[2023-12-12|23:16:33.348] service start                                module=consensus wal=/cometbft/data/cs.wal/wal msg="Starting Group service" impl=Group
I[2023-12-12|23:16:33.348] service start                                module=consensus msg="Starting TimeoutTicker service" impl=TimeoutTicker
I[2023-12-12|23:16:33.539] commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=161 commit_round=0 proposal= commit=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7
I[2023-12-12|23:16:33.540] received complete proposal block             module=consensus height=161 hash=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7
I[2023-12-12|23:16:33.541] finalizing commit of block                   module=consensus height=161 hash=64435B51697E445BC739B2506942479A5D6084289530A0D2B90CEC963CC452D7 root=0171A0E40220F5606000816CD4F26001D8E36A8EF77C2D4E3A96E8B5E2F4D787AE09997D2AED num_txs=0
I[2023-12-12|23:16:36.539] Timed out                                    module=consensus dur=3s height=161 round=0 step=RoundStepPropose
I[2023-12-12|23:16:42.209] executed block                               module=state height=161 num_valid_txs=0 num_invalid_txs=0
I[2023-12-12|23:16:42.214] committed state                              module=state height=161 num_txs=0 app_hash=0171A0E402205C67B725A384C23B4C062531116A3B907AD5C30E945352309E6EA9FA0153B57A
I[2023-12-12|23:16:42.217] Timed out                                    module=consensus dur=-8.177873012s height=162 round=0 step=RoundStepNewHeight
I[2023-12-12|23:16:42.220] indexed block events                         module=txindex height=161
I[2023-12-12|23:16:42.472] commit is for a block we do not know about; set ProposalBlock=nil module=consensus height=162 commit_round=0 proposal= commit=C5DC4C09C938FE9D912304D8F6768DA1A909993453565C507E65C9D1A03DEA4E
I[2023-12-12|23:16:42.472] received complete proposal block             module=consensus height=162 hash=C5DC4C09C938FE9D912304D8F6768DA1A909993453565C507E65C9D1A03DEA4E
I[2023-12-12|23:16:42.474] finalizing commit of block                   module=consensus height=162 hash=C5DC4C09C938FE9D912304D8F6768DA1A909993453565C507E65C9D1A03DEA4E root=0171A0E402205C67B725A384C23B4C062531116A3B907AD5C30E945352309E6EA9FA0153B57A num_txs=0
I[2023-12-12|23:16:42.486] executed block                               module=state height=162 num_valid_txs=0 num_invalid_txs=0
I[2023-12-12|23:16:42.495] committed state                              module=state height=162 num_txs=0 app_hash=0171A0E402205C67B725A384C23B4C062531116A3B907AD5C30E945352309E6EA9FA0153B57A
I[2023-12-12|23:16:42.506] indexed block events                         module=txindex height=162

The nodes can be stopped with cargo make snapshot-teardown, leaving only node-0 which can be stopped with cargo make teardown. The whole snapshot test routine can be done with cargo make, and the node-1|2|3 parts with cargo make test.

The PR also fixes the CMT_ env vars in the infra scripts: we missed the [rpc] and [p2p] parts in the config.

If we want we can enable snapshots by default now.