warp sync

[staking4all]

Is there an existing issue?

• I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

• This is not a support question.

Description of bug

When using warp sync. If the node has few cpu resources it will die/terminate/kill itself when starting the import process

Increasing cpu resolves this

Steps to reproduce

Steps to produce
->Create a node with few cpu resources, 3 threads as an example
->Start warp sync
->Will terminate itself when it starts import part

[bkchr]

What network do you try to warp sync?

Can you please also run with -ltrace and link the logs?

[staking4all]

Hi

The network is polkadot, below is the command
./polkadot --validator --sync warp

Complete logs

2023-01-22 06:36:01 Parity Polkadot
2023-01-22 06:36:01 ✌️  version 0.9.37-645723987cf
2023-01-22 06:36:01 ❤️  by Parity Technologies <[email protected]>, 2017-2023
2023-01-22 06:36:01 📋 Chain specification: Polkadot
2023-01-22 06:36:01 🏷  Node name: lyrical-waves-9672
2023-01-22 06:36:01 👤 Role: AUTHORITY
2023-01-22 06:36:01 💾 Database: RocksDb at /root/.local/share/polkadot/chains/polkadot/db/full
2023-01-22 06:36:01 ⛓  Native runtime: polkadot-9370 (parity-polkadot-0.tx20.au0)
2023-01-22 06:36:04 🔨 Initializing Genesis block/state (state: 0x29d0…4e17, header-hash: 0x91b1…90c3)
2023-01-22 06:36:04 👶 Creating empty BABE epoch changes on what appears to be first startup.
2023-01-22 06:36:04 🏷  Local node identity is: 12D3KooWAwAH4XKNdvhJ4BcyCJyHrNJWAKUxyDQAmBJn9Ws8H39v
2023-01-22 06:36:04 💻 Operating system: linux
2023-01-22 06:36:04 💻 CPU architecture: x86_64
2023-01-22 06:36:04 💻 Target environment: gnu
2023-01-22 06:36:04 💻 CPU: AMD EPYC Processor
2023-01-22 06:36:04 💻 CPU cores: 3
2023-01-22 06:36:04 💻 Memory: 3835MB
2023-01-22 06:36:04 💻 Kernel: 5.4.0-135-generic
2023-01-22 06:36:04 💻 Linux distribution: Ubuntu 20.04.5 LTS
2023-01-22 06:36:04 💻 Virtual machine: yes
2023-01-22 06:36:04 📦 Highest known block at #0
2023-01-22 06:36:04 〽️ Prometheus exporter started at 127.0.0.1:9615
2023-01-22 06:36:04 Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2023-01-22 06:36:04 Running JSON-RPC WS server: addr=127.0.0.1:9944, allowed origins=["http://localhost:*", "http://127.0.0.1:*", "https://localhost:*", "https://127.0.0.1:*", "https://polkadot.js.org"]
2023-01-22 06:36:04 🏁 CPU score: 699.70 MiBs
2023-01-22 06:36:04 🏁 Memory score: 15.74 GiBs
2023-01-22 06:36:04 🏁 Disk score (seq. writes): 1.41 GiBs
2023-01-22 06:36:04 🏁 Disk score (rand. writes): 516.08 MiBs
2023-01-22 06:36:05 Starting with an empty approval vote DB.
2023-01-22 06:36:05 👶 Starting BABE Authorship worker
2023-01-22 06:36:05 Failed to fetch basics from runtime API relay_parent=0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3 err=Execution { runtime_api_name: "session_index_for_child", source: Application(Execution(Other("Exported method ParachainHost_session_index_for_child is not found"))) }
2023-01-22 06:36:05 Our view contains 0x91b1…90c3, but not in active heads added=0x91b1…90c3
2023-01-22 06:36:05 Failed to fetch runtime API data for job err=Execution { runtime_api_name: "validator_groups", source: Application(Execution(Other("Exported method ParachainHost_validator_groups is not found"))) }
2023-01-22 06:36:05 error=Sender(Runtime(RuntimeRequest(Execution { runtime_api_name: "session_index_for_child", source: Application(Execution(Other("Exported method ParachainHost_session_index_for_child is not found"))) }))) ctx="on FromOrchestra"
2023-01-22 06:36:05 error=RollingSessionWindow(SessionsUnavailable { kind: RuntimeApi(Execution { runtime_api_name: "session_index_for_child", source: Application(Execution(Other("Exported method ParachainHost_session_index_for_child is not found"))) }), info: None })
2023-01-22 06:36:05 cannot fetch session index from runtime API: ApiError relay_parent=0x91b171bb158e2d3848fa23a9f1c25182fb8e20313b2c1eb49219da7a70ce90c3
2023-01-22 06:36:05 🔍 Discovered new external address for our node: /ip4/135.181.111.5/tcp/30333/p2p/12D3KooWAwAH4XKNdvhJ4BcyCJyHrNJWAKUxyDQAmBJn9Ws8H39v
2023-01-22 06:36:12 ⏩ Warping, Downloading finality proofs, 15.94 Mib (37 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.4MiB/s ⬆ 69.4kiB/s
2023-01-22 06:36:19 ⏩ Warping, Downloading finality proofs, 31.85 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.8MiB/s ⬆ 128.1kiB/s
2023-01-22 06:36:25 ⏩ Warping, Downloading finality proofs, 47.83 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.9MiB/s ⬆ 164.6kiB/s
2023-01-22 06:36:25 ❌ Error while dialing /dns/telemetry.polkadot.io/tcp/443/x-parity-wss/%2Fsubmit%2F: Custom { kind: Other, error: Timeout }
2023-01-22 06:36:30 ⏩ Warping, Downloading state, 61.58 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.8MiB/s ⬆ 122.7kiB/s
2023-01-22 06:36:35 ⏩ Warping, Downloading state, 100.15 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 5.0MiB/s ⬆ 274.4kiB/s
2023-01-22 06:36:40 ⏩ Warping, Downloading state, 136.29 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 5.0MiB/s ⬆ 243.4kiB/s
2023-01-22 06:36:45 ⏩ Warping, Downloading state, 174.85 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 5.0MiB/s ⬆ 249.7kiB/s
2023-01-22 06:36:50 ⏩ Warping, Downloading state, 213.40 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 5.2MiB/s ⬆ 310.0kiB/s
2023-01-22 06:36:55 ⏩ Warping, Downloading state, 249.55 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 4.8MiB/s ⬆ 257.5kiB/s
2023-01-22 06:37:00 ⏩ Warping, Downloading state, 273.57 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.2MiB/s ⬆ 201.1kiB/s
2023-01-22 06:37:00 ❌ Error while dialing /dns/telemetry.polkadot.io/tcp/443/x-parity-wss/%2Fsubmit%2F: Custom { kind: Other, error: Timeout }
2023-01-22 06:37:05 ⏩ Warping, Downloading state, 291.92 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.8MiB/s ⬆ 133.3kiB/s
2023-01-22 06:37:10 ⏩ Warping, Downloading state, 321.10 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.7MiB/s ⬆ 150.0kiB/s
2023-01-22 06:37:15 ⏩ Warping, Downloading state, 345.12 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.1MiB/s ⬆ 84.1kiB/s
2023-01-22 06:37:20 ⏩ Warping, Downloading state, 366.63 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.8MiB/s ⬆ 156.3kiB/s
2023-01-22 06:37:25 ⏩ Warping, Downloading state, 385.17 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.8MiB/s ⬆ 121.1kiB/s
2023-01-22 06:37:30 ⏩ Warping, Downloading state, 405.54 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 4.0MiB/s ⬆ 185.9kiB/s
2023-01-22 06:37:35 ⏩ Warping, Downloading state, 421.83 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.5MiB/s ⬆ 159.8kiB/s
2023-01-22 06:37:40 ⏩ Warping, Downloading state, 438.13 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.0MiB/s ⬆ 183.8kiB/s
2023-01-22 06:37:45 ⏩ Warping, Downloading state, 451.18 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.3MiB/s ⬆ 174.4kiB/s
2023-01-22 06:37:50 ⏩ Warping, Downloading state, 467.47 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.5MiB/s ⬆ 289.7kiB/s
2023-01-22 06:37:55 ⏩ Warping, Downloading state, 481.71 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.5MiB/s ⬆ 429.2kiB/s
2023-01-22 06:38:00 ⏩ Warping, Downloading state, 497.99 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.4MiB/s ⬆ 495.4kiB/s
2023-01-22 06:38:05 ⏩ Warping, Downloading state, 514.27 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.5MiB/s ⬆ 462.1kiB/s
2023-01-22 06:38:10 ⏩ Warping, Downloading state, 519.02 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 1.0MiB/s ⬆ 433.4kiB/s
2023-01-22 06:38:15 ⏩ Warping, Downloading state, 524.22 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 889.9kiB/s ⬆ 291.8kiB/s
2023-01-22 06:38:20 ⏩ Warping, Downloading state, 535.00 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 1.8MiB/s ⬆ 179.8kiB/s
2023-01-22 06:38:25 ⏩ Warping, Downloading state, 543.80 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 1.7MiB/s ⬆ 223.9kiB/s
2023-01-22 06:38:30 ⏩ Warping, Downloading state, 552.98 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 1.4MiB/s ⬆ 187.8kiB/s
2023-01-22 06:38:35 ⏩ Warping, Downloading state, 552.98 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 205.4kiB/s ⬆ 187.0kiB/s
2023-01-22 06:38:40 ⏩ Warping, Downloading state, 552.98 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 378.1kiB/s ⬆ 250.9kiB/s
2023-01-22 06:38:45 ⏩ Warping, Downloading state, 555.55 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 304.0kiB/s ⬆ 160.7kiB/s
2023-01-22 06:38:50 ⏩ Warping, Downloading state, 555.55 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 319.3kiB/s ⬆ 315.3kiB/s
2023-01-22 06:38:55 ⏩ Warping, Downloading state, 558.14 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 490.7kiB/s ⬆ 222.8kiB/s
2023-01-22 06:39:00 ⏩ Warping, Downloading state, 558.14 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 260.2kiB/s ⬆ 257.0kiB/s
2023-01-22 06:39:05 ⏩ Warping, Downloading state, 560.72 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 456.4kiB/s ⬆ 190.3kiB/s
2023-01-22 06:39:10 ⏩ Warping, Downloading state, 560.72 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 191.9kiB/s ⬆ 189.4kiB/s
2023-01-22 06:39:15 ⏩ Warping, Downloading state, 563.30 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 498.1kiB/s ⬆ 230.3kiB/s
2023-01-22 06:39:20 ⏩ Warping, Downloading state, 573.12 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.2MiB/s ⬆ 299.0kiB/s
2023-01-22 06:39:25 ⏩ Warping, Downloading state, 588.92 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.5MiB/s ⬆ 267.9kiB/s
2023-01-22 06:39:30 ⏩ Warping, Downloading state, 605.23 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.6MiB/s ⬆ 256.6kiB/s
2023-01-22 06:39:35 ⏩ Warping, Downloading state, 622.30 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 3.4MiB/s ⬆ 262.0kiB/s
2023-01-22 06:39:40 ⏩ Warping, Importing state, 634.58 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 2.4MiB/s ⬆ 279.9kiB/s
2023-01-22 06:39:45 ⏩ Warping, Importing state, 634.58 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 420.9kiB/s ⬆ 419.2kiB/s
2023-01-22 06:39:50 ⏩ Warping, Importing state, 634.58 Mib (40 peers), best: #0 (0x91b1…90c3), finalized #0 (0x91b1…90c3), ⬇ 357.8kiB/s ⬆ 353.0kiB/s
Killed

[bkchr]

You should run it with -ltrace (this is an extra CLI flag)

[arkpar]

It looks like the process is killed by the OS because the system runs out of memory. Check your dmesg output.

[staking4all]

@bkchr - the ltrace is to big, even zipped. So unable to attach here.

@arkpar - it is resource related, cld be memory or cpu. Was testing on small instance to have less costs. Larger instances works fine.

[bkchr]

Reason should be: paritytech/polkadot-sdk#4 aka we keep everything in memory.