Elastic Scaling

[eskimor]

Eventually parachains should be able to occupy more than one core at once in a single block, which allows them to scale beyond a single core if their collators are powerful enough.

We decided to split up the work on elastic scaling into two/three phases. First phase is make it work for parachains which trust their collator set and don't share collations with untrusted machines. With this restriction it is possible to launch elastic/fixed factor scaling without any changes to the candidate receipt. Phase 2 will then actually make changes to the candidate receipt in order for the collator set to be untrusted again. (We put the core index in the candidate commitments, so it is not possible to push the same collation to multiple backing groups). Also phase 2 or potentially phase 3, we actually implement everything that is necessary on the cumulus side for elastic/on-demand scaling vs. fixed factor, where we just buy n cores all the time.

This ticket is about phase 1 and entails the following tasks:

Tasks

Preview Give feedback

1. Slot based collations #3168
   +0
   
2. collator protocol: Provide parent head data #3128
   +0
   
3. Coretime: prevent parachains from acquiring multiple cores per block if elastic scaling not support #3309
   T8-polkadot +1
4. prospective parachains API chain: Provide N candidates #3129
   +0
   
5. Fix (backing) relying on para ids being unique #3144
   +0
   
6. ParaInherent create: update apply_weight_limit wrt elastic scaling #3573
   R0-silent T8-polkadot +2
7. Enable elastic scaling node feature in local testnets genesis #3509
   R0-silent T10-tests T8-polkadot +3
8. statement-distribution: elastic scaling support #3775
   I5-enhancement T16-async_backing T8-polkadot +3
   
9. Add new staging Runtime API: candidates_pending_availability #3576
   T4-runtime_API +1
   
10. Add new variant of persisted_validation_data runtime API for elastic scaling #3776
    T4-runtime_API T8-polkadot +2
11. Provisioner: Elastic Scaling #3130
    +0
    
12. Runtime API: make backing_state return multiple candidates pending availability #3202
    +0
    
13. Runtime: Enactment of candidates takes dependencies into account #3132
    +0
    
14. Look-ahead collator: build on all assigned cores #3837
    I5-enhancement T9-cumulus +2
15. prospective-parachains: limit count of nodes in a fragment tree #3219
    T0-node T8-polkadot +2
    
16. prospective-parachains: allow chained candidates coming out-of-order #3541
    I5-enhancement T8-polkadot +2
    
17. Runtime: Accept full chain for para on multiple cores (elastic scaling) #3131
    +0
    
18. Check scheduler/coretime/.. that there are no restrictions on multiple cores with same para.
19. Expose and use claim queue for asynchronous on-demand operability #1797
    +0
    
20. Fun and profit for phase one.
21. Cumulus: add CLI switch for slot based collator #4467
    T9-cumulus +1
22. Elastic scaling: document how to enable elastic scaling in parachain runtime #4468
    T11-documentation +1
    
23. slot-based-collator: improve relay chain fork selection #4547
    I2-bug I9-optimisation T9-cumulus +3

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/scaling-a-polkadot-evm-parachain/4319/2

[eskimor]

Get core index into the candidate receipt, also useful here.

[eskimor]

#2984

[bkchr]

collator protocol: Add an additional (backwards compatibility) advertise collation message, which triggeres a different collation fetch request (requesting also the head data of the parent) -> Make use of that head data in candidate validation.

What does that mean?

[alindima]

collator protocol: Add an additional (backwards compatibility) advertise collation message, which triggeres a different collation fetch request (requesting also the head data of the parent) -> Make use of that head data in candidate validation.

What does that mean?

Assuming we have two validator groups, two cores for a parachain and we want to back two candidates in the same relay chain block, the second group needs to know the parent head data (which is the output of validation of the first candidate). Considering that candidate A hasn't yet been backed, the only source for this information on relay nodes would be the prospective-parachains subsystem (which records this data after getting a Seconded statement).

This latency of transmitting the seconded statement will partly prevent parallel execution of the two candidate validations.
We can fix this by passing in the head data in the collation fetching response (it only contains the hash for now).

[Polkadot-Forum]

This issue has been mentioned on Polkadot Forum. There might be relevant details there:

https://forum.polkadot.network/t/parity-tech-update-for-february/6630/1