TG Llama CCL Perf Master

[SeanNijjar]

CCL Add New Commands for Perf

• Enable Fine-Grain (per page/per packet) flow control between producer and consumer CCL commands #16491

Reduce Scatter

• enable tensor slice lowering to noc read/write bursts in reduce scatter #16490
• Uplift reduce scatter to use write-atomic mode for tensor slice and noc bursts
  • Depends on Enable Fine-Grain (per page/per packet) flow control between producer and consumer CCL commands #16491

All Reduce

• Implement Fused All-Reduce-Async #16641
• Enable tensor slice lowering for fused all-reduce
• Uplift fused all-reduce to use write-atomic mode for tensor slice and noc bursts

March to 12GBps per link

CCL Backend Kernels Optimization:

Optimize main loop and noc burst commands:

• CCL V2 (Reference Interpreter Kernel) Optimizations #16255
• partly on branch) Optimize outer loop overheads
  • Measured improvement (on branch): flatten try advance switch-case statements merge the advance and completion checks (big savings of ~150-170ns per outer loop iteration (once for every command or packet, whichever is more frequent) which can represent 15-20% improvement
• (on branch) User "one_packet" data flow APIs
  • Estimated improvements: save 50-100ns per CB packet
• Allow multiple open packets in flight to avoid blocking read/write barriers for commands
• Optimize packet header initialization
  • a surprising amount of time is spent on this
  • enable caching
  • simplify packet structure (see Fabric EDM Optimizations)

Fabric EDM Optimization:

• Merge command type and noc command type in packet header.
  • @tt-aho has in progress commit here:
• Decouple worker acks from EDM acks.
  • This will be essential if we keep one worker per link
  • move EDM to raw read/write offsets into channel
  • Update worker connection open/close to get credits and channel offsets
    • currently each worker requires EDM in the loop for every channel buffer (ack)
      • Instead, EDM and worker should be able to manage read/write pointers independently without blocking
    • Will inadvertently change state machine so may negate next item
• Short circuit sender states from send to worker ack
• Enable write-atomic operation
  • Use available back 16B to store atomic inc info

Perf Reporting

• Enable perf reporting for reduce-scatter async, all-gather async, all-reduce async (when added) #16648