"Async" CCLs Support Master Issue

[SeanNijjar]

Phase 1: Initial Async CCLs (TG Llama configs only ini initial support list, full tensor granularity for synchronization)

EDM Fabric

• Linear Topology EDM Fabric
  • Chip Unicast command support
  • Chip Multicast command support
  • Noc Unicast
  • Noc Multicast
  • Noc Atomic Inc Command

CCL Command Support

• Bringup CCL command "CB_to_Tensor" (name to be workshopped :) ) in local mode

  • Adapt to fabric writes (unicast) Implement mcast on sender side to start after EDM fabric bringup
  • Update to mcast after chip mcast support added

• Bringup "Tensor_to_CB"

• Bringup semaphore inc

• Bringup semaphore wait

• Enable offset reads/writes into pages for Tensor stream commands #16403

• (deprioritized - nice to have) Bringup CCL command "ccl shift tensor" for local writes

  • Adapt to fabric_writes after EDM fabric availability

• Enable New All Gather #15060

  • CCL commands to read (reader kernel) and mcast tensor (writer kernel)
  • Add atomic inc at end of write

• Enable new reduce-scatter (For Llama demo level generality): #15006

  • writer: same as all-reduce excluding last mcast command, replaced with local output tensor write
  • reader: use existing CCL command to read tensor slices (because for reduce scatter, scatter dimension matters)

• Enable CCL V2 (composite) all-reduce #16400

• Persistent Fabric Support

  • APIs to setup/teardown persistent fabric along with some op tests that are running on subdevice alongside persistent fabric

• Integrate CCL V2 into TG Llama #16607

  • Add arbitrary shard-grid core range set support to addrgen in command processor (ttnn/cpp/ttnn/operations/ccl/common/kernels/ccl_send_reader_two_input.cpp) #16608
  • Integrate V2 All-reduce into TG Llama codebase #16397
    • Construct reduce-scatter global semaphores outside of op invocation and pass in to all-reduce
    • Construct all-gather global semaphores outside of op invocation and pass in to all-gather #16396

• Construct reduce-scatter global semaphores outside of op invocation and pass in to reduce-scatter #16398

  • Construct all-gather global semaphores outside of op invocation and pass in to reduce-scatter

Bugs

• [Bug] Reduce Scatter (Cluster-axis only) fails when running with persistent fabric #16390
• All-gather v2 hangs when running with cluster axis API on persistent fabric #16391

Perf: Parallel with end of Phase 1 and Phase 2

• EDM Fabric tuning (optimize EDM)

• Worker count tuning (May need to add more workers)

  • Add fabric endpoint to merge incoming traffic

• Convert tensor level commands to resolved addresses/page IDs (improved perf due to lower SW overheads) - all gather #16395

• Enable CCL V2 (Fused) all-reduce

Phase 2: Add finer granularity tensor synchronization

• Expose slicing to model developer, inject atomic incs after each one
  • Model dev/op caller gets full control over granularity and schedule