[Fabric] Add controller kernel to sync the producers #17629
+178
−28
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Link to Github Issue
Problem description
Currently the producer kernels wait for the signal from host to start transmitting packets. This could cause the workers to transmit the packets in non-overlapping time frames thus impacting perf calculations. This could get as worse as the workers running serially because of fabric not allowing context switch for the slow dispatch commands to go through.
What's changed
Add a new controller kernel that now synchronizes all the producers on the same chip. This kernel now waits for the signal from the host and then noc mcasts the notification to the worker kernels, thus reducing the skew to the order of 10s of cycles. This kernel can be extended to setup other handshakes in the future as well.
Checklist