You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, Matmul1D with gather_in0=True does not handle shapes (K, N) that do not divide by the number of cores in the ring. In the current use case in the Llama models, the activations and weights need to be padded in order to use the matmul. However, this results in significant overhead caused by padding/slicing. Therefore, this matmul must support shapes that do not divide with the number of cores, and handle the padding implicitly.
The text was updated successfully, but these errors were encountered:
Problem
Currently, Matmul1D with
gather_in0=True
does not handle shapes (K, N) that do not divide by the number of cores in the ring. In the current use case in the Llama models, the activations and weights need to be padded in order to use the matmul. However, this results in significant overhead caused by padding/slicing. Therefore, this matmul must support shapes that do not divide with the number of cores, and handle the padding implicitly.The text was updated successfully, but these errors were encountered: