[QST] What does "pointer array" mean in the recent Hopper examples?

[hyhieu]

Such as 56_hopper_ptr_array_batched_gemm.

Also, is this concept of "pointer array" demonstrated elsewhere in CUTLASS, or in other contexts for GPU programming?

Thanks!

[hwu36]

it is the same as cublas. there are two types of batch gemm, one is called pointer array: pointers to the input and output matrices are stored in arrays such as ptr_A[batch_id], ptr_B[batch_id]. the other one is called strided batch gemm, ptrA[batch_id] = ptrA[0]+stride x batch_id

[hyhieu]

That's very helpful. Thank you, @hwu36.

So, a follow up question: it looks like storing pointers to input and output matrices using ptr_A[batch_id], ptrBb[batch_id], etc. is clearly more flexible than using strided batch GEMM, because I can just put any pointers into ptr_A[...] and ptr_b[..].

Are there any tradeoffs? Am I missing anything?

[hwu36]

it depends on your application. if you choose to use the pointer array, you need to set up the pointers and store them in the memory before you launch the kernel.

[thakkarV]

ptr array is not as well optimized and/or supported. Prefer using strided batched as long as you can.