feat: [INFRA-2749] change labels to match for gpu to use gpu-based instances for build

[jozwior]

https://linear.app/worldcoin/issue/INFRA-2749/add-github-self-hosted-gpu-runners

change labels to match for gpu to use gpu-based instances for build

[linear]

INFRA-2749 Add GitHub Self-hosted GPU runners

This is required to run CI tests on https://github.com/worldcoin/gpu-iris-mpc

Screenshot 2024-07-12 at 11.40.41.png

The instance needs to have 3 GPUs at least, e.g.: g4dn.12xlarge

Screenshot 2024-07-12 at 15.01.59.png

[dkales]

Seems like the self-hosted runner images do not have the same tooling as the gh-provided ones, ran into that myself once or twice.
I would recommend https://github.com/dtolnay/rust-toolchain/ as an action to install rust which should work across both images.

[dkales]

I added a job to run the e2e tests on the GPU runner. The image is missing the CUDA and NCCL libs atm, should they be baked into the image used or installed during runtime?

also the build is pretty slow on them, could probably use some form of caching like https://github.com/Swatinem/rust-cache

[philsippl]

After a long battle with this PR, it seems I finally found the issue.
As soon as m (lda) in the gemm_ex is not divisible by 4, we're getting wrong results on the Nvidia L40 with Cuda 12.2 (what the github runner uses). I've verified this issue with a CUDA C implementation.

Meanwhile on the Nvidia H100, this works without any issues and that's why we didn't find it for so long. This PR now makes sure thatm = chunksize is always divisible by 4 and asserts that in the gemm_ex. I'm also fairly sure that this is also the cause for staging issues (Nvidia T4), but here it doesn't just silently give wrong results, but raise a cuBLAS error. This PR now also tries to get more details about this error by calling cublasGetStatusString.

@wojciechsromek what's the deal with this dockerfile? that's the only failing check now.