Hermetic CUDA Toolkit

[cloudhan]

This issue track the progress of Hermetic CUDA Toolkit implementation.

In 0.2.x

• refactor: split monolithic BUILD.local_cuda into template fragments #284
  • fix: add culibos to cuda_runtime #302
  • feat: add static version cufft (for device callback) #303
  • feat: add nvjpeg #304
  • feat: add nvjitlink #305
  • ci: make chatty #306
• fix: add :compiler_deps to cuda_toolchain #301
• feat: add compiler_label attr to cuda_toolchain for in repo compiler executable #309
• fix: add rdc filter to cuda_binary #310

---

In 0.3.x

• refactor!: change for hermetic with minor user interface breaking #300
• feat: hermetic ctk with deliverable #285
• feat: load("@rules_cuda_redist_json//:redist.bzl", "rules_cuda_components") #286

[udaya2899]

@cloudhan, super excited for this! Thanks for starting the work on this. Roughly when do you see this to be done?

[cloudhan]

I am currently on my way of jumping ship, that is, I am joining NVIDIA ;). It may take sometime for me to settle down so it might take a little bit longer time. I'd hope I can have a working version by the end of next month.

[honeway]

First of all, I wish you all the best in your work! Thank you for your efforts on this. We're eagerly looking forward to seeing progress on this feature, as it’s something we truly need. Please let us know if there’s any way we can assist or contribute.

[udaya2899]

@cloudhan, I hope your time at NVIDIA is going well! We're really excited about the possibilities of using hermetic CUDA in RBE. We're currently facing a decision about whether to build a temporary non-hermetic solution for RBE or wait for this issue to be resolved.

Could you give us an update on your plan here? Any information you can share would help us make the best decision for our project's roadmap. +1 to @honeway and we'll be happy to assist some way too.

[hofbi]

One this effort start, I can recommend using a rule based toolchain which was announced on the last BazelCon to be the modern way of writing toolchains.

[cloudhan]

@udaya2899 The cloudhan/hermetic-ctk-2 branch is actually working months ago with. Better test on it and provide some feedback.

[cloudhan]

@hofbi Seem to be very interesting. But it seems to be in a very early stage. Better just wait now.

[cloudhan]

For a preview,

https://github.com/cloudhan/cuda-samples/blob/bazel-cuda-components/WORKSPACE.bazel shows how a manually configured repo will be. Branch cloudhan/hermetic-ctk-2 contains the related feature.

https://github.com/cloudhan/cuda-samples/blob/bazel-cuda-redist-json/WORKSPACE.bazel shows how a automatically configured repo will be for WORKSPACE based project. Branch cloudhan/hermetic-ctk-3 contains the related feature.

[udaya2899]

Thanks for working on this now. We're on holiday season and I couldn't get time to experiment with your dev branch until now.

Expect to hear from me by the second week of January.

Unfortunately, we don't support WORKSPACE in our setup, and only use MODULE.bazel. Which is the most recent branch to try from? Is MODULE.bazel considered working in the tmp branch? Or is it hermetic-ctk-2 branch?

[udaya2899]

Happy New Year 2025! I'm just back from vacation and trying to try out your branch locally using git_override or local_path_override for giving some earlier feedback if any. Which branch has a possible working solution for MODULE.bazel? I see hermetic-ctk-2, hermetic-breaking-changes as well as tmp. Let me know what's the best way to try this out on our RBE setup.

[cloudhan]

@udaya2899 I updated previous comment. The branchs are stacked one by one, so blindly pick the last one should be OK.

[cloudhan]

You can also find MODULE base config in the referenced cuda-samples repo.

[cloudhan]

Auto config with redistrib.json in MODULE based project is not implemented at the moment. Maybe in future PRs. Another unsolved feature is how can we make switch cuda version easier. Say export or maybe a flag to build against different releases of cuda. A possible solution is to extend the current alias mapping to a versioned mapping with select in between.