Replicate SDK folder for using clang as front-end

[dbeurle]

Hi again!

Would it be possible to add a shim package that collates the CUDA bits so clang can be used instead of nvcc as a front end?

Relevant links:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=882505
https://bugs.llvm.org/show_bug.cgi?id=26966
https://llvm.org/docs/CompileCudaWithLLVM.html

[scaronni]

I'm about to roll out CUDA 9.2 (testing building other components), and I would like to address this.

Just to understand, you are asking for a compatibility package that just mimics the original structure of the CUDA installation?

[dbeurle]

Basically yeah. Clang just expects whatever the stock nvidia installer spits out.

[scaronni]

So the include dir is the only difference. Assuming $CUDA_ROOT is now /usr, it goes from $CUDA_ROOT/include to $CUDA_ROOT/include/cuda. How does Clang play with the variables that are currently in /etc/profile.d/*sh?

Before I start modifying packages, can you try with this (example)?

mkdir /opt/cuda
ln -sf /usr/lib64 /opt/cuda/lib64
ln -sf /usr/bin /opt/cuda/bin
ln -sf /usr/include/cuda /opt/cuda/include

Then point everything to /opt/cuda.

Might want to do also CUPTI and nvvm if required, but I doubt:

extras/CUPTI
nvvm

[dbeurle]

Currently echo $CUDA_ROOT doesn't return anything on my machine. The clang site suggests:

When compiling, you may also need to pass --cuda-path=/path/to/cuda if you didn’t install the CUDA SDK into /usr/local/cuda, /usr/local/cuda-7.0, or /usr/local/cuda-7.5.

I'll try the above now.

[scaronni]

Currently echo $CUDA_ROOT doesn't return anything on my machine. The clang site suggests:

Correct, it's not set, was just an example.

[dbeurle]

After setting up the symlinks I get:

$ clang --cuda-path=/opt/cuda cuda_test.cu
clang-6.0: error: cannot find libdevice for sm_20. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.
clang-6.0: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.
clang-6.0: error: cannot find CUDA installation.  Provide its path via --cuda-path, or pass -nocudainc to build without CUDA includes.

and setting up a symlink

sudo ln -s /usr/share/cuda/libdevice.10.bc /opt/cuda/

doesn't help either.

[scaronni]

Clang 6 is not supported even in 9.2:

https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#system-requirements

[dbeurle]

The whole point of using clang is to remove nvcc as the frontend CUDA code compiler. The PTX should be output by the LLVM backend instead of nvcc.

AFAIK the clang project isn't supported by NVIDIA so I think that version of clang refers to the CXX compiler for coupling with nvcc.

[scaronni]

AFAIK the clang project isn't supported by NVIDIA so I think that version of clang refers to the CXX compiler for coupling with nvcc.

You are right sorry. I will make some test and see if I can get a sane result. This is my status at the moment, CUDA is detected but libdevice is not found:

mkdir /opt/cuda/nvvm
ln -sf /usr/lib64 /opt/cuda/lib64
ln -sf /usr/bin /opt/cuda/bin
ln -sf /usr/include/cuda /opt/cuda/include
ln -sf /usr/lib64 /opt/cuda/nvvm/lib64
ln -sf /usr/share/cuda/libdevice /opt/cuda/nvvm/libdevice

$ clang -v --cuda-path=/opt/cuda /usr/share/cuda/samples/0_Simple/cdpSimplePrint/cdpSimplePrint.cu 
clang version 6.0.1 (tags/RELEASE_601/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/8
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-redhat-linux/6.4.0
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-redhat-linux/7.3.0
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/8
Found candidate GCC installation: /usr/lib64/gcc/x86_64-redhat-linux/6.4.0
Found candidate GCC installation: /usr/lib64/gcc/x86_64-redhat-linux/7.3.0
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/8
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
Found CUDA installation: /opt/cuda, version unknown
clang-6.0: error: cannot find libdevice for sm_20. Provide path to different CUDA installation via --cuda-path, or pass -nocudalib to build without linking with libdevice.

[mjmg]

Hi All just noticed something.

I was just wondering why the errors refers to sm_20 which has been officially removed since CUDA 9.X; unless the installed CUDA is 8.X and below.

The references mentioned refers to versions of CUDA that still support compute capability 2.X.

[dbeurle]

True - in clang's wrapper for CUDA I can see

// WARNING: Preprocessor hacks below are based on specific details of
// CUDA-7.x headers and are not expected to work with any other
// version of CUDA headers.
#include "cuda.h"
#if !defined(CUDA_VERSION)
#error "cuda.h did not define CUDA_VERSION"
#elif CUDA_VERSION < 7000 || CUDA_VERSION > 9000                                                                                       
#error "Unsupported CUDA version!"
#endif

but I also see support for CUDA 9 in other places

#if CUDA_VERSION >= 9000
// CUDA-9 has conflicting prototypes for __float2half_rn(float f) in
// cuda_fp16.h[pp] and device_functions.hpp. We need to get the one in
// device_functions.hpp out of the way.
#define __float2half_rn  __float2half_rn_disabled
?#endif

which indicates that CUDA 9 is not yet fully supported.

[scaronni]

Is there any update on this if it is supported or not? Because I'm about to update to CUDA 10...

[dbeurle]

Sorry this issue slipped my mind. What was the current status again? -_-

[dbeurle]

Right - it should be supported if built from source, see https://llvm.org/docs/CompileCudaWithLLVM.html#prerequisites.

I'm away from my workstation with an NVIDIA card so I cannot test until tonight.

[dbeurle]

It seems to work for me when I put in the symlinks from above

clang++ saxpy.cu -o saxpy --cuda-gpu-arch=sm_30 -nocudalib -lcudart_static -ldl -lrt -pthread --cuda-path=/opt/cuda

however if I remove -nocudalib it fails to find libdevice as previously reported. This is with CUDA 10.

[dbeurle]

See https://gitlab.kitware.com/cmake/cmake/issues/16586 for progress with CMake using clang as a CUDA compiler. They are attempting to search for the CUDA toolkit as part of that work.