Discuss ALPAKA_ASSERT_OFFLOAD

[fwyzard]

I find the current implementation of ALPAKA_ASSERT_OFFLOAD very confusing.

It's currently defined as

#if defined(ALPAKA_DEBUG_OFFLOAD_ASSUME_HOST) || defined(SYCL_EXT_ONEAPI_ASSERT)
#    define ALPAKA_ASSERT_OFFLOAD(EXPRESSION) ALPAKA_ASSERT(EXPRESSION)
#elif defined __AMDGCN__ && (!defined NDEBUG)
#    define ALPAKA_ASSERT_OFFLOAD(EXPRESSION)                                         \
        do                                                                            \
        {                                                                             \
            if(!(EXPRESSION))                                                         \
                __builtin_trap();                                                     \
        } while(false)
#else
#    define ALPAKA_ASSERT_OFFLOAD(EXPRESSION)                                         \
        do                                                                            \
        {                                                                             \
        } while(false)
#endif

My understanding is that

• while building CPU code - including code for CPU-based accelerators - ALPAKA_ASSERT_OFFLOAD() becomes assert(), that is controlled by NDEBUG;
• while building CUDA code, ALPAKA_ASSERT_OFFLOAD() is always disabled;
• while building HIP/ROCm code, ALPAKA_ASSERT_OFFLOAD() is controlled by NDEBUG;
• while building SYCL/oneAPI code, ALPAKA_ASSERT_OFFLOAD() becomes assert(), that is controlled by NDEBUG.

However, if ALPAKA_DEBUG_OFFLOAD_ASSUME_HOST is defined, ALPAKA_ASSERT_OFFLOAD() always becomes assert(), and is controlled by NDEBUG.

I find two aspects problematic:

• ALPAKA_ASSERT_OFFLOAD() is disabled by default for CUDA targets, and it can only be enabled by setting ALPAKA_DEBUG_OFFLOAD_ASSUME_HOST;
• ALPAKA_ASSERT_OFFLOAD() cannot be disabled for GPU targets without setting NDEBUG, which disabled asserts also for the standard CPU compilation.

In the CMS software we are using a somewhat different approach:

• assert() are always enabled for CPU code (because we never define NDEBUG);
• assert() are disabled by default in CUDA code, and can be enabled by setting the GPU_DEBUG macro.

The rationale is that in host code assert() are cheap enough and the benefit from the error checking outweighs the minor performance hit we may have, while in CUDA code the impact can be significant¹ so we enable assert() only while debugging.

As we are porting our code from CUDA to Alpaka, we are looking to implement a similar functionality:

• assert() are always enabled for CPU code (because we never define NDEBUG);
• assert() are disabled by default in CUDA, HIP/ROCm and SYCL/oneAPI code, and can be enabled setting an appropriate preprocessor macro.

Is this an approach that would be useful in general for Alpaka ?

If so, we can make a PR to rework ALPAKA_ASSERT_OFFLOAD(); otherwise we can implement it in CMSSW.

Footnotes

1. this is due in part due to the extra checks and trap() functionality, in part due to the impact on register usage caused by potentially printing messages. ↩

[psychocoderHPC]

IMO the strange behavior is a BUG and should be fixed, the macro should be guarded by NDEBUG

[fwyzard]

If it is guarded only by NDEBUG, should it replace ALPAKA_ASSERT ?

[fwyzard]

Looks like I'm a repetitive person... this is mostly a duplicate of #2001 .

[psychocoderHPC]

If it is guarded only by NDEBUG, should it replace ALPAKA_ASSERT ?

I think ALPAKA_ASSERT_OFFLOAD should be removed and ALPAKA_ASSERT should be used instead. The offload version was a workaround from when we added OpenAcc and OpenMP offload. IMO the OFFLOAD version is not required anymore.

[psychocoderHPC]

I opened #2199 to solve the issue.