NVProfilerService assertion failed at preModuleBeginStream

[czangela]

The NVProfilerService crashes at

cmsRun: /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_11_3_0_pre4-slc7_amd64_gcc900/build/CMSSW_11_3_0_pre4
-build/tmp/BUILDROOT/d54b69ba517bf01e04ce547c2ddffd98/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_11_3_0_pre4/
src/HeterogeneousCore/CUDAServices/plugins/NVProfilerService.cc:618: 
void NVProfilerService::preModuleBeginStream(const edm::StreamContext&, const edm::ModuleCallingContext&): 
Assertion `stream_modules_[sid][mid] == nvtxInvalidRangeId' failed.

For full backtrace see:
nvprofilercrash.txt

I was trying to run it on the machine cmg-gpu1080.cern.ch.
I used cmsCudaSetup.sh before running cmsRun, which set up everything just fine:
cmscudasetup.txt

The same bug appeared at this validation, see unsuccessful profile.py logs.

Since that, @fwyzard encountered the same problem.

[cmsbuild]

A new Issue was created by @czangela .

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[fwyzard]

assign heterogeneous

[cmsbuild]

New categories assigned: heterogeneous

@makortel,@fwyzard you have been requested to review this Pull request/Issue and eventually sign? Thanks