Pythia8EGun generator module requires synchronization on LuminosityBlock boundaries

[jordan-martins]

We have 2 Guns (which we know does not work with the concurrent modules) samples from the Run3Winter22 campaign that are failing 100% at the GEN step with a very odd error to us:

Exception Message:
The framework is configured to use at least two streams, but the following modules
require synchronizing on LuminosityBlock boundaries:
Pythia8EGun generator

It seems that it has some stream defined anywhere but we (PdmV nor PnR) can not identify from where this config is set up (and we haven't set up anything on our own). Here are the two wfs:

wf1
wf2

Turnaround proposed to deal with the problem is:

The situation can be fixed by either
* modifying the modules to support concurrent LuminosityBlocks (preferred), or
* setting 'process.options.numberOfConcurrentLuminosityBlocks = 1' in the configuration file

However, we rather prefer to see the fix from the module perspective if possible.

In addition, the error is not reproducible from lxplus nor validation jobs from McM tool.

Thanks,
Jordan

[cmsbuild]

A new Issue was created by @jordan-martins Jordan Martins.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign generators

[cmsbuild]

New categories assigned: generators

@mkirsano,@alberto-sanchez,@SiewYan,@GurpreetSinghChahal,@Saptaparna you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

The Pythia8EGun is declared as one of the module types for which cmsDriver should explicitly set the numberOfConcurrentLuminosityBlocks = 1

cmssw/Configuration/Generator/python/concurrentLumisDisable.py

Lines 1 to 11 in 7f2d0d8

	# list of generator EDModules (C++ type) that do not support concurrentLuminosityBlocks
	noConcurrentLumiGenerators = [
	"AMPTGeneratorFilter",
	"BeamHaloProducer",
	"CosMuoGenProducer",
	"Herwig7GeneratorFilter",
	"HydjetGeneratorFilter",
	"Hydjet2GeneratorFilter",
	"PyquenGeneratorFilter",
	"Pythia6GeneratorFilter",
	"Pythia8EGun",

so in that sense this behavior is expected (more details on "why" in #25090). It is currently unclear what overrides this parameter for these workflows (investigation continues in https://mattermost.web.cern.ch/cms-o-and-c/pl/sw55fqpr7pna8x581q453wcsne).

Nevertheless it would be better to migrate these generator modules to support concurrent lumis :)

[makortel]

The problem in ConfigBuilder is fixed in #37417. I'll make backports to 12_3_X and 12_2_X after the review of that has completed.