CRITICAL INTERNAL FASTJET ERROR in 2024 data reprocessing

[vlimant]

casually looking into the 2024 CDE data reprocessing, I came across a workflow that segfault

https://cmsweb.cern.ch/reqmgr2/fetch?rid=pdmvserv_Run2024D_JetMET1_2024CDEReprocessing_241219_083401_7231

https://dmytro.web.cern.ch/dmytro/cmsprodmon/workflows.php?prep_id=ReReco-Run2024D-JetMET1-2024CDEReprocessing-00001

https://cms-unified.web.cern.ch/cms-unified/report/pdmvserv_Run2024D_JetMET1_2024CDEReprocessing_241219_083401_7231

https://cms-unified.web.cern.ch/cms-unified/joblogs/pdmvserv_Run2024D_JetMET1_2024CDEReprocessing_241219_083401_7231/8901/DataProcessing/

in particular that one:

https://cms-unified.web.cern.ch/cms-unified/joblogs/pdmvserv_Run2024D_JetMET1_2024CDEReprocessing_241219_083401_7231/8901/DataProcessing/39c18e3a-4a95-4ee9-862d-5fda51287f53-61-3-logArchive/job/WMTaskSpace/cmsRun1/cmsRun1-stdout.log

Begin processing the 5578th record. Run 380401, Event 6067611, LumiSection 5 on stream 7 at 26-Dec-2024 09:44:48.925 UTC
Begin processing the 5579th record. Run 380401, Event 6276920, LumiSection 5 on stream 2 at 26-Dec-2024 09:44:49.644 UTC
fastjet::Error:  *** CRITICAL INTERNAL FASTJET ERROR *** CONTACT THE AUTHORS *** trying to recomine an object that has previsously been recombined


A fatal system signal has occurred: segmentation violation
The following is the call stack containing the origin of the signal.

one may get the configuration from https://cms-unified.web.cern.ch/cms-unified/joblogs/pdmvserv_Run2024D_JetMET1_2024CDEReprocessing_241219_083401_7231/8901/DataProcessing/39c18e3a-4a95-4ee9-862d-5fda51287f53-61-3-logArchive/job/WMTaskSpace/cmsRun1/ PSet.py and PSet.pkl

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @vlimant.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[Dr15Jones]

assign reconstruction

[cmsbuild]

New categories assigned: reconstruction

@jfernan2,@mandrenguyen you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

The relevant traceback for the segmentation fault is

#4  <signal handler called>
#5  fastjet::SharedPtr<fastjet::PseudoJet::UserInfoBase>::__SharedCountingPtr::operator-- (this=0x700000000) at ./../include/fastjet/SharedPtr.hh:556
#6  fastjet::SharedPtr<fastjet::PseudoJet::UserInfoBase>::_decrease_count (this=0x147ea2ace640) at ./../include/fastjet/SharedPtr.hh:587
#7  fastjet::SharedPtr<fastjet::PseudoJet::UserInfoBase>::~SharedPtr (this=0x147ea2ace640, __in_chrg=<optimized out>) at ./../include/fastjet/SharedPtr.hh:377
#8  fastjet::PseudoJet::~PseudoJet (this=0x147ea2ace630, __in_chrg=<optimized out>) at ./../include/fastjet/PseudoJet.hh:104
#9  std::_Destroy<fastjet::PseudoJet> (__pointer=0x147ea2ace630) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2-build/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_construct.h:151
#10 std::_Destroy_aux<false>::__destroy<fastjet::PseudoJet*> (__last=<optimized out>, __first=0x147ea2ace630) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2-build/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_construct.h:163
#11 std::_Destroy<fastjet::PseudoJet*> (__last=<optimized out>, __first=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2-build/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_construct.h:196
#12 std::_Destroy<fastjet::PseudoJet*, fastjet::PseudoJet> (__last=0x147ea2b29de0, __first=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2-build/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/alloc_traits.h:850
#13 std::vector<fastjet::PseudoJet, std::allocator<fastjet::PseudoJet> >::~vector (this=0x1480cdbd60d8, __in_chrg=<optimized out>) at /data/cmsbld/jenkins/workspace/auto-builds/CMSSW_14_0_0_pre2-el8_amd64_gcc12/build/CMSSW_14_0_0_pre2-build/el8_amd64_gcc12/external/gcc/12.3.1-40d504be6370b5a30e3947a6e575ca28/include/c++/12.3.1/bits/stl_vector.h:730
#14 fastjet::ClusterSequence::~ClusterSequence (this=0x1480cdbd6080, __in_chrg=<optimized out>) at ClusterSequence.cc:183
#15 0x00001480c9edd940 in fastjet::ClusterSequenceActiveArea::_run_AA(fastjet::GhostedAreaSpec const&) [clone .cold] () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw-patch/CMSSW_14_0_19_patch2/external/el8_amd64_gcc12/lib/libfastjet.so.0
#16 0x00001480c9f13623 in fastjet::ClusterSequenceActiveArea::_initialise_and_run_AA (this=0x14802b274880, jet_def_in=..., ghost_spec=..., writeout_combinations=<optimized out>) at ClusterSequenceActiveArea.cc:61
#17 0x0000148099509c24 in void fastjet::ClusterSequenceArea::initialize_and_run_cswa<fastjet::PseudoJet>(std::vector<fastjet::PseudoJet, std::allocator<fastjet::PseudoJet> > const&, fastjet::JetDefinition const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_19/lib/el8_amd64_gcc12/pluginRecoJetsJetProducers_plugins.so
#18 0x000014809950a7d9 in fastjet::ClusterSequenceArea::ClusterSequenceArea<fastjet::PseudoJet>(std::vector<fastjet::PseudoJet, std::allocator<fastjet::PseudoJet> > const&, fastjet::JetDefinition const&, fastjet::AreaDefinition const&) [clone .lto_priv.0] () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_19/lib/el8_amd64_gcc12/pluginRecoJetsJetProducers_plugins.so
#19 0x0000148099522fa7 in FastjetJetProducer::runAlgorithm(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_19/lib/el8_amd64_gcc12/pluginRecoJetsJetProducers_plugins.so
#20 0x0000148099563698 in VirtualJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_19/lib/el8_amd64_gcc12/pluginRecoJetsJetProducers_plugins.so
#21 0x000014809951cc1d in FastjetJetProducer::produce(edm::Event&, edm::EventSetup const&) () from /cvmfs/cms.cern.ch/el8_amd64_gcc12/cms/cmssw/CMSSW_14_0_19/lib/el8_amd64_gcc12/pluginRecoJetsJetProducers_plugins.so

[jfernan2]

@nurfikri89 as JetMET RECO contact, can you please have a look and comment? Thanks

[nurfikri89]

@jfernan2 I just saw this, sorry. Did not get notification probably due to typo. Will look into it.