Use cleaned PF candidates in UE subtraction in HI miniAOD (112X)

[mandrenguyen]

PR description:

In #31668 a cleaning procedure was added to HI miniAOD to remove badly reconstructed particles. This procedure was applied prior to jet clustering (akCs4PFJets), but not applied to the towering algorithm (PFTowers) that is used for the UE subtraction. This leads to an inconsistency with our previous AOD-only workflow. The cleaning is only effective occasionally such that the net effect is a small migration of in the jet pt spectrum.

PR validation:

Wfs 140.5611 and 158.01

if this PR is a backport please specify the original PR and why you need to backport that PR:

Backport of #32325
Needed for HI reMiniAOD in 11_2_X

[cmsbuild]

A new Pull Request was created by @mandrenguyen (Matthew Nguyen) for CMSSW_11_2_X.

It involves the following packages:

RecoHI/HiJetAlgos

@perrotta, @jpata, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@yslai, @jazzitup, @yenjie, @kurtejung, @dgulhan, @yetkinyilmaz this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

• Backported from Use cleaned PF candidates in UE subtraction in HI miniAOD #32325

[slava77]

@cmsbuild please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-12-01-1100/slc7_amd64_gcc900: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/11233/console Started: 2020/12/01 19:05

[slava77]

backport of #32325

[cmsbuild]

+1
Tested at: 82942e0
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5bd579/11233/summary.html
CMSSW: CMSSW_11_2_X_2020-12-01-1100
SCRAM_ARCH: slc7_amd64_gcc900

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5bd579/11233/summary.html

There are some workflows for which there are errors in the baseline:
4.53 step 2
136.793 step 2
136.874 step 2
4.22 step 2
136.88811 step 2
1000.0 step 2
1001.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

• file /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/logRootQA.log matches

[slava77]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5bd579/11233/summary.html

There are some workflows for which there are errors in the baseline:
4.53 step 2
136.793 step 2
136.874 step 2
4.22 step 2
136.88811 step 2
1000.0 step 2
1001.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

* file /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/logRootQA.log matches

the summary looks kind of broken.

@smuzaffar from the recent updates I could possibly blame my cms-sw/cms-bot#1430, but it's odd considering that the results looked fine in the PR tests
Did anything else break?

[slava77]

@smuzaffar from the recent updates I could possibly blame my cms-sw/cms-bot#1430, but it's odd considering that the results looked fine in the PR tests

I see some working PR summaries, e.g. in cms-sw/cmsdist#6474 (comment), which was triggered after cms-sw/cms-bot#1430 was merged; so, at least logRootQA is not fully broken

[smuzaffar]

@slava77 , today there was cmseos issues and baseline had few errors [a]. cms-sw/cmsdist#6474 used an older IB CMSSW_10_2_X_2020-11-30-1100 thats why it work.

[a]
5 27 26 19 13 3 0 0 0 tests passed, 0 7 0 0 0 0 0 0 0 failed

[slava77]

Comparison Summary:

* file /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/logRootQA.log matches

I'm guessing that some special characters in https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_11_2_X_2020-12-01-1100+5bd579/40234/validateJR/logRootQA.log
are breaking the grep call at https://github.com/cms-sw/cms-bot/blob/96a624848b1c23d02763f020df2f6ee417aab6b4/run-pr-comparisons#L296

[cmsbuild]

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cms-bot#1436

• CMSSW_11_2_X_2020-12-01-1100/slc7_amd64_gcc900: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/11243/console Started: 2020/12/01 23:46

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_11_2_X IBs after it passes the integration tests and once validation in the development release cycle CMSSW_11_3_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

[cmsbuild]

-1

Tested at: 82942e0

CMSSW: CMSSW_11_2_X_2020-12-01-1100
SCRAM_ARCH: slc7_amd64_gcc900
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-09cd6c/11243/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests

• Unit Tests:

I found errors in the following unit tests:

---> test runtestPhysicsToolsPatAlgos had ERRORS

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-09cd6c/11243/summary.html

There are some workflows for which there are errors in the baseline:
4.53 step 2
136.793 step 2
136.874 step 2
4.22 step 2
136.88811 step 2
1000.0 step 2
1001.0 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

• You potentially added 2968 lines to the logs
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 28
• DQMHistoTests: Total histograms compared: 2226842
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2226825
• DQMHistoTests: Total skipped: 16
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 27 files compared)
• Checked 128 log files, 27 edm output root files, 28 DQM output files

[silviodonato]

please test

[cmsbuild]

The tests are being triggered in jenkins.

• CMSSW_11_2_X_2020-12-02-1100/slc7_amd64_gcc900: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/11256/console Started: 2020/12/02 12:58

[cmsbuild]

+1
Tested at: 82942e0
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5bd579/11256/summary.html
CMSSW: CMSSW_11_2_X_2020-12-02-1100
SCRAM_ARCH: slc7_amd64_gcc900

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5bd579/11256/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 35
• DQMHistoTests: Total histograms compared: 2529593
• DQMHistoTests: Total failures: 9
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 2529561
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 34 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 148 log files, 37 edm output root files, 35 DQM output files

[qliphy]

+1