[TF] Update Tensorflow to 2.17.0 build with c++20 standard #9438

smuzaffar · 2024-09-27T12:54:43Z

This PR updates tensorflow to version 2.17.0 for special TF_X IBs. It also contains following changes

Bazel: Update to 6.5.0
Eigen: https://gitlab.com/libeigen/eigen/-/commit/c1d637433e3b3f9012b226c2c9125c494b470ae6 used by new TF 2.17.0
- Thsi also container CMS changes : cms-externals/eigen-git-mirror@3cbe8e7
flatbuffers: 24.3.25 (TF 2.17.0 requires version >=24)
tensorboard: 2.18.0
cleanup TF spec to not hand pick headers and libs. Just use what is distributed by TF wheel
- cleanup TF scram tools (removed tools which are not in use)
- Apply fixes to build TF with c++20
Updated tensorflow-xla-runtime, tensorflow and tfaot build recipes

cmsbuild · 2024-09-27T12:55:06Z

A new Pull Request was created by @smuzaffar for branch IB/CMSSW_14_2_X/tf.

@aandvalenzuela, @iarspider, @smuzaffar can you please review it and eventually sign? Thanks.
@antoniovilela, @mandrenguyen, @rappoccio, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2024-09-27T12:55:07Z

cms-bot internal usage

smuzaffar · 2024-09-27T12:55:29Z

test parameters:

full_cmssw = true

smuzaffar · 2024-09-27T12:55:37Z

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

smuzaffar · 2024-09-27T13:32:49Z

@fwyzard , our eigen patch cms-externals/eigen-git-mirror@3cbe8e7 is getting bigger and bigger and harder to apply ( mostly due to eigen code formatting). Do you think these EIGEN_DEVICE_FUNC type additions are really needed? should we try building without our patch ?

cmsbuild · 2024-09-28T04:30:36Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41784/summary.html
COMMIT: 5e451b0
CMSSW: CMSSW_14_2_TF_X_2024-09-24-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9438/41784/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41784/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41784/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test test_G2GainsValidator had ERRORS

Comparison Summary

Summary:

You potentially removed 113 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 24569 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3331063
DQMHistoTests: Total failures: 93667
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3237375
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 119.3020000000001 KiB( 43 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 1.715 KiB OfflinePV/Alignment
DQMHistoSizes: changed ( 12834.0,... ): 5.430 KiB HLT/Vertexing
DQMHistoSizes: changed ( 140.063 ): 0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 24834.911,... ): 3.590 KiB HLT/Vertexing
Checked 193 log files, 163 edm output root files, 44 DQM output files
TriggerResults: found differences in 9 / 42 workflows

smuzaffar · 2024-09-28T09:23:07Z

enable gpu

smuzaffar · 2024-09-28T09:23:25Z

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

smuzaffar · 2024-09-28T09:23:39Z

please test for CMSSW_14_2_TF_X/el8_aarch64_gcc12

smuzaffar · 2024-09-28T09:24:53Z

please test for CMSSW_14_2_TF_X/slc7_amd64_gcc12

cmsbuild · 2024-09-28T12:36:03Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41803/summary.html
COMMIT: 5e451b0
CMSSW: CMSSW_14_2_TF_X_2024-09-24-1100/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9438/41803/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

+ find bazel-out/-opt/bin -path '*/external/nsync/libnsync_cpp.pic.a'
+ xargs --no-run-if-empty -i cp '{}' /data/cmsbld/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/d4bd83914671513f28ce8f407c5b7724/opt/cmssw/el8_aarch64_gcc12/external/tensorflow-sources/2.17.0-d4bd83914671513f28ce8f407c5b7724/lib-xla-runtime
find: 'bazel-out/-opt/bin': No such file or directory
+ for lib in libfft.pic.a libfft_wrapper.pic.a libmutex.pic.a libnsync_cpp.pic.a
+ test -e /data/cmsbld/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/BUILDROOT/d4bd83914671513f28ce8f407c5b7724/opt/cmssw/el8_aarch64_gcc12/external/tensorflow-sources/2.17.0-d4bd83914671513f28ce8f407c5b7724/lib-xla-runtime/libfft.pic.a
error: Bad exit status from /data/cmsbld/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.p1JiI1 (%install)


RPM build errors:
line 42: It's not recommended to have unversioned Obsoletes: Obsoletes: external+tensorflow-sources+2.17.0-d4bd83914671513f28ce8f407c5b7724
Bad exit status from /data/cmsbld/jenkins_a/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.p1JiI1 (%install)

cmsbuild · 2024-09-28T13:19:16Z

Pull request #9438 was updated.

smuzaffar · 2024-09-28T13:19:17Z

please test for CMSSW_14_2_TF_X/el8_aarch64_gcc12

cmsbuild · 2024-09-28T16:46:32Z

Pull request #9438 was updated.

smuzaffar · 2024-09-28T16:46:36Z

please test for CMSSW_14_2_TF_X/el8_amd64_gcc12

cmsbuild · 2024-09-29T06:50:03Z

-1

Failed Tests: UnitTests GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41807/summary.html
COMMIT: 03bb073
CMSSW: CMSSW_14_2_TF_X_2024-09-24-1100/el8_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9438/41807/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41807/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41807/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test test_G2GainsValidator had ERRORS

GPU Unit Tests

I found 11 errors in the following unit tests:

---> test simpleCholeskyTest had ERRORS
---> test testTFGraphLoadingCUDA had ERRORS
---> test testTFConstSessionCUDA had ERRORS
and more ...

Comparison Summary

Summary:

You potentially removed 112 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 24567 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3331063
DQMHistoTests: Total failures: 93683
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3237359
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 119.3020000000001 KiB( 43 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 1.715 KiB OfflinePV/Alignment
DQMHistoSizes: changed ( 12834.0,... ): 5.430 KiB HLT/Vertexing
DQMHistoSizes: changed ( 140.063 ): 0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 24834.911,... ): 3.590 KiB HLT/Vertexing
Checked 193 log files, 163 edm output root files, 44 DQM output files
TriggerResults: found differences in 9 / 42 workflows

smuzaffar · 2024-09-29T10:12:09Z

please test for CMSSW_14_2_TF_X/el8_aarch64_gcc12

smuzaffar · 2024-09-29T10:12:33Z

please test for CMSSW_14_2_TF_X/slc7_amd64_gcc12

cmsbuild · 2024-09-29T13:47:43Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41809/summary.html
COMMIT: 03bb073
CMSSW: CMSSW_14_2_TF_X_2024-09-24-1100/el8_aarch64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9438/41809/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41809/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41809/git-merge-result

Unit Tests

I found 2 errors in the following unit tests:

---> test testAOTTools had ERRORS
---> test test_G2GainsValidator had ERRORS

smuzaffar · 2024-09-29T20:31:51Z

looks good , thought gpu TF unit tests ae failing

cmsbuild · 2024-09-30T05:16:06Z

-1

Failed Tests: UnitTests GpuUnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41810/summary.html
COMMIT: 03bb073
CMSSW: CMSSW_14_2_TF_X_2024-09-24-1100/slc7_amd64_gcc12
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/9438/41810/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41810/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-5cc892/41810/git-merge-result

Unit Tests

I found 1 errors in the following unit tests:

---> test test_G2GainsValidator had ERRORS

GPU Unit Tests

I found 13 errors in the following unit tests:

---> test simpleCholeskyTest had ERRORS
---> test testCUDAService had ERRORS
---> test testTorchSimpleDnnCUDA had ERRORS
and more ...

Comparison Summary

Summary:

You potentially removed 889 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 69153 differences found in the comparisons
DQMHistoTests: Total files compared: 44
DQMHistoTests: Total histograms compared: 3331063
DQMHistoTests: Total failures: 522287
DQMHistoTests: Total nulls: 336
DQMHistoTests: Total successes: 2808420
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 125.88700000000009 KiB( 43 files compared)
DQMHistoSizes: changed ( 1000.0,... ): 1.715 KiB OfflinePV/Alignment
DQMHistoSizes: changed ( 12834.0,... ): 5.430 KiB HLT/Vertexing
DQMHistoSizes: changed ( 13034.0 ): 5.658 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 140.063 ): 0.012 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.042 ): 0.008 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.044 ): 0.020 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 141.046 ): -0.004 KiB JetMET/SUSYDQM
DQMHistoSizes: changed ( 24834.911,... ): 3.590 KiB HLT/Vertexing
DQMHistoSizes: changed ( 250202.181 ): 0.117 KiB SiStrip/MechanicalView
DQMHistoSizes: changed ( 25202.0 ): 0.787 KiB SiStrip/MechanicalView
Checked 193 log files, 163 edm output root files, 44 DQM output files
TriggerResults: found differences in 17 / 42 workflows

Update Tensorflow to 2.17.0

5e451b0

cmsbuild added externals-pending pending-signatures tests-started orp-pending labels Sep 27, 2024

cmsbuild added tests-rejected and removed tests-started labels Sep 28, 2024

cmsbuild added tests-started and removed tests-rejected labels Sep 28, 2024

tensorflow: typo for non-x86_64

b0c2507

cmsbuild added tests-pending and removed tests-started labels Sep 28, 2024

cmsbuild added tests-started and removed tests-pending labels Sep 28, 2024

set tensorflow libs permission so that rpm can scan these for provides

03bb073

cmsbuild added tests-pending and removed tests-started labels Sep 28, 2024

cmsbuild added tests-started and removed tests-pending labels Sep 28, 2024

cmsbuild added tests-rejected and removed tests-started labels Sep 29, 2024

cmsbuild added tests-started and removed tests-rejected labels Sep 29, 2024

smuzaffar merged commit 917ef6a into IB/CMSSW_14_2_X/tf Sep 29, 2024
18 of 24 checks passed

cmsbuild added tests-rejected and removed tests-started labels Sep 30, 2024

smuzaffar deleted the tf-2.17.0 branch October 8, 2024 08:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[TF] Update Tensorflow to 2.17.0 build with c++20 standard #9438

[TF] Update Tensorflow to 2.17.0 build with c++20 standard #9438

smuzaffar commented Sep 27, 2024

cmsbuild commented Sep 27, 2024

cmsbuild commented Sep 27, 2024 •

edited

Loading

smuzaffar commented Sep 27, 2024

smuzaffar commented Sep 27, 2024

smuzaffar commented Sep 27, 2024

cmsbuild commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

smuzaffar commented Sep 28, 2024 •

edited

Loading

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 28, 2024

cmsbuild commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 29, 2024

smuzaffar commented Sep 29, 2024

smuzaffar commented Sep 29, 2024

cmsbuild commented Sep 29, 2024

smuzaffar commented Sep 29, 2024

cmsbuild commented Sep 30, 2024

[TF] Update Tensorflow to 2.17.0 build with c++20 standard #9438

[TF] Update Tensorflow to 2.17.0 build with c++20 standard #9438

Conversation

smuzaffar commented Sep 27, 2024

cmsbuild commented Sep 27, 2024

cmsbuild commented Sep 27, 2024 • edited Loading

smuzaffar commented Sep 27, 2024

smuzaffar commented Sep 27, 2024

smuzaffar commented Sep 27, 2024

cmsbuild commented Sep 28, 2024

Unit Tests

Comparison Summary

smuzaffar commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

smuzaffar commented Sep 28, 2024 • edited Loading

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 28, 2024

External Build

cmsbuild commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 28, 2024

smuzaffar commented Sep 28, 2024

cmsbuild commented Sep 29, 2024

Unit Tests

GPU Unit Tests

Comparison Summary

smuzaffar commented Sep 29, 2024

smuzaffar commented Sep 29, 2024

cmsbuild commented Sep 29, 2024

Unit Tests

smuzaffar commented Sep 29, 2024

cmsbuild commented Sep 30, 2024

Unit Tests

GPU Unit Tests

Comparison Summary

cmsbuild commented Sep 27, 2024 •

edited

Loading

smuzaffar commented Sep 28, 2024 •

edited

Loading