Extend CMSSW to a distributed application over MPI #32632
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
A new Pull Request was created by @fwyzard (Andrea Bocci) for CMSSW_11_2_X. It involves the following packages: HeterogeneousCore/MPICore @makortel, @cmsbuild, @fwyzard can you please review it and eventually sign? Thanks. cms-bot commands are listed here |
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32632/20729 ERROR: Build errors found during clang-tidy run. |
|
please test |
cmsbuild
added
code-checks-pending
tests-started
and removed
code-checks-rejected
tests-pending
labels
|
-code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32632/20747 ERROR: Build errors found during clang-tidy run. |
|
please test |
|
+1 Size: This PR adds an extra 40KB to repository Comparison SummarySummary:
|
Note: this is a quick workaround to let the device code use the device collection, while being able to access the actual number of pf rechits on the host side. It should replaced with a better and more general implementation, and the use of the host collection should be removed.
Let multiple CMSSW processes on the same or different machines coordinate event processing and transfer data products over MPI. The implementation is based on four CMSSW modules. Two are responsible for setting up the communication channels and coordinate the event processing: - the MPIController - the MPISource and two are responsible for the transfer of data products: - the MPISender - the MPIReceiver . The MPIController is an EDProducer running in a regular CMSSW process. After setting up the communication with an MPISource, it transmits to it all EDM run, lumi and event transitions, and instructs the MPISource to replicate them in the second process. The MPISource is a Source controlling the execution of a second CMSSW process. After setting up the communication with an MPIController, it listens for EDM run, lumi and event transitions, and replicates them in its process. Both MPIController and MPISource produce an MPIToken, a special data product that encapsulates the information about the MPI communication channel. The MPISender is an EDProducer that can read a collection of a predefined type from the Event, serialise it using its ROOT dictionary, and send it over the MPI communication channel. The MPIReceiver is an EDProducer that can receive a collection of a predefined type over the MPI communication channel, deserialise is using its ROOT dictionary, and put it in the Event. Both MPISender and MPIReceiver are templated on the type to be transmitted and de/serialised. Each MPISender and MPIReceiver is configured with an instance value that is used to match one MPISender in one process to one MPIReceiver in another process. Using different instance values allows the use of multiple MPISenders/MPIReceivers in a process. Both MPISender and MPIReceiver obtain the MPI communication channel reading an MPIToken from the event. They also produce a copy of the MPIToken, so other modules can consume it to declare a dependency on the previous modules. An automated test is available in the test/ directory.
Let MPISender and MPIReceiver consume, send/receive and produce collections of arbitrary types, as long as they have a ROOT dictionary and can be persisted. Note that any transient information is lost during the transfer, and needs to be recreated by the receiving side. The documentation and tests are updated accordingly. Warning: this approach is a work in progress! TODO: - improve framework integration - add checks between send/receive types
cmsbuild
added
reconstruction-pending
tests-pending
code-checks-pending
and removed
tests-approved
code-checks-approved
labels
|
+code-checks Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-32632/42857
|
|
Pull request #32632 was updated. @Dr15Jones, @cmsbuild, @fwyzard, @jfernan2, @makortel, @mandrenguyen, @smuzaffar can you please check and sign again. |
|
please test |
|
+1 Size: This PR adds an extra 28KB to repository Comparison SummarySummary:
|
|
type ngt |
This was referenced Dec 11, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR description:
Let multiple CMSSW processes on the same or different machines coordinate event processing and transfer data products over MPI.
The implementation is based on four CMSSW modules.
Two are responsible for setting up the communication channels and coordinate the event processing:
MPIControllerMPISourceand two are responsible for the transfer of data products:
MPISenderMPIReceiver.
The
MPIControlleris anEDProducerrunning in a regular CMSSW process. After setting up the communication with anMPISource, it transmits to it all EDM run, lumi and event transitions, and instructs theMPISourceto replicate them in the second process.The
MPISourceis aSourcecontrolling the execution of a second CMSSW process. After setting up the communication with anMPIController, it listens for EDM run, lumi and event transitions, and replicates them in its process.Both
MPIControllerandMPISourceproduce anMPIToken, a special data product that encapsulates the information about the MPI communication channel.The
MPISenderis anEDProducerthat can read one or more collections from the Event, serialise them using their ROOT dictionaries, and send them over the MPI communication channel.The
MPIReceiveris anEDProducerthat can receive a set number of collections over the MPI communication channel, deserialise them using their ROOT dictionaries, and put them in the Event with a configurable instance label.In principle any non-transient collection with a ROOT dictionary can be transmitted. Any transient information is lost during the transfer, and needs to be recreated by the receiving side.
Each
MPISenderandMPIReceiveris configured with an instance value that is used to match oneMPISenderin one process to oneMPIReceiverin another process. Using different instance values allows the use of multipleMPISenders/MPIReceiversin a process.Both
MPISenderandMPIReceiverobtain the MPI communication channel reading anMPITokenfrom the event. They also produce a copy of theMPIToken, so other modules can consume it to declare a dependency on the previous modules.An automated test is available in the
test/directory.Current limitations
MPIDriveris a "one" module that supports only a single luminosity block at a time;MPISenderandMPIReceiversupport a single compile-time type;MPISendermatches the type expected by theMPIReceiver.Expected future developments
PortableCollectiontypes;MPISenderand the one expected by theMPIReceivermatch;extend theMPISenderandMPIReceiverto send and receive multiple collections;rewrite theMPISenderandMPIReceiverto send and receive arbitrary run-time collections;MPIControllerto be aglobalmodule rather than aonemodule;MPISourceaccept connections and events from multipleMPIControllermodules in different jobs;MPIControllerconnect and sent events to multipleMPISourcemodules in different jobs;ProcessingHistoryfrom theMPIControllerto theMPISource? and vice-versa ?MPIControllerto theMPISource? and vice-versa ?ProcessingHistoryas the ongoing run ?PR validation:
This PR includes an automated unit test.