CPU integration of G4HepEM using the tracking manager #316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
Yes, the tracking manager does not produce bit-identical results. That's because of some optimizations (internal MSC stepping, which can be turned off) and also because we do the navigation directly, with potentially different rounding than |
|
I see, that's good to know. I have done a run of 32 ttbar events in CMS, the average difference in total edep is around 3%, though the highest was 7%. The speedup from using the tracking manager was 1.09. |
In principle we don't expect a difference in energy deposit... Are 32 ttbar events enough to get stable numbers for validation? 10% speedup from the tracking manager sounds in line with expectations. |
JuanGonzalezCaminero
force-pushed
the
hepem_trackingmanager
branch
from
fdb01f3 to
cc2e232
Compare
5 tasks
agheata
pushed a commit
that referenced
this pull request
Jan 7, 2025
This PR introduces the required changes needed to use the latest version of G4HepEm. In G4HepEm the representation for the cross sections changed in this [PR](mnovak42/g4hepem#115). Additionally, the e-+/gamma-nuclear reactions were added in G4HepEm. This PR 1. uses the new function `G4HepEmGammaManager::SampleInteraction` and fixes the numIALeft. **_Note_**: that since we don't register the gamma/lepton nuclear processes in the physics list they are currently ignored by AdePT and G4HepEm. 2. adopts the TrackingManager approach by using a derived class of the G4HepEmTrackingManager. This supersedes #316. The G4HepEmTrackingManagerSpecialized is instantiated as a member of the AdePTTrackingmanager and it takes care of all CPU propagation and cleans up the propagation loop. 3. adds a direct conversion of the Geant4 NavigationHistory to a VecGeom NavigationState. This cleans up many problems at boundaries that can become costly for edges of GPU regions, where particles are frequently moved between CPU and GPU, or very tangent particles could get stuck. This reduces the run time for 1e6 10 GeV electrons being shot at testEm3 where all layers are GPU regions (but the calorimeter is not) from Before: Run time: 1582.26 Now: Run time: 1281.32 4. cleans up the AdePT physics. Now `FTFP_BERT_AdePT` derives from `G4EmStandardPhysics`. Currently, the gamma/lepton nuclear processes are not registered in the PhysicsList. However, if they are registered, G4HepEm treats them correctly, while AdePT just ignores them. For electrons, they are suppressed by an infinite interaction length, for gamma (since they are handled internally in G4HepEm), we simply do a empty step. There is a implementation commented out that re-draws in case a gamma-nuclear reaction is drawn. It re-draws up to 3 times and in the (unlikely, but possible) case that we still end up with gamma-nuclear, we do a redundant step limited by gamma-nuclear but not invoking any process. However, using this changes the physics by ~ 1 % at the tail, leading to worse agreement, therefore we for now accept the performance penalty of empty steps. **Testing:** For the testing, different variations with testEM3 where tested: 1. making everything a GPU region via `/adept/setTrackInAllRegions true`. 2. making all layers a GPU region (but not the encompassing `Calorimeter` region. 3. using 3 different regions that include various layers, without specific order 4. using no regions at all. These settings were tested against running with the `--noadept` option, which uses the `FTFP_BERT_HepEm` physics list. All 4 configurations were tested with this PR and with the `master` branch. **Outcome:** This PR shows excellent agreement between AdePT and G4 with G4HepEm: (the plot is for making everything a GPU region) <img width="577" alt="Screenshot 2025-01-01 at 16 58 52" src="https://github.com/user-attachments/assets/13688705-c99c-4714-b523-cb24128d097b" /> It is slightly improved to the `master` branch: <img width="575" alt="Screenshot 2025-01-01 at 17 00 27" src="https://github.com/user-attachments/assets/265b61f6-df42-479a-8a56-1430b06a548a" /> No major differences are observed using 2. all layers being regions, 3. using only certain regions, and 4. using no regions. 2. All layers GPU: <img width="576" alt="Screenshot 2025-01-01 at 18 53 07" src="https://github.com/user-attachments/assets/8359ad1e-276d-43d6-8271-87691c128df1" /> 3. Various GPU regions: <img width="568" alt="Screenshot 2025-01-01 at 18 53 48" src="https://github.com/user-attachments/assets/6029fb71-8804-43b7-abcb-31088016db54" /> 4. No GPU regions: <img width="568" alt="Screenshot 2025-01-01 at 18 54 21" src="https://github.com/user-attachments/assets/7854d0e4-b048-4026-bd5c-82ba41bb88d0" /> In the `master`, 3. gets stuck and never finishes. It could still be off in the plot above, to be investigated with higher statistics. Since it didn't work at all in the `master` this is still a significant improvement. The overall run time for G4 + HepEm in our example is improved using the tracking manager approach: From `Run time: 2842.99 s` (master) to `Run time: 2255.76 s` using this PR (16 threads, 1e6 10 GeV electrons being shot at testEm3) Additionally the energy deposition of CMS was checked and found to be at good agreement. The run time for CMS is similar, maybe slightly improved using AdePT everywhere. To Do: - [x] Proper validation after #326 is merged. - [x] Validate run with CMS. - [x] After [this PR](mnovak42/g4hepem#117) is merged in G4HepEm, tag that version of G4HepEm and update CI - [x] implement changes also in split kernels - [x] validate with split kernels (validate kernels give roughly the same result but not to machine precision anymore, however, before this PR it was not compiling anymore). NOTE: These are breaking changes, as older versions of G4HepEm cannot be used anymore.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR shows how to use G4HepEM through the TrackingManager instead of the G4 process. However, it can be used only to run on CPU, and will not work with AdePT since one TrackingManager will replace the other.
In order to be able to run G4HepEM with tracking manager on full CPU and CPU + AdePT modes we would need: