Multi-Threading in DDG4

7 Multi-Threading in DDG4

7.1 Introductory Remarks
7.2 Thread related contexts
7.3 Thread-Shared Components
7.4 Backwards- and Single-Thread-Compatibility
7.5 Support for Python Setup in Multi-Threading Mode
7.6 DDG4 Multi-Threading Example

7.1 Introductory Remarks

Multi-threading as supported by Geant4 is event driven. This means that the simulation of a given event is handled by one single thread. Geant4 provides speciﬁc extensions to ease the users the use of its multi-threaded extensions [13] ⁴ . These extension divide in a formalized manner all actions to be performed to setup a Geant4 multi-threaded program into

common actions to be performed and shared by all threads. This includes the setup of the geometry and the physics list. The other main area are
thread-speciﬁc actions to be performed for each thread. These are composed by the user actions called during the processing of each run. These are the run-, event-, generation-, tracking-, stepping and stacking actions.

To understand the interplay between DDG4 and Geant4 let us quickly recapitulate the Geant4 mechanism how to conﬁgure multiple threads. The setup of a multi-threaded application in Geant4 is centered around two additional classes, which both deal with single- and multi-threaded issues:

G4VUserActionInitialization class with 2 major callbacks: Build() which is executed for each worker thread and BuildForMaster() which is executed for master thread only.
G4VUserDetectorConstruction class with the callbacks Construct(), where the shared geometry is constructed and ConstructSDandField() where the sensitive detectors and the electro magnetic ﬁelds are provided.

Both these Geant4 provided hooks are modeled in the standard DDG4 way as action queues, which allow a modular and ﬁne grained setup as shown in Figure 12 and Figure 13.

Figure 12: The Geant4 user initialization sequence to setup DDG4 in multi-threaded mode. The callbacks buildMaster() is only called in multi-threaded mode.

The DDG4 framework ensures that all user callbacks are installed properly to the Geant4 run manager, which calls them appropriately at the correct time.

DDG4 provides three callbacks for each sequence. Each callback receives a temporary context argument, which may be used to shortcut access to basic useful quantities:     struct Geant4DetectorConstructionContext  {
      /// Reference to geometry object
      Geometry::LCDD&     lcdd;
      /// Reference to the world after construction
      G4VPhysicalVolume*  world;
      /// The cached geometry information
      Geant4GeometryInfo* geometry;
      /// G4 User detector initializer
      G4VUserDetectorConstruction* detector;
};

Figure 13: The Geant4 detector initialization sequence to setup DDG4. If supplied, Geant4 calls the components both, in the single-threaded and in the multi-threaded mode.

The callbacks and the expected functionality are:

First the detector geometry is constructed. This happens in the callback constructGeo(...). If a standard $D D 4 h e p$ geometry is present, this is translation of the geometry could be done by simply calling the plugin Geant4DetectorGeometryConstruction. Alternatively a user deﬁned plugin could perform this action.
Next the electromagnetic ﬁelds for the Geant4 particle tracking is constructed. A generic plugin Geant4FieldTrackingConstruction may be attached. The corresponding setup parameters are listed in Section 8. Alternatively a user deﬁned plugin could perform this action.
Finally the Geant4 sensitive detectors are instantiated and attached to the sensitive volumes. For generic setups the plugin Geant4DetectorSensitivesConstruction may be attached. Alternatively a user deﬁned plugin could perform this action.

7.2 Thread related contexts

DDG4 provides thread related context, which may be accessed or modiﬁed by user code. This context, the Geant4Context and its sub-components, as discussed in Section 5 are available as separate instances for each event and as such also independently for each worker thread. Hence, no user level locking of the event context is necessary in any worker thread.

7.3 Thread-Shared Components

Some actions, though executed in the context of a single thread context may only execute as singletons. An example would be a GeneratorAction, which read input events from ﬁle. Clearly the reading of data from ﬁle must be protected and the reading of one event in a given thread must ﬁnish, before the next thread may take over. Another example are data analysis components, which e.g. ﬁll a histogram. Typically the ﬁlling mechanism of a histogram is not thread safe and hence must be protected.

To solve such issues all actions, which may involve such shared activities, a shared action is provided, which adopts a singleton instance and executes the relevant callbacks in a protected manner. The shared actions execute the user component in a thread safe envelope.

Clearly no run- or event related state in such shared actions may be carried by the component object across callbacks. The action objects may not be aware of the event related context outside the callback. Default implementations for such shared actions exist for

the Geant4RunAction, where the calls to Geant4RunAction::begin and Geant4RunAction::end are globally locked and the sequential execution of the entire sequence is ensured;
the Geant4EventAction,
the Geant4TrackingAction,
the Geant4SteppingAction and
the Geant4StackingAction.

In the latter cases the framework ensures thread safety, but does not ensure the reentrant execution order of the entire sequence.

General Remark: Simple callbacks registered to the run-, event, etc.-actions cannot be protected. These callbacks may under no circumstances use any event related state information of the called object.

7.4 Backwards- and Single-Thread-Compatibility

As in the single threaded mode of Geant4, also in the multi-threaded mode all user actions are called by an instance of the G4RunManager or a sublass thereof, the G4MTRunManager [13].

If the recommended actions in sub-section 7.1 are used to conﬁgure the Geant4 application, then in a rather transparent way both single-threaded and multi-threaded setups can coexist simply by changing the concrete instance of the G4RunManager. There is one single exception: The user initialization function G4VUserActionInitialization::BuildForMaster() is only executed in multi-threaded mode. For this reason, we deprecate the usage. Try to ﬁnd solutions, without master speciﬁc setup using e.g. shared actions.

7.5 Support for Python Setup in Multi-Threading Mode

The setup of DDG4 in multi-threaded mode requires separate callbacks for the global conﬁguration (geometry, etc.) and the conﬁguration of the worker threads. In python this setup is performed within python callable objects, which are either functions or member functions of objects. These functions may have arguments. The python speciﬁc conﬁguration actions

The user initialization action Geant4PythonInitialization allows to conﬁgure python callbacks for the master and the worker thread setup using the calls:
      /// Set the Detector initialization command
      void setMasterSetup(PyObject* callable, PyObject* args);
      /// Set the field initialization command
      void setWorkerSetup(PyObject* callable, PyObject* args);
to be used in python as a call sequence within the master thread:
     init_seq = kernel.userInitialization(True)
     init_action = UserInitialization(kernel,’Geant4PythonInitialization/PyG4Init’)
     init_action.setWorkerSetup(worker_setup_call, < worker_args > )
     init_action.setMasterSetup(master_setup_call, < master_args > )
     init_seq.adopt(init_action)
The callback argument list $< w o r k e r_a r g s >$ and $< m a s t e r_a r g s >$ are python tuples containing all arguments expected by the callable objects $w o r k e r_s e t u p_c a l l$ and $m a s t e r_s e t u p_c a l l$ respecyively. The class Geant4PythonInitialization is a subclass of Geant4UserInitialization and will call the provided functions according to the protocol explained earlier in this section. If a callback is not set, the corresponding actiion is not executed.
The detector construction action Geant4PythonDetectorConstruction is the corresponding python action to populate the detector construction sequencer. and supports three ccallbacks:
      /// Set the Detector initialization command
      void setConstructGeo(PyObject* callable, PyObject* args);
      /// Set the field initialization command
      void setConstructField(PyObject* callable, PyObject* args);
      /// Set the sensitive detector initialization command
      void setConstructSensitives(PyObject* callable, PyObject* args);
to be used in python as call sequence within the master thread:
    init_seq = self.master().detectorConstruction(True)
    init_action = DetectorConstruction(self.master(),name_type)
    init_action.setConstructGeo(geometry_setup_call, < geometry_args > )
    init_action.setConstructField(field_setup_call, < field_args > )
    init_action.setConstructSensitives(sensitives_setup_call, < sensitives_args >)
    init_seq.adopt(init_action)
If any of the three callback is not set, the corresponding actiion is not executed. Hereby are $g e o m e t r y_s e t u p_c a l l$ , $f i e l d_s e t u p_c a l l$ and $s e n s i t i v e s_s e t u p_c a l l$ the callable objects to conﬁgure the geometry, the tracking ﬁeld and the sensitive detectors. $< g e o m e t r y_a r g s >$ , $< f i e l d_a r g s >$ and $< s e n s i t i v e s_a r g s >$ are the corresponding callable arguments in the form of a python tuple object.

All python callbacks are supposed to return the integer ’1’ on success. Any other return code is assumed to be failure.

7.6 DDG4 Multi-Threading Example

"""

   DD4hep simulation example setup DDG4
   in multi-threaded mode using the python configuration

   @author  M.Frank
   @version 1.0

"""
import os, time, DDG4

def setupWorker(geant4):
  kernel = geant4.kernel()
  print ’#PYTHON: +++ Creating Geant4 worker thread ....’
  print "#PYTHON:  Configure Run actions"
  run1 = DDG4.RunAction(kernel,’Geant4TestRunAction/RunInit’)
    ...
  print "#PYTHON:  Configure Event actions"
  prt = DDG4.EventAction(kernel,’Geant4ParticlePrint/ParticlePrint’)
  kernel.eventAction().adopt(prt)
    ...
  print "\n#PYTHON:  Configure I/O\n"
  evt_root = geant4.setupROOTOutput(’RootOutput’,’CLICSiD_’+time.strftime(’%Y-%m-%d_%H-%M’))
    ...
  gen = DDG4.GeneratorAction(kernel,"Geant4GeneratorActionInit/GenerationInit")
  kernel.generatorAction().adopt(gen)
  print "#PYTHON:  First particle generator: pi+"
  gen = DDG4.GeneratorAction(kernel,"Geant4IsotropeGenerator/IsotropPi+");
    ...
  print "#PYTHON:  Merge all existing interaction records"
  gen = DDG4.GeneratorAction(kernel,"Geant4InteractionMerger/InteractionMerger")
  kernel.generatorAction().adopt(gen)
  print "#PYTHON:  Finally generate Geant4 primaries"
  gen = DDG4.GeneratorAction(kernel,"Geant4PrimaryHandler/PrimaryHandler")
  kernel.generatorAction().adopt(gen)
  print "#PYTHON:  ....and handle the simulation particles."
  part = DDG4.GeneratorAction(kernel,"Geant4ParticleHandler/ParticleHandler")
  kernel.generatorAction().adopt(part)

  user = DDG4.Action(kernel,"Geant4TCUserParticleHandler/UserParticleHandler")
    ...
  part.adopt(user)
  print ’#PYTHON: +++ Geant4 worker thread configured successfully....’
  return 1

def setupMaster(geant4):
  kernel = geant4.master()
  print ’#PYTHON: +++ Setting up master thread for ’,kernel.NumberOfThreads,’ workers.’
  return 1

def setupSensitives(geant4):
  print "#PYTHON:  Setting up all sensitive detectors"
  geant4.printDetectors()
  print "#PYTHON:  First the tracking detectors"
  seq,act = geant4.setupTracker(’SiVertexBarrel’)
    ...
  print "#PYTHON:  Now setup the calorimeters"
  seq,act = geant4.setupCalorimeter(’EcalBarrel’)
    ...
  return 1

def run():
  kernel = DDG4.Kernel()
  lcdd = kernel.lcdd()
  install_dir = os.environ[’DD4hepINSTALL’]
  DDG4.Core.setPrintFormat("%-32s %6s %s")
  kernel.loadGeometry("file:"+install_dir+"/DDDetectors/compact/SiD.xml")
  DDG4.importConstants(lcdd)

  kernel.NumberOfThreads = 3
  geant4 = DDG4.Geant4(kernel,tracker=’Geant4TrackerCombineAction’)
  print "#  Configure UI"
  geant4.setupCshUI()

  print "#  Geant4 user initialization action"
  geant4.addUserInitialization(worker=setupWorker, worker_args=(geant4,),
                               master=setupMaster,master_args=(geant4,))
  print "#  Configure G4 geometry setup"
  seq,act = geant4.addDetectorConstruction("Geant4DetectorGeometryConstruction/ConstructGeo")

  print "# Configure G4 sensitive detectors: python setup callback"
  seq,act = geant4.addDetectorConstruction("Geant4PythonDetectorConstruction/SetupSD",
                                           sensitives=setupSensitives,sensitives_args=(geant4,))
  print "# Configure G4 sensitive detectors: atach’em to the sensitive volumes"
  seq,act = geant4.addDetectorConstruction("Geant4DetectorSensitivesConstruction/ConstructSD")

  print "#  Configure G4 magnetic field tracking"
  seq,field = geant4.addDetectorConstruction("Geant4FieldTrackingConstruction/MagFieldTrackingSetup")
  field.stepper            = "HelixGeant4Runge"
  field.equation           = "Mag_UsualEqRhs"
  field.eps_min            = 5e-05 * mm
  ...
  print "#  Setup random generator"
  rndm = DDG4.Action(kernel,’Geant4Random/Random’)
  rndm.Seed = 987654321
  rndm.initialize()
  print "#  Now build the physics list:"
  phys = geant4.setupPhysics(’QGSP_BERT’)
  geant4.run()

if __name__ == "__main__":
  run()

[prev] [prev-tail] [front] [up]