First of all make sure that all the necessary system tools and development libraries are available on the System. For Ubuntu we provide the script: checkpr.sh that ensures the system is ready. Opticks requires Geant4, nvidia cuda and nvidia Optix among other libraries. CaTS in addition will require ROOT. If all these libraries and development headers are available on your machine skip directly to (Building opticks vs. existing libraries). On a 'blank' computing system it makes sense to build CLHEP, then Geant4 and finally ROOT in that order assuring that all the necessary development libraries and headers are installed.
Check the release note for the required CLHEP version for the Geant4 release you are using. For example for Geant4 11.2 this can be found at: https://geant4.web.cern.ch/download/release-notes/notes-v11.2.0.html so in this case the CLHEP version is 2.4.7.1.
CLHEP can be found at: https://proj-clhep.web.cern.ch/proj-clhep/
to build it from scratch using cmake (used cmake version > 3.22.0)
cd to the directory where you want to build CLHEP (replace version with the appropriate one e.g. 2.4.7.1)
wget https://proj-clhep.web.cern.ch/proj-clhep/dist1/clhep-version}$$ .tgz
tar xzvf clhep-version.tgz
cd version/
mkdir CLHEP-build
cd CLHEP-build
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../CLHEP-install -DCLHEP_BUILD_CXXSTD=-std=c++17 ../CLHEP
ninja
ninja install
Note the default install directory is /usr/local but one needs root privileges to install it there:
cd to the directory where you want to build clhep
wget https://proj-clhep.web.cern.ch/proj-clhep/dist1/clhep-version.tgz
tar xzvf clhep-version.tgz
cd version/
mkdir CLHEP-build
cd CLHEP-build
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCLHEP_BUILD_CXXSTD=-std=c++17 ../CLHEP
ninja
sudo ninja install
Geant4 versions are available at: https://geant4.web.cern.ch/support/download (replace version with the appropriate one e.g. v11.2.1)
cd to the directory where you want to install Geant4
wget https://geant4-data.web.cern.ch/releases/geant4-version.tar.gz
tar xzvf geant4-version.tar.gz
mkdir geant4-version-build
cd geant4-version-build
cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=../geant4-version-install -DGEANT4_BUILD_BUILTIN_BACKTRACE=OFF -DGEANT4_BUILD_VERBOSE_CODE=OFF -DGEANT4_INSTALL_DATA=ON -DGEANT4_USE_SYSTEM_CLHEP=ON -DGEANT4_USE_GDML=ON -DGEANT4_USE_SYSTEM_EXPAT=ON -DGEANT4_USE_SYSTEM_ZLIB=ON -DGEANT4_USE_QT=ON -DGEANT4_BUILD_MULTITHREADED=ON -DGEANT4_USE_OPENGL_X11=ON ../geant4-version
ninja
ninja install
. ../geant4-version-install/bin/geant4.sh
check the output for any error.
Instructions how to build ROOT from ssource can be found at: https://root.cern/install/build_from_source/
# cd to the diretory where you want to install root
git clone --branch latest-stable https://github.com/root-project/root.git root_src
mkdir root-build
cd root-build
cmake -GNinja -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=../root-install -Droot7=ON -Dxrootd=OFF ../root_src/
cmake --build . --target install
or if you use make instead of ninja:
cmake -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=../root-install -Droot7=ON -Dxrootd=OFF ../root_src/
# speed up the make process
new=" -j$(($(grep -c ^processor /proc/cpuinfo) - 1))"
case ":${MAKEFLAGS:=$new}:" in
*:"$new":*) ;;
*) MAKEFLAGS="$MAKEFLAGS:$new" ;;
esac
cmake --build . --target install
check the output for any error, install any development packages that might be necessary.
. ../root-install/bin/thisroot.sh
root
cuda is available at the NVIDIA web site just follow the instruction depending on the system you are using.
https://developer.nvidia.com/cuda-downloads
Note this will also install the corresponding NVIDIA graphics driver you might have to reboot.
A good way to check that things are working properly is to build the cuda samples and execute them
# cd to the directory where you want to build the cuda samples. E.g. the commands deviceQueryDrv and deviceQuery provide useful information.
mkdir cuda-test
cd cuda-test
cp -r /usr/local/cuda-11.3/samples .
cd samples/
which nvcc
make
bin/x86_64/linux/release/deviceQuery
bin/x86_64/linux/release/deviceQueryDrv
Note later versions of cuda don't provide the samples anymore. Instead the samples are provided in a git repository
git clone https://github.com/nvidia/cuda-samples
cd cuda-samples
make --ignore-errors
After the cuda-samples are build copy them to /usr/local/cuda or create a symbolic link /usr/local/cuda/sampes that points to the installation of cuda-samples. Opticks uses some of the sample headers and therefore building opticks will not suceed without them.
Two tools for monitoring Nvidia GPUs On Linux can be found here: https://www.linuxuprising.com/2019/06/2-tools-for-monitoring-nvidia-gpus-on.html
https://developer.nvidia.com/designworks/optix/download
Optix comes with precompiled samples and one might want to try them:
# cd to the Optix installation directory
cd SDK-precompiled-samples
export LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH
# execute e.g.:
./optixMDLSphere
./optixSphere
./optixTutorial
# etc.
This are instructions how to build opticks making use of preinstalled libraries available on the system. These libraries include CLHEP, xerces-c, boost and Geant4. For geant 4 we use the current version at the time of writing which is Geant4.10.7.p2. We make use of the fact that the om-cmake function of om.bash is sensitive to the CMAKE_PREFIX_PATH envvar so that we can point to the directories where the libraries are installed and avoid having to rebuild them. In principle just cut and paste the following line to a file change the envars of the different directories to match your system and source the resulting script. (replace version with the appropriate one e.g. v0.2.6)
cd to the directory where you want to install Opticks (the WORK_DIR environmental variable will point to this directory).
git clone https://github.com/simoncblyth/opticks.git
cd opticks
git checkout version
git fetch --all --tags
change opticks/optickscore/OpticksSwitches.h
so that:
#define WITH_SKIPAHEAD 1
is set. To create a setup_opticks.sh you can use the cat statement below. But you have to edit the created file so that the environmental variables WORK_DIR, OptiX_INSTALL_DIR, OPTICKS_COMPUTE_CAPABILITy, CUDA_INSTALL_DIR, CUDA_SAMPLES, G4INSTALL correspond to the correct values of your system.
cat > setup_opticks.sh << +EOF
# ----------------------------------------------------------------------------------------------------------------------
# --- you need to modify the following environmental variables so that point to the specific directories on your system
# ---
export WORK_DIR=/data/software/gpu4112
export OptiX_INSTALL_DIR=/data/software/NVIDIA-OptiX-SDK-6.5.0-linux64
export OPTICKS_COMPUTE_CAPABILITY=86
export CUDA_INSTALL_DIR=/usr/local/cuda
export CUDA_SAMPLES=${CUDA_INSTALL_DIR}/samples
export G4INSTALL=/data/software/geant4-v11.0.2-install
export LOCAL_BASE=${WORK_DIR}/local
# ----------------------------------------------------------------------------------------------------------------------
export CMAKE_PREFIX_PATH=${G4INSTALL}:${LOCAL_BASE}/opticks/externals:${OptiX_INSTALL_DIR}:${WORK_DIR}/opticks/cmake/Modules/:${WORK_DIR}/local /opticks:${WORK_DIR}/local/opticks:${WORK_DIR}/local/opticks/externals/
export PYTHONPATH=$WORK_DIR
export OPTICKS_HOME=${WORK_DIR}/opticks
export OPTICKS_PREFIX=${WORK_DIR}/local/opticks
export OPTICKS_INSTALL_PREFIX=$LOCAL_BASE/opticks
export OPTICKS_OPTIX_PREFIX=/data/software/NVIDIA-OptiX-SDK-6.5.0-linux64/
export OPTICKS_CUDA_PREFIX=${CUDA_INSTALL_DIR}
export OPTICKS_EMBEDDED_COMMANDLINE_EXTRA="--rngmax 10 --rtx 1 --skipaheadstep 10000"
#
# setup Geant4 and root
#
. ${G4INSTALL}/bin/geant4.sh
. /data/software/root-install/bin/thisroot.sh
opticks-(){ . ${OPTICKS_HOME}/opticks.bash && opticks-env $* ; }
op(){ op.sh $* ; }
o(){ cd $(opticks-home) ; hg st ; }
_path_prepend() {
if [ -n "$2" ]; then
case ":$(eval "echo \$$1"):" in
*":$2:"*) :;;
*) eval "export $1=$2\${$1:+\":\$$1\"}" ;;
esac
else
case ":$PATH:" in
*":$1:"*) :;;
*) export PATH="$1${PATH:+":$PATH"}" ;;
esac
fi
}
_path_append() {
if [ -n "$2" ]; then
case ":$(eval "echo \$$1"):" in
*":$2:"*) :;;
*) eval "export $1=\${$1:+\"\$$1:\"}$2" ;;
esac
else
case ":$PATH:" in
*":$1:"*) :;;
*) export PATH="${PATH:+"$PATH:"}$1" ;;
esac
fi
}
_path_prepend "${LOCAL_BASE}/bin"
# make sure to add the compiler options
new=" -fPIC"
case ":${CXXFLAGS:=$new}:" in
*:"$new":*) ;;
*) CXXFLAGS="$CXXFLAGS:$new" ;;
esac
new=" -fPIC"
case ":${CFLAGS:=$new}:" in
*:"$new":*) ;;
*) CFLAGS="$CFLAGS:$new" ;;
esac
# speed up the make process
new=" -j$(($(grep -c ^processor /proc/cpuinfo) - 1))"
case ":${MAKEFLAGS:=$new}:" in
*:"$new":*) ;;
*) MAKEFLAGS="$MAKEFLAGS:$new" ;;
esac
# deal with the $LD_LIBRARYPATH
new=${OptiX_INSTALL_DIR}/lib64/
case ":${LD_LIBRARY_PATH:=$new}:" in
*:"$new":*) ;;
*) LD_LIBRARY_PATH="$new:$LD_LIBRARY_PATH" ;;
esac
new=${OPTICKS_HOME}/externals/lib
case ":${LD_LIBRARY_PATH:=$new}:" in
*:"$new":*) ;;
*) LD_LIBRARY_PATH="$new:$LD_LIBRARY_PATH" ;;
esac
new=${CUDA_INSTALL_DIR}/lib64/
case ":${LD_LIBRARY_PATH:=$new}:" in
*:"$new":*) ;;
*) LD_LIBRARY_PATH="$new:$LD_LIBRARY_PATH" ;;
esac
new=${LOCAL_BASE}/opticks/lib/
case ":${LD_LIBRARY_PATH:=$new}:" in
*:"$new":*) ;;
*) LD_LIBRARY_PATH="$new:$LD_LIBRARY_PATH" ;;
esac
opticks-
new=${CUDA_INSTALL_DIR}/bin
case ":${PATH:=$new}:" in
*:"$new":*) ;;
*) PATH="$new:$PATH" ;;
esac
new=${OPTICKS_HOME}/bin/
case ":${PATH:=$new}:" in
*:"$new":*) ;;
*) PATH="$new:$PATH" ;;
esac
new=${OPTICKS_HOME}/ana/
case ":${PATH:=$new}:" in
*:"$new":*) ;;
*) PATH="$new:$PATH" ;;
esac
new=${LOCAL_BASE}/opticks/lib/
case ":${PATH:=$new}:" in
*:"$new":*) ;;
*) PATH="$new:$PATH" ;;
esac
new=${CUDA_SAMPLES}/bin/x86_64/linux/release/
case ":${PATH:=$new}:" in
*:"$new":*) ;;
*) PATH="$new:$PATH" ;;
esac
oinfo-(){
echo 'LD_LIBRARY_PATH:';
echo '================';
echo ${LD_LIBRARY_PATH}| tr : \\n;
echo;
echo 'PATH:';
echo '=====';
echo ${PATH}| tr : \\n;
echo;
echo 'CMAKE_PREFIX_PATH:';
echo '==================';
echo ${CMAKE_PREFIX_PATH}| tr : \\n;
}
dinfo-(){
nvidia-smi;
${CUDA_SAMPLES}/bin/x86_64/linux/release/deviceQuery
}
+EOF
Change setup_opticks.sh so that the environmental variables are set to correspond to your installation as instructed in the script. Then get some information about your system: source setup_opticks.sh oinfo- dinfo- To build the opticks external packages do:
mkdir -p ${WORK_DIR}/local/opticks/externals/
cd ${WORK_DIR}/local/opticks/externals/
ln -s ${OptiX_INSTALL_DIR} OptiX
cd ${WORK_DIR}
opticks-externals-install >& install_ext.log &
scan the log file or any errors and correct them. Now build opticks:
cd ${WORK_DIR}
opticks-full >& install_full.log &
scan the log file or any errors and correct them. Before you run opticks-t
you want to to create the geocache using e.g. one of the gdml files provided by CaTS (https://github.com/hanswenzel/CaTS)
geocache-
geocache-create- --gdmlpath ${WORK_DIR}/local/opticks/opticksdata/export/juno1808/g4_00.gdml
export OPTICKS_KEY=`output from above command"
opticks-t
if the geocache-create- command doesn't work try:
${WORK_DIR}/local/opticks/lib/OKX4Test --okx4test --g4codegen --deletegeocache --gdmlpath ${WORK_DIR}/CaTS/gdml/simpleLArTPC.gdml
export OPTICKS_KEY=`output from above command"
opticks-t
Also CaTS ( see instructions how to build and run CaTS below) rebuilds the geocache each time you run it and it will print out what to set the OPTICKS_KEY to: e.g. export OPTICKS_KEY=CaTS.X4PhysicalVolume.World_PV.6a511c07e6d72b5e4d71b74bd548e8fd
Instructions for building and running CaTS can be found here: README
| Product | Version |
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
to check version use the following command:
ubuntu: lsb_release -a
cuda: nvcc --version
the setup script setup_opticks.sh above also defines two useful functions which will print out a lot of information about your system
oinfo-
dinfo-
Simon Blyth also maintains a page describing reference_versions https://bitbucket.org/simoncblyth/opticks/src/master/notes/reference_versions.rst