Build/install HiPACE++

Developers

If you are new to CMake, this short tutorial from the HEP Software foundation is the perfect place to get started with it. If you just want to use CMake to build the project, jump into sections 1. Introduction, 2. Building with CMake and 9. Finding Packages.

Dependencies

HiPACE++ depends on the following popular third party software. Please see installation instructions below in the Developers section.

  • a mature C++17 compiler: e.g. GCC 7, Clang 7, NVCC 11.0, MSVC 19.15 or newer

  • CMake 3.18.0+

  • AMReX development: we automatically download and compile a copy of AMReX

  • openPMD-api 0.15.1+: we automatically download and compile a copy of openPMD-api

    • HDF5 1.8.13+ (optional; for .h5 file support)

    • ADIOS2 2.7.0+ (optional; for .bp file support)

Platform-dependent, at least one of the following:

Optional dependencies include:

  • MPI 3.0+: for multi-node and/or multi-GPU execution

  • OpenMP 3.1+: for threaded CPU execution

  • CCache: to speed up rebuilds (needs 3.7.9+ for CUDA)

Please choose one of the installation methods below to get started:

HPC platforms

If you want to use HiPACE++ on a specific high-performance computing (HPC) systems, jump directly to our HPC system-specific documentation.

../_images/spack.svg

Using the Spack package manager

The dependencies can be installed via the package manager Spack (macOS/Linux):

spack env create hipace-dev
spack env activate hipace-dev
spack add adios2  # for .bp file support
spack add ccache
spack add cmake
spack add fftw
spack add hdf5    # for .h5 file support
spack add mpi
spack add pkgconfig  # for fftw
# optional:
# spack add cuda
spack install

(in new terminals, re-activate the environment with spack env activate hipace-dev again)

Note

On Linux distributions, the InstallError "OpenMPI requires both C and Fortran compilers" can occur because the Fortran compilers are sometimes not set automatically in Spack. To fix this, the Fortran compilers must be set manually using spack config edit compilers (more information can be found here). For GCC, the flags f77 : null and fc : null must be set to f77 : gfortran and fc : gfortran.

On macOS, a Fortran compiler like gfortran might be missing and must be installed by hand to fix this issue.

../_images/brew.svg

Using the Brew package manager

The dependencies can be installed via the package manager Homebrew (macOS/Linux):

brew update
brew install adios2  # for .bp file support
brew install ccache
brew install cmake
brew install fftw
brew install hdf5-mpi  # for .h5 file support
brew install libomp
brew install pkg-config  # for fftw
brew install open-mpi

Now, cmake --version should be at version 3.15.0 or newer.

Configure your compiler

For example, using a GCC on macOS:

export CC=$(which gcc)
export CXX=$(which g++)

If you also want to select a CUDA compiler:

export CUDACXX=$(which nvcc)
export CUDAHOSTCXX=$(which g++)

Build & Test

If you have not downloaded HiPACE++ yet, please clone it from GitHub via

git clone https://github.com/Hi-PACE/hipace.git $HOME/src/hipace # or choose your preferred path

From the base of the HiPACE++ source directory, execute:

# find dependencies & configure
cmake -S . -B build

# build using up to four threads
cmake --build build -j 4

# run tests
(cd build; ctest --output-on-failure)

Note: the from_file tests require the openPMD-api with python bindings. See documentation of the openPMD-api for more information. An executable HiPACE++ binary with the current compile-time options encoded in its file name will be created in bin/. Additionally, a symbolic link named hipace can be found in that directory, which points to the last built HiPACE++ executable. You can inspect and modify build options after running cmake .. with either

ccmake build

or by providing arguments to the CMake call

cmake -S . -B build -D<OPTION_A>=<VALUE_A> -D<OPTION_B>=<VALUE_B>

CMake Option

Default & Values

Description

CMAKE_BUILD_TYPE

RelWithDebInfo/Release/Debug

Type of build, symbols & optimizations

HiPACE_COMPUTE

NOACC/CUDA/SYCL/HIP/OMP

On-node, accelerated computing backend

HiPACE_MPI

ON/OFF

Multi-node support (message-passing)

HiPACE_PRECISION

SINGLE/DOUBLE

Floating point precision (single/double)

HiPACE_OPENPMD

ON/OFF

openPMD I/O (HDF5, ADIOS2)

HiPACE_PUSHER

LEAPFROG/AB5

Use leapfrog or fifth-order Adams-Bashforth plasma pusher

HiPACE++ can be configured in further detail with options from AMReX, which are documented in the AMReX manual.

Developers might be interested in additional options that control dependencies of HiPACE++. By default, the most important dependencies of HiPACE++ are automatically downloaded for convenience:

CMake Option

Default & Values

Description

HiPACE_amrex_src

None

Path to AMReX source directory (preferred if set)

HiPACE_amrex_repo

https://github.com/AMReX-Codes/amrex.git

Repository URI to pull and build AMReX from

HiPACE_amrex_branch

development

Repository branch for HiPACE_amrex_repo

HiPACE_amrex_internal

ON/OFF

Needs a pre-installed AMReX library if set to OFF

HiPACE_openpmd_mpi

ON/OFF (default is set to value of HiPACE_MPI)

Build openPMD with MPI support, although I/O is always serial

HiPACE_openpmd_src

None

Path to openPMD-api source directory (preferred if set)

HiPACE_openpmd_repo

https://github.com/openPMD/openPMD-api.git

Repository URI to pull and build openPMD-api from

HiPACE_openpmd_branch

0.15.2

Repository branch for HiPACE_openpmd_repo

HiPACE_openpmd_internal

ON/OFF

Needs a pre-installed openPMD-api library if set to OFF

AMReX_LINEAR_SOLVERS

ON/OFF

Compile AMReX multigrid solver.

For example, one can also build against a local AMReX copy. Assuming AMReX’ source is located in $HOME/src/amrex, add the cmake argument -DHiPACE_amrex_src=$HOME/src/amrex. Relative paths are also supported, e.g. -DHiPACE_amrex_src=../amrex.

Or build against an AMReX feature branch of a colleague. Assuming your colleague pushed AMReX to https://github.com/WeiqunZhang/amrex/ in a branch new-feature then pass to cmake the arguments: -DHiPACE_amrex_repo=https://github.com/WeiqunZhang/amrex.git -DHiPACE_amrex_branch=new-feature.

You can speed up the install further if you pre-install these dependencies, e.g. with a package manager. Set -DHiPACE_<dependency-name>_internal=OFF and add installation prefix of the dependency to the environment variable CMAKE_PREFIX_PATH. Please see the short CMake tutorial that we linked in the Developers section if this sounds new to you.

Documentation

The documentation is written at the RST format, to compile the documentation locally use

cd docs
# optional:                                 --user
python3 -m pip install -r requirements.txt          # only the first time
make html
open build/html/index.html

The last line would work on MacOS. On another platform, open the html file with your favorite browser.