Jupiter Booster @ JSC

This page only provides HiPACE++ specific instructions. For more information please visit the JSC documentation.

Log in with <yourid>@login.jupiter.fz-juelich.de. Note that you first need to log into JuDoor to be added to a compute project, set up an SSH key and set up 2FA.

Running on GPU

Create a file profile.hipace and source it whenever you log in and want to work with HiPACE++:

# please set your project account
export proj=<your project id>
# required dependencies
module load CMake
module load GCC
module load OpenMPI
module load CUDA
module load HDF5
module load ccache # optional, accelerates recompilation
# optimize CUDA compilation for GH200
export AMREX_CUDA_ARCH=9.0

Install HiPACE++ (the first time, and whenever you want the latest version):

source profile.hipace
git clone https://github.com/Hi-PACE/hipace.git $HOME/src/hipace # only the first time
cd $HOME/src/hipace
rm -rf build
cmake -S . -B build -DHiPACE_COMPUTE=CUDA
cmake --build build -j 16

You can get familiar with the HiPACE++ input file format in our Get started section, to prepare an input file that suits your needs. You can then create your directory in your $SCRATCH_<project id>, where you can put your input file and adapt the following submission script:

#!/bin/bash -l
#SBATCH -A <your project id>
#SBATCH --partition=booster
#SBATCH --nodes=2
#SBATCH --ntasks=8
#SBATCH --ntasks-per-node=4
#SBATCH --gres=gpu:4
#SBATCH --time=00:05:00
#SBATCH --job-name=hipace
#SBATCH --output=hipace-%j-%N.txt
#SBATCH --error=hipace-%j-%N.err
export OMP_NUM_THREADS=1
module load GCC
module load OpenMPI
module load CUDA
module load HDF5
# fix issues with MPI
export UCX_CUDA_COPY_REG_WHOLE_ALLOC=on
export UCX_CUDA_COPY_MAX_REG_RATIO=0
export UCX_MEMTYPE_REG_WHOLE_ALLOC_TYPES=cuda
export UCX_MEMTYPE_CACHE=n
srun -n 8 --cpu_bind=sockets $HOME/src/hipace/build/bin/hipace inputs

and use it to submit a simulation.

Tip

Parallel simulations can be largely accelerated by using GPU-aware MPI. To utilize GPU-aware MPI, the input parameter comms_buffer.on_gpu = 1 must be set. Additionally comms_buffer.pre_register_memory = 1 can help improve startup time.

Note that using GPU-aware MPI may require more GPU memory.