Installation on Frontera

Author: [email protected], [email protected]

Frontera is the new machine at TACC. Installation is very similar to Stampede2.

Before you begin:

to connect: ssh [email protected]
do not work/compile on the login node, but submit an interactive job: idev -m 120
using all available cores will require more memory than is available on development nodes, so we only use 28 cores to compile
use $SCRATCH or $WORK to compile and install, not $HOME (edit: earlier only $SCRATCH worked)
running binaries with MPI does not work (hang) unless you use ibrun, for exampleibrun -n X ./aspect, leaving out the -n X will use however many cores are available within this job
ld.gold seems to have issues linking deal.II, we use ld.bfd instead, by setting -D DEAL_II_LINKER_FLAGS='-fuse-ld=bfd'
Intel MKL lapack on Frontera is not automatically recognized by deal.II, that is why we set -DDEAL_II_WITH_LAPACK=ON -D LAPACK_LIBRARIES='...'

Installation steps:

0. modules

module reset
module load cmake intel/19.1.1 impi/19.0.9

echo "module reset;module load cmake intel/19.1.1 impi/19.0.9" >> $HOME/.bashrc

1. candi

Below we install a development version of deal.II. If you prefer a stable release (e.g. 9.5.2) set STABLE_BUILD=true and DEAL_II_VERSION=v9.5.2.

cd $WORK2
git clone https://github.com/dealii/candi.git
cd candi
echo "MKL=ON" > local.cfg
echo "MKL_DIR=$MKLROOT/lib/intel64" >> local.cfg
echo "NATIVE_OPTIMIZATIONS=ON" >> local.cfg
echo "DEAL_II_VERSION=master" >> local.cfg
echo "BUILD_EXAMPLES=OFF" >> local.cfg
echo "DEAL_II_CONFOPTS=\"-D DEAL_II_WITH_64BIT_INDICES=ON -D DEAL_II_COMPONENT_EXAMPLES=OFF -DDEAL_II_WITH_COMPLEX_VALUES=OFF -DDEAL_II_WITH_LAPACK=ON -D LAPACK_LIBRARIES='/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_lp64.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_intel_thread.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/mkl/lib/intel64_lin/libmkl_core.so;/opt/intel/compilers_and_libraries_2020.1.217/linux/compiler/lib/intel64_lin/libiomp5.so;-lm;-ldl' -D DEAL_II_LINKER_FLAGS='-fuse-ld=bfd'\"" >> local.cfg

./candi.sh -j 28 --packages="once:p4est once:hdf5 once:sundials once:trilinos dealii" -p $WORK2/libs
echo ". $WORK2/libs/configuration/enable.sh" >> $HOME/.bashrc

2. ASPECT

Frontera's Intel Compiler does not seem to like precompiling the bundled boost library inside deal.II. Therefore, disable precompiling headers.

cd $WORK2
. $HOME/.bashrc
git clone https://github.com/geodynamics/aspect.git
mkdir aspect-build
cd aspect-build
cmake -DASPECT_PRECOMPILE_HEADERS=OFF ../aspect
make release
make -j 50

Running:

ibrun $WORK2/aspect-build/aspect bla.prm

Jobs

check the queue, submit a job, cancel a job:

squeue -u <username>
sbatch <script>
scancel <id>

no longer relevant

Make sure to use a candi version that includes the fix for Intel MKL described here: https://github.com/dealii/candi/pull/150. E.g. take this version: https://github.com/tjhei/candi-old/tree/trilinos_mkl.
intel 19.0.5 compilers have difficulties with CXX17, https://github.com/dealii/dealii/issues/8755 so we work around it by setting -D DEAL_II_WITH_CXX17=OFF

Provide feedback

Saved searches