-
Notifications
You must be signed in to change notification settings - Fork 235
Compiling and Running ASPECT on TACC Stampede
Please note that Stampede is no longer in service and has been replaced with Stampede2. Please see the page Installation on TACC Stampede2 for instructions of to build and run ASPECT on Stampede2.
D. Sarah Stamps [email protected]
and Jonathan Perry-Houts [email protected]
updated from previous document by Eric Heien published on October 9, 2014
Setting up ASPECT on TACC Stampede is much easier than on previous systems since most of the necessary libraries are already correctly installed and configured. Following all the instructions in this document should allow you to start running ASPECT within 30 minutes. For further information on Stampede you can also refer to the user guide at http://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide
Also, please note that most parts of this document have changed since the previous version. Even if you have set up ASPECT before, please follow all sections of this document to make sure your current setup has all necessary updates.
All steps of this instructions are included in an installation script that can execute the commands automatically.
You first need to obtain an XSEDE account at http://portal.xsede.org/. If you need one send an email to: [email protected].
- To start edit your
$\sim$ /.bashrc file to load the correct modules. Update the file with the following in the section that says PLACE MODULE COMMANDS HERE and ONLY HERE:
module load git
module load gcc/4.7.1
module load mkl/13.0.2.146
module load cmake
To ensure the setup is correct, type source your
> source ~/.bashrc
> module list
The result should include all the modules above as well as some default modules:
Currently Loaded Modules:
1) TACC-paths 4) xalt/0.4.6 7) git/1.8.5.1 10) mkl/13.0.2.146
2) Linux 5) cluster 8) gcc/4.7.1 11) cmake/3.1.0
3) cluster-paths 6) TACC 9) mvapich2/1.9a2
- Get trilinos on Stampede for compiling locally.
mkdir $HOME/Downloads/
Download a copy of trilinos from http://www.trilinos.org. When creating this document we used trilinos-11.12.1-Source.tar because it is a known working version for the dealii-8.5.0 version we install. You can use other trilinos versions, but doing so requires updating the script aspect-setup.sh with the version you choose.
scp trilinos-11.12.1-Source.tar [email protected]:/work/02668/login/Downloads/
- Obtain and run currently working script for installing trilinos, p4est, and deal.II aspect-setup_stampede.sh
Obtain the script with:
wget https://raw.githubusercontent.com/geodynamics/aspect/master/doc/install/aspect-setup_stampede.sh
Since some of these libraries require a long time to compile, it may take a long time to finish if it is done on a login node (which is shared with dozens or hundreds of other users). It is recommended you log onto a compute node when compiling. To do so, use the following command: \
srun -p development -t 1:00:00 -n 16 --pty /bin/bash -l
You will need to modify the script if you choose to install a different version of trilinos. See comments within the script for additional information.\
aspect-setup_stampede.sh trilinos
aspect-setup_stampede.sh p4est
aspect-setup_stampede.sh deal.ii
You should now have the following directory structure in
$HOME/packages:
build deal.ii p4est trilinos
At this point you have automatically linked to these directories in your .bashrc file. Source your .bashrc file again before configuring and building ASPECT to ensure you can access to the deal.II directory.
source ~/.bashrc
echo $DEAL_II_DIR
Here we assume you have a cloned version of ASPECT on github.
cd $HOME/packages
git clone https://github.com/your_github_account_name/aspect.git
cd aspect
git remote add upstream https://github.com/geodynamics/aspect.git
git pull upstream master
mkdir build
cd build
cmake ..
You will get a lot of warnings, but as long as you see the following you are ready to compile. \ -- Generating done \ -- Build files have been written to:
/home1/02668/login/packages/aspect/build
make -j8
When testing ASPECT, it can be useful to run in interactive mode on the cluster. This way you can use multiple processors for fast testing, but still make changes to files and test different options without needing to wait in the batch queue after each change. To start an interactive shell with multiple processors, use:
srun -p development -t <maximum time> -n <#cores> --pty /bin/bash -l
For example:
srun -p development -t 0:30:00 -n 4 --pty /bin/bash -l
Once logged in you can run ASPECT in parallel using the command:
ibrun ./build/aspect <parameter file>
The development queue only allows small short runs ($<$2 hours, $<$256 cores). To perform larger runs, you submit a job to the SLURM scheduler. The format of the submission script is detailed in the Stampede user guide (http://www.tacc.utexas.edu/user-services/ user-guides/stampede-user-guide). An example is included below:
#!/bin/bash
#SBATCH -J aspect_run # job name
#SBATCH -o aspect_run.o\%j # output and error file name (\% expands to jobID)
#SBATCH -n 32 # total number of mpi tasks requested
#SBATCH -p normal # queue (partition) -- normal, development, etc.
#SBATCH -t 12:00:00 # run time (hh:mm:ss) - 12 hours
ibrun $WORK/aspect/build/aspect $WORK/cookbooks/shell_simple_3d.prm
The script is submitted to the run queue using:
sbatch <script file>
Once a simulation is finished, you may wish to transfer the resulting output files to a local machine for further analysis. There are two common methods of doing so, scp or Globus. scp (secure copy) works best for smaller transfers (1GB or less). The syntax is similar to normal copies: \
scp user1@host1:file1 user2@host2:file2
To recursively copy the contents of a directory, use the -r option. For example, to copy the results of a computation from Stampede to the local directory:
scp -r [email protected]:/work/01766/login/aspect/output .
For larger transfers, Globus is recommended since it optimizes transfer rates and tolerates failures during the transfer. For details about using Globus with Stampede please refer to:
https://www.tacc.utexas.edu/user-services/user-guides/stampede-user-guide#transferring.