Recent Changes - Search:

Starting With Fargo3D in Syntagma

FARGO3D knows about CUDA through the environment variable CUDA so we need to give it the path to where CUDA is installed.

export CUDA=/opt/cuda/toolkit/4.2.9/
mkdir $HOME/bin
export PATH=$HOME/bin:$PATH

There are three ways to compile and run Fargo: GPU (not parallel - compiled with cuda), MPI (parallel, runs on CPUs - compiled with openmpi) and GPU+MPI (parallel, runs on GPUs and CPUs - compiled with CUDA and openmpi). We provide instructions to compile and run all three of these versions. We place the binaries (fargo.cuda, fargo.mpi, fargo.cuda+mpi) in the $HOME/bin directory

GPU build

tar zxf fargo3d-0.9.tar.gz
cd fargo3d-0.9

module load cuda

make PARALLEL=0 GPU=1
mv fargo3d fargo3d.cuda
mv fargo3d.cuda $HOME/bin

Getting a fresh installation source

Before compiling a new version/configuration, make sure to remove the source and use a fresh installation source.

cd..
rm -rf fargo3d-0.9
tar zxf fargo3d-0.9.tar.gz

MPI build

tar zxf fargo3d-0.9.tar.gz
cd fargo3d-0.9

module load openmpi/1.6-gcc

make PARALLEL=1 GPU=0
mv fargo3d fargo3d.mpi
mv fargo3d.mpi $HOME/bin

GPU & MPI build

(DonĀ“t forget to start from a fresh installation source)

tar zxf fargo3d-0.9.tar.gz
cd fargo3d-0.9

module load cuda
module load openmpi/1.6-gcc

make PARALLEL=1 GPU=1
mv fargo3d $HOME/bin/fargo3d.cuda+mpi

First run (testing whether all builds are ok)

For this we use the example provided in the installation source of Fargo3d.

Run must be on a host with a GPU (i.e. compute-0-0 or compute-0-1 in syntagma)

cd fargo3d

./fargo3d.cuda -m setups/fargo/fargo.par

First Job

jobs must be sent into the queue from syntamga login node

mkdir -p examples/fargo3d.ex1
cp -r fargo3d-0.9/setups/fargo examples/fargo3d.ex1/
cp -r fargo3d-0.9/std examples/fargo3d.ex1/
cp -r fargo3d-0.9/planets examples/fargo3d.ex1/
cd examples/fargo3d.ex1

create a job script called fargo3d.ex1.sge

#!/bin/bash
#$ -cwd
#$ -j n
#$ -notify
#$ -m abes
#$ -M my@email.com
#$ -N fargo3d.ex1
#$ -S /bin/bash
#$ -q hp.q

echo "Fargo3d GPU (cuda) example execution"

# be sure module environmnet is properly loaded
. /etc/profile.d/modules.sh

# load the cuda environment
module load cuda

# include the fargo binary directories in the path
export PATH=$HOME/bin:$PATH

# run the example

fargo3d.cuda -m fargo/fargo.par

echo "Done"

qsub fargo3d.ex1.sge
watch qstat
[jcm@syntagma fargo3d]$ qstat
job-ID  prior   name       user         state submit/start at     queue                          slots ja-task-ID 
-----------------------------------------------------------------------------------------------------------------
  19823 0.50500 fargo3d.ex nspillane    qw    10/13/2014 15:02:19                                    1        
[jcm@syntagma fargo3d]$