To clone Pencil Code together with Astaroth: -------------------------------------------- git clone -b master --recurse-submodules https://@github.com/pencil-code/pencil-code.git or git clone -b master --recurse-submodules https://@pencil-code.org/git/ pencil-code cd pencil-code source sourceme.sh cd src/astaroth/submodule git checkout develop To add Astaroth later to already installed Pencil Code: ------------------------------------------------------- cd $PENCIL_HOME git checkout master git submodule update --init --remote cd src/astaroth/submodule git checkout origin develop Don't forget to set $PENCIL_HOME correctly - missing this is a common source of mess. To build with Astaroth: ----------------------- On Clusters, load appropriate CUDA (for NVIDIA) or HIP (for AMD) and MPI modules, perhaps also CMake. Set GPU = gpu_astaroth GPU_VENDOR = [nvidia|amd] # nvidia is default MULTITHREADING = openmp in Makefile.local. At first time, this implies to repeat pc_setupsrc. Build Pencil Code as usual by make or pc_build - when building with make, you need to set the MODULE_[PRE|IN|SUF]FIX environment variables as given in the config file for the chosen compiler on the given machine - when building with pc_build, use the flag -s|--serial for safety For gfortran: the compilation flag std95 must be set to f2003, you must have LDFLAGS_MAIN = -Bdynamic -Wl,--export-dynamic in Makefile.src For CRAY: you must have LDFLAGS_MAIN = -h dynamic -Wl,--export-dynamic in Makefile.src To run: ------- In SLURM, set the following in the sbatch script: --ntasks = --ntasks-per-node = <...> # ideally the number of all GPUs per node --cpus-per-task = <...> # ideally the number of CPUs per node, divided by ntasks-per-node (some systems won't allow this maximum, so try less) on Puhti: --gres = gpu:: on LUMI/Dardel: --gpus-per-node = Also, there are usually separate partitions for GPU use. On some imperfectly configured machines, additional settings are required. [which OMP/MPI enironment settings?] ln -s /cfs/klemming/home/b/brandenb/scr/GPU-test/pencil-code/src/astaroth/submodule/acc-runtime/samples/gputest submodule/acc-runtime/samples/gputest module load rocm/5.0.2 craype-accel-amd-gfx90a cmake/3.20.1 # Touko's demonstration session on Dardel: git clone git@github.com:pencil-code/pencil-code.git pencil-code cd pencil-code source sourceme.sh # => alternative to "pc_build": # use "csh" variant to prepare for a later "make": ### source sourceme.csh git checkout gputestv5 git submodule update --init --remote cd src/astaroth/submodule git checkout PCinterface_2019-8-12 cd ../../../samples/gputest pc_setupsrc pc_build -f hosts/lumi/host-uan01-GNU_Linux # => alternative to "pc_build": # configure your "Makefile.src" and then execute: ### make cd src/astaroth module load rocm craype-accel-amd-gfx90a cmake/3.20.1 make cd ../.. pc_build -f hosts/lumi/host-uan01-GNU_Linux # => alternative to "pc_build": ### make # => adapt settings in "disbatch.sh" sbatch disbatch.sh Axel: setenv PENCIL_HOME /cfs/klemming/home/b/brandenb/data/pencil-code-git module load aocm module spider rocm module load rocm/5.7.0 module load rocm craype-accel-amd-gfx90a cmake/3.20.1 pc_build -f Cray -s FFLAGS+=-O0 AXEL (2024-Dec-28) git clone -b gputestv6 --recurse-submodules https://AxelBrandenburg@pencil-code.org/git/ pencil-code cd pencil-code/src/astaroth/submodule git checkout PCinterface_2019-8-12 #AB: Matthias to email Toc Puro about detatched branch #AB: actual build, problems without O0 pc_setupsrc pc_build -f compilers/Cray_MPI FFLAGS+=" -g" LDFLAGS+='-Wl,--no-relax' pc_build -f compilers/Cray_MPI FFLAGS+=" -g -O0" LDFLAGS+='-Wl,--no-relax -L /opt/cray/pe/lib64 -lmpi_gtl_hsa' brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> find . -name '*auto*' ./src/astaroth/submodule/build/acc-runtime/api/autotune.csv #AB: Matthias to check RELOAD brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> cp ../../../../pencil-code/samples/gputest/disbatch.sh . brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/pencil-code/samples/gputest> sinfo -s sbatch job.sh /cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a/src/astaroth/submodule/acc-runtime/built-in/default_values.h fehlt git branch brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/pencil-code> cd src/astaroth/submodule/ brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/pencil-code/src/astaroth/submodule> rm -r * brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/pencil-code> git submodule update --init --recursive go to git checkout PCinterface_2019-8-12 1017 11:13 h brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/pencil-code/src/astaroth/submodule> git checkout PCinterface_2019-8-12 can build brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> ls -l src/astaroth/submodule/ /pc_bu brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> ml rocm brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> ml cmake ml pc_build -f compilers/Cray_MPI FFLAGS+=" -g" LDFLAGS+='-Wl,--no-relax' brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> vi src/astaroth/DSL/local/equations.h brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> cp ../k600_nu5em7_k4_Pm5/src/astaroth/DSL/local/equations.h src/astaroth/DSL/local/ brandenb@login1:/cfs/klemming/home/b/brandenb/data/GPU/axel/decay/reconnection/k600_nu5em7_k4_Pm5_128a> ls -l src/astaroth/submodule/acc-runtime/acc/