Installing DOLFINX on Habrok using spack

# Intro [Spack](https://github.com/spack/spack) is a package management tool designed for HPC, allowing to easily build optimized software in multiple versions and configurations. Many packages are supported (see the [package list](https://spack.readthedocs.io/en/latest/package_list.html)), among them the fenicsx/dolfinx package [py-fenics-dolfinx](https://spack.readthedocs.io/en/latest/package_list.html#py-fenics-dolfinx). This small guide explains how to install fenicsx on Habrok@RUG using spack. # Spack configuration On Habrok, clone Spack into your home directory: ```sh cd && git clone -c feature.manyFiles=true https://github.com/spack/spack.git ``` Source the Spack environment: ```sh source $HOME/spack/share/spack/setup-env.sh ``` (To load the Spack environment automatically in a terminal, consider appending the line `[ -f "$HOME/spack/share/spack/setup-env.sh" ] && source "$HOME/spack/share/spack/setup-env.sh"` to your `~/.bashrc`.) Load the GCC compiler and let Spack find it: ```sh module load GCC spack compiler find ``` Verify that gcc 12.2.0 was found: ``` $ spack compiler list ==> Available compilers -- gcc rocky8-x86_64 -------------------------------------------- gcc@12.2.0 gcc@8.5.0 ``` Spack needs to use the system OpenMPI in order for mpi to work, instead of compiling it from source. In addition, we use the system modules of OpenBLAS and OpenSSL (recommended in the [docs](https://spack.readthedocs.io/en/latest/getting_started.html#system-packages)). For this purpose, create a file `~/.spack/packages.yaml` with the following content: ```yaml packages: openmpi: externals: - spec: openmpi@4.1.4 modules: [OpenMPI/4.1.4-GCC-12.2.0] buildable: False openssl: externals: - spec: openssl@1.1.1k modules: [OpenSSL/1.1] buildable: False openblas: externals: - spec: openblas@0.3.21 modules: [OpenBLAS/0.3.21-GCC-12.2.0] buildable: False ``` Check this by executing ```sh spack config get packages ``` Let Spack detect all additional system modules: ```sh spack external find ``` # dolfinx installation Create a new Spack environment, for instance with the name `fenicsx` and activate it: ```sh spack env create fenicsx spack env activate fenicsx ``` Add the dolfinx package to the environment: ```sh spack add py-fenics-dolfinx arch=linux-rocky8-zen3 cflags="-O3" fflags="-O3" ``` This will select the latest stable release (at the time of writing `v0.6.0`, see the list of tags [here](https://spack.readthedocs.io/en/latest/package_list.html#py-fenics-dolfinx)). To install the development version, use `py-fenics-dolfinx@main`. The -O3 compiler flags optionally apply level 3 optimization (https://github.com/FEniCS/dolfinx#spack). The `arch` setting specifies the [microarchitecture](https://spack.readthedocs.io/en/latest/basic_usage.html#support-for-specific-microarchitectures) to determine the optimal compiler flags. Usually Spack will detect the correct settings automatically, if not compiling for a different machine. Because the login node and the compute nodes use different processors, in the command above we enforce the target archictecture for the compute nodes (`zen3`). (Confirm with `srun -n 1 spack arch`.) This setting may not be necessary if we compile the packages on a compute node instead of the login node (strongly recommended). Now the packages can be installed with `spack install`. This may take a while, so better create a job for a compute node: ```sh #!/bin/bash #SBATCH --time=06:00:00 #SBATCH --ntasks=1 #SBATCH --mem=8GB module load GCC source $HOME/spack/share/spack/setup-env.sh spack env activate fenicsx srun spack install ``` and run the job with `sbatch job.sh`. dolfinx should now be installed. Check the logs if openmpi and openblas were correctly picked up from the system instead of compiling them from source. Test your installation with, e.g., ```sh #!/bin/bash #SBATCH --time=00:10:00 #SBATCH --ntasks=2 #SBATCH --mem=4GB module load GCC source $HOME/spack/share/spack/setup-env.sh spack env activate fenicsx srun python3 $HOME/demo_poisson.py ``` with `demo_poisson.py`, e.g., from here: https://github.com/FEniCS/dolfinx/blob/main/python/demo/demo_poisson.py.