External Libraries

From WikiROMS
Jump to navigationJump to search
External Libraries

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

ARPACK (ARnoldi PACKage)

Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method (IRAM) for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the BLAS and LAPACK libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

To obtain ARPACK, execute the following:

> svn checkout https://www.myroms.org/svn/src/libs/ARPACK
> cd ARPACK

To compile, customize ARmake.inc for the desired compiler and its flags for the serial (FC and FFLAGS) and parallel (PFC and PFFLAGS) versions of the library. Once configured type:

> make lib plib

to build libarpack.a and libparpack.a. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the my_build_paths script.


Note Note: Prior to SVN revision 1080 (July 23, 2021), ARPACK was distributed in the ROMS Lib/ARPACK subdirectory. For more information about why this was changed see TRAC ticket #891

ESMF (Earth System Modeling Framework)

High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model (ESM) components. It includes the NUOPC (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The NUOPC cap file is a Fortran module layer that sits on top of each ESM component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the NUOPC layer includes the native CoupleSets for ESM applications with nested grids. Various types of connections are possible, like nest-to-nest, fine-to-coarse, or coarse-to-fine. The user decides which nested grids to connect.

  • Configure with environment variables:
    > export ESMF_DIR=/projects/dmcs_1/src/esmf
    > export ESMF_COMM=mvapich2
    > export ESMF_COMPILER=intel
    > export ESMF_BOPT=O
    > export ESMF_OPTLEVEL=3
    > export ESMF_ABI=64
    > export ESMF_INSTALL_PREFIX=/ESMF/install/dir ESMF_NETCDF="/path/to/netcdf/bin/nc-config"
    > export ESMF_NETCDF_LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lhdf5_hl -lhdf5 -lm -lz -lcurl"
    > export ESMF_NETCDF_LIBPATH="/path/to/netcdf/lib /path/to/hdf5/lib"
    > export ESMF_SHARED_LIB_BUILD=OFF
    > export ESMF_MPIRUN=/path/to/cluster/submission/script
    Note Note that ESMF_MPIRUN is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -np, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
    # 1) <this_script> -np N prog
    # runs N copies of executable "prog" in parallel. The script must hide all
    # of the system specific details, such as going through a queueing system
    # and/or calling a system specific mpirun script with modified arguments.
    #
    # 2) The output of the application must arrive at the calling shell via
    # stdout and stderr.

    if [ "$1" != "-np" ]
    then
    echo "Usage: mpirun -np # prog"
    exit 1
    fi

    num_procs=$2
    shift 2
    prog=$*

    echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
    srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog
  • Build, test, and install:
    > make -j 10
    > make all_tests
    > make install

MCT (Model Coupling Toolkit)

Open-source library distributed as a set of Fortran-90 modules for constructing a coupled model system from Earth System Model (ESM) components (Larson et al., 2004). Each component model has its own grid and runs on its own set of processors. The MCT library provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred.

The MCT library is used to couple ROMS atmosphere models (like WRF) and wave models (like SWAN and WW3). For example, SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

MCT is available via github:

> git clone https://github.com/MCSclimate/MCT
> cd MCT
> ./configure
> make

NetCDF (Network Common Data Form)

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides API in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses NetCDF for all its input and output data management. Its NetCDF3 or NetCDF4 type files can be processed using the standard library developed and distributed UCAR/Unidata, the Parallel-IO (PIO) developed at ANL/NCAR (Dennis et al., 2012; Hartnett and Edwards, 2021), or the Software for Cashing Output and Reads for Parallel I/O (SCORPIO) library available in E3SM. We have migrated to the Fortran-90 API interface, which consists of a library and one or two module files. See the makefile and build script for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

Building NetCDF-4/HDF5

NetCDF-4 uses HDF5 for it's underlying data format so you first need to build HDF5. We recommend 1.10.6 because 1.10.7 causes errors in the NetCDF-C test suite. Here we will describe how to build parallel versions for use on an HPC cluster. We will note what changes are necessary if you are building for a system that doesn't have parallel I/O.

Building HDF5

  • Configure HDF5:
    > ./configure CC=mpicc CXX=mpicxx FC=mpifort \
    RUNPARALLEL="srun --nodes=1 --ntasks=6 --cpus-per-task=1 --time=00:13:00 --export=ALL " \
    --prefix=/path/to/parallel/hdf5/1.10.6 --enable-static-exec --enable-shared=no \
    --enable-parallel --disable-silent-rules --enable-fortran
    Note Note: RUNPARALLEL is only needed if your system uses a scheduler to run MPI jobs. The example here is for a SLURM system but you will need to adjust this for your cluster.

    Note Note: To compile serial, remove RUNPARALLEL=... and --enable-parallel and change the CC, CXX, and FC to the corresponding serial compilers (e.g. gcc, g++, and gfortran).

    Note Note: If you would like shared libraries to be built, also remove --enable-shared=no.

  • Build, test, and install HDF5:
    > make -j 5
    > make all_tests
    > make install

Building NetCDF-C

The C and Fortran APIs for NetCDF are in separate packages and the C version must be built first.

  • Configure NetCDF-C:
    > ./configure CC=mpicc FC=mpifort CFLAGS=-O3 CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include" \
    LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib" \
    --with-mpiexec=/path/to/job/submistion/script \
    --prefix=/path/to/parallel/netcdf/4.7.4 \
    --enable-parallel-tests --disable-shared --disable-silent-rules
    Note Note that --with-mpiexec is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -n, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
    # 1) <this_script> -n N prog
    # runs N copies of executable "prog" in parallel. The script must hide all
    # of the system specific details, such as going through a queueing system
    # and/or calling a system specific mpirun script with modified arguments.
    #
    # 2) The output of the application must arrive at the calling shell via
    # stdout and stderr.

    if [ "$1" != "-n" ]
    then
    echo "Usage: mpirun -n # prog"
    exit 1
    fi

    num_procs=$2
    shift 2
    prog=$*

    echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
    srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog
    Note Note: To compile serial, remove --with-mpiexec and --enable-parallel-tests and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).

    Note Note: If you would like shared libraries to be built, also remove --disable-shared.

  • Build, test, and install NetCDF-C:
    > make -j 5
    > make check
    > make install

Building NetCDF-Fortran

Now we can build the Fortran API.

  • There is a bug/typo in configure that makes nf-config report the wrong linking information to ROMS. To fix the error, edit the configure file (and configure.ac if you plan to run autoconf -i again) and change the line:
    NC_FLIBS="-lnetcdff $NC_FLIBS"
    to
    NC_FLIBS="-lnetcdff $NC_LIBS"
    more details can be found here.

  • Configure NetCDF-Fortran. Note that we set prefix to the same as NetCDF-C so the C and Fortran libraries live together:
    > ./configure CC=mpicc FC=mpifort CFLAGS=-O3 FFLAGS=-O3 \
    CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include -I/path/to/parallel/netcdf/4.7.4/include" \
    LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib -L/path/to/parallel/netcdf/4.7.4/lib" \
    LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl" \
    --with-mpiexec=/path/to/job/submistion/script \
    --prefix=/path/to/parallel/netcdf/4.7.4 \
    --enable-parallel-tests --disable-shared --disable-silent-rules
    Note Note that --with-mpiexec should point to the submision script you created while building NetCDF-C and is only needed for running test on clusters that use scheduling systems like SLURM.

    Note Note: To compile serial remove --with-mpiexec and --enable-parallel-tests and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).

    Note Note: If you would like shared libraries to be built, also remove --disable-shared.

  • If you are running on clusters that use scheduling systems like SLURM, you will need to edit 4 testing scripts replacing all occurrences of mpiexec with the full path to the submision script you created for NetCDF-C:
    > vi examples/F90/run_f90_par_examples.sh
    > vi nf03_test4/run_f90_par_test.sh
    > vi nf_test4/run_f77_par_test_03.sh
    > vi nf_test4/run_f77_par_test.sh
  • Build, test, and install NetCDF-C:
    > make -j 5
    > make check
    > make install

PIO (Parallel I/O)

Open-source parallel library for reading and writing distributed arrays to several scientific data formats like pNetCDF, NetCDF3, and NetCDF4/HDF5. PIO provides the flexibility to regulate the number of I/O tasks through data rearrangement between computational and I/O processes to improve performance and memory usage. It allows both synchronous and asynchronous I/O. In synchronous mode, a subset of processes performs both I/O and computations (MPI intra-communications). Alternatively, in asynchronous mode, the I/O is carried out by a set of disjointed and dedicated processes (MPI inter-communications).

The PIO or SCORPIO library is intended for ROMS Message Passing Interface (MPI) applications running on a large number of processes in an HPC computer with a Parallel File System (Lustre, GPFS, and so on). It uses the MPI-IO interface to facilitate the partitioning of data across computational or dedicated I/O processes. Both PIO and SCORPIO will work with ROMS svn revision 1064 (May 10, 2021) or newer but we recommend PIO for its better performance. We recommend you read the manuals for all the build options, but below we show an example build process for both PIO and SCORPIO.

Building PIO (autotools)

PIO can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for PIO, PnetCDF is optional. NoteHowever, the ROMS build system assumes that PIO is built with PnetCDF support and will need customization if PIO is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

PIO recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the PIO library but ROMS Parallel I/O is not currently implemented this way. In order to build PIO with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The PIO library can be built with the classic Autotools or Cmake. For PIO we show the Autotools procedure but will show Cmake for SCORPIO in the next section.

  • If you downloaded the gzipped tarball, you should run make clean because some Fortran .mod files are included and need to be removed:
    > make clean
  • Configure PIO:
    > ./configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \
    CPPFLAGS="-I/path/to/netcdf/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \
    LDFLAGS="-L/path/to/netcdf/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \
    LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz" \
    --with-mpiexec=/path/to/cluster/submission/script \
    --prefix=/pio/install/path --enable-fortran --disable-timing \
    --disable-silent-rules --disable-shared
    Note Note that --with-mpiexec is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -n, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
    # 1) <this_script> -n N prog
    # runs N copies of executable "prog" in parallel. The script must hide all
    # of the system specific details, such as going through a queueing system
    # and/or calling a system specific mpirun script with modified arguments.
    #
    # 2) The output of the application must arrive at the calling shell via
    # stdout and stderr.

    if [ "$1" != "-n" ]
    then
    echo "Usage: mpirun -n # prog"
    exit 1
    fi

    num_procs=$2
    shift 2
    prog=$*

    echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
    srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog
  • Build, test, and install PIO:
    > make -j 5
    > make check
    > make install
    Warning Warning: If you are using the Intel compilers, you will probably need to remove "test_darray_async_many" from line 50 of test/cunit/run_tests.sh for make check to complete without errors. Details here.

Building SCORPIO (Cmake)

Note Note: SORPIO recently released version 1.2.0 and 1.2.1 but ROMS has not been tested with these versions. It is likely that the new version will work fine with ROMS but use these new versions at your own risk.

SCORPIO can use NetCDF (version 4.3.3+), PnetCDF (version 1.6.0+) or ADIOS (version 2.6.0+) for I/O. Note ADIOS is not implemented in ROMS.

Ideally, the NetCDF version should be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. Additionally, HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP. If your NetCDF is built with DAP support, you will need to edit cmake/TryNetCDF_DAP.c, changing line 8 from

#if NC_HAS_DAP==1

to

#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1
  • Download the SCORPIO source code:
    > git clone https://github.com/E3SM-Project/scorpio
    > cd scorpio
    > git checkout tags/scorpio-v1.1.6
    If your NetCDF is built with DAP support, you will need to edit cmake/TryNetCDF_DAP.c, changing line 8 from
    #if NC_HAS_DAP==1
    to
    #if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1
  • Configure SCORPIO:
    > cd ..
    > mkdir build_scorpio
    > cd build_scorpio
    > CC=mpicc CXX=mpicxx FC=mpifort cmake \
    -DNetCDF_PATH=/path/to/netcdf \
    -DPnetCDF_PATH=/path/to/pnetcdf \
    -DMPIEXEC=/path/submission/scipt \
    -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \
    -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \
    -DCMAKE_INSTALL_PREFIX=/scorpio/install/path ../scorpio
    Note Note that -DMPIEXEC is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -n, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
    # 1) <this_script> -n N prog
    # runs N copies of executable "prog" in parallel. The script must hide all
    # of the system specific details, such as going through a queueing system
    # and/or calling a system specific mpirun script with modified arguments.
    #
    # 2) The output of the application must arrive at the calling shell via
    # stdout and stderr.

    if [ "$1" != "-n" ]
    then
    echo "Usage: mpirun -n # prog"
    exit 1
    fi

    num_procs=$2
    shift 2
    prog=$*

    echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
    srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog
  • Compile, test, and install:
    > make -j 6
    > make -j 6 tests
    > ctests
    > make install
Retrieved from "https://www.myroms.org/wiki/index.php?title=External_Libraries&oldid=6035"