External Libraries

External Libraries

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

ARPACK (ARnoldi PACKage)

Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method (IRAM) for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the BLAS and LAPACK libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

It is distributed in the ROMS Lib/ARPACK subdirectory. To compile, customize Lib/ARPACK/ARmake.inc for the desired compiler and its flags for the serial (FC and FFLAGS) and parallel (PFC and PFFLAGS) versions of the library. Once configured type:

make lib plib

to build libarpack.a and libparpack.a. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the my_build_paths script described in roms.in.

ESMF (Earth System Modeling Framework)

High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model (ESM) components. It includes the NUOPC (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The NUOPC cap file is a Fortran module layer that sits on top of each ESM component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the NUOPC layer includes the native CoupleSets for ESM applications with nested grids. Various types of connections are possible, like nest-to-nest, fine-to-coarse, or coarse-to-fine. The user decides which nested grids to connect.

• Downloaded from github:
  > git clone https://github.com/esmf-org/esmf.git esmf
  > cd esmf
  > git checkout tags/ESMF_8_1_0

• Configure with environment variables:
  > export ESMF_DIR=/projects/dmcs_1/src/esmf
  > export ESMF_COMM=mvapich2
  > export ESMF_COMPILER=intel
  > export ESMF_BOPT=O
  > export ESMF_OPTLEVEL=3
  > export ESMF_ABI=64
  > export ESMF_INSTALL_PREFIX=/ESMF/install/dir ESMF_NETCDF="/path/to/netcdf/bin/nc-config"
  > export ESMF_NETCDF_LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lhdf5_hl -lhdf5 -lm -lz -lcurl"
  > export ESMF_NETCDF_LIBPATH="/path/to/netcdf/lib /path/to/hdf5/lib"
  > export ESMF_SHARED_LIB_BUILD=OFF
  > export ESMF_MPIRUN=/path/to/cluster/submission/script
  Note that ESMF_MPIRUN is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -np, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
  # 1) <this_script> -np N prog
  # runs N copies of executable "prog" in parallel. The script must hide all
  # of the system specific details, such as going through a queueing system
  # and/or calling a system specific mpirun script with modified arguments.
  #
  # 2) The output of the application must arrive at the calling shell via
  # stdout and stderr.
  
  if [ "$1" != "-np" ]
  then
  echo "Usage: mpirun -np # prog"
  exit 1
  fi
  
  num_procs=$2
  shift 2
  prog=$*
  
  echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
  srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog

• Build, test, and install:
  > make -j 10
  > make all_tests
  > make install

NetCDF (Network Common Data Form)

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides API in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses NetCDF for all its input and output data management. Its NetCDF3 or NetCDF4 type files can be processed using the standard library developed and distributed UCAR/Unidata, the Parallel-IO (PIO) developed at ANL/NCAR (Dennis et al., 2012; Hartnett and Edwards, 2021), or the Software for Cashing Output and Reads for Parallel I/O (SCORPIO) library available in E3SM. We have migrated to the Fortran-90 API interface, which consists of a library and one or two module files. See the makefile and build script for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

MCT (Model Coupling Toolkit)

We used the Model Coupling Toolkit (MCT; http://www-unix.mcs.anl.gov/mct/; Larson, et al., 2004; Jacob et al., 2005) to couple ROMS with SWAN (Warner et al., in prep), and ROMS to WRF. MCT is an open-source software library, distributed as a set of Fortran90 modules for constructing a coupled model system from individual component models. Each component model has its own grid and runs on its own set of processors. The MCT provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred. SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides to SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

PIO (Parallel I/O)

The PIO or SCORPIO library can be used to facilitate Parallel I/O in ROMS. Both SCORPIO and PIO will work with ROMS svn revision 1064 (May 10, 2021) or newer but we recommend PIO for its better performance. We recommend you read the manuals for all the build options but below we show an example build process for both PIO and SCORPIO.

Building PIO (autotools)

PIO can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for PIO, PnetCDF is optional. However, the ROMS build system assumes that PIO is built with PnetCDF support and will need customization if PIO is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

PIO recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the PIO library but ROMS Parallel I/O is not currently implemented this way. In order to build PIO with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The PIO library can be built with the classic Autotools or Cmake. For PIO we show the Autotools procedure but will show Cmake for SCORPIO in the next section.

• First download the PIO source code as a gzipped tarball or via git clone:
  > wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz
  or
  > git clone https://github.com/NCAR/ParallelIO
  > cd ParallelIO
  > git checkout tags/pio2_5_4
  > autoconf -i

• If you downloaded the gzipped tarball, you should run make clean because some Fortran .mod files are included and need to be removed:
  > make clean

• Configure PIO:
  > configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \
  CPPFLAGS="-I/path/to/netcdf/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \
  LDFLAGS="-L/path/to/netcdf/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \
  LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz" \
  --with-mpiexec=/path/to/cluster/submission/script \
  --prefix=/pio/install/path --enable-fortran --disable-timing \
  --disable-silent-rules --disable-shared
  Note that --with-mpiexec is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -n, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
  # 1) <this_script> -n N prog
  # runs N copies of executable "prog" in parallel. The script must hide all
  # of the system specific details, such as going through a queueing system
  # and/or calling a system specific mpirun script with modified arguments.
  #
  # 2) The output of the application must arrive at the calling shell via
  # stdout and stderr.
  
  if [ "$1" != "-n" ]
  then
  echo "Usage: mpirun -n # prog"
  exit 1
  fi
  
  num_procs=$2
  shift 2
  prog=$*
  
  echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
  srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog

• Build, test, and install PIO:
  > make -j 5
  > make check
  > make install
  Warning: If you are using the Intel compilers, you will probably need to remove "test_darray_async_many" from line 50 of test/cunit/run_tests.sh for make check to complete without errors. Details here.

Building SCORPIO (Cmake)

Note: SORPIO recently released version 1.2.0 and 1.2.1 but ROMS has not been tested with these versions. It is likely that the new version will work fine with ROMS but use these new versions at your own risk.

SCORPIO can use NetCDF (version 4.3.3+), PnetCDF (version 1.6.0+) or ADIOS (version 2.6.0+) for I/O. ADIOS is not implemented in ROMS.

Ideally, the NetCDF version should be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. Additionally, HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP. If your NetCDF is built with DAP support, you will need to edit cmake/TryNetCDF_DAP.c, changing line 8 from

#if NC_HAS_DAP==1

to

#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1

• Download the SCORPIO source code:
  > git clone https://github.com/E3SM-Project/scorpio
  > cd scorpio
  > git checkout tags/scorpio-v1.1.6
  
  If your NetCDF is built with DAP support, you will need to edit cmake/TryNetCDF_DAP.c, changing line 8 from
  #if NC_HAS_DAP==1
  to
  #if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1

• Configure SCORPIO:
  > cd ..
  > mkdir build_scorpio
  > cd build_scorpio
  > CC=mpicc CXX=mpicxx FC=mpifort cmake \
  -DNetCDF_PATH=/path/to/netcdf \
  -DPnetCDF_PATH=/path/to/pnetcdf \
  -DMPIEXEC=/path/submission/scipt \
  -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \
  -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \
  -DCMAKE_INSTALL_PREFIX=/scorpio/install/path ../scorpio
  Note that -DMPIEXEC is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take -n, N (number of processes requested), and programName (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:
  # 1) <this_script> -n N prog
  # runs N copies of executable "prog" in parallel. The script must hide all
  # of the system specific details, such as going through a queueing system
  # and/or calling a system specific mpirun script with modified arguments.
  #
  # 2) The output of the application must arrive at the calling shell via
  # stdout and stderr.
  
  if [ "$1" != "-n" ]
  then
  echo "Usage: mpirun -n # prog"
  exit 1
  fi
  
  num_procs=$2
  shift 2
  prog=$*
  
  echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"
  srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog

• Compile, test, and install:
  > make -j 6
  > make -j 6 tests
  > ctests
  > make install