build roms

Build Script - build.sh, build.bash

As mentioned in makefile, you need to provide settings for some user-defined choices before you can compile ROMS. If you have more than one application (or more than one compiler), you may get tired of editing the makefile. One option is to have a makefile for each configuration. The recommended solution, however, is to used the ROMS build script.

Introduction

There are two of these build scripts in the ROMS/Bin directory: build.sh (which is surprisingly a csh script) and build.bash. The build scripts use environment variables to provide values for the user-defined make variables, overwriting those found in the ROMS makefile. Just as in the multiple makefile option, you will need as many copies of the build script as you have applications. However, the scope of these variables is local to the build script, allowing you to compile different applications at the same time from the same sources as long as each $(SCRATCH_DIR) is unique.

Usage:

./build.sh [options]

Options:

-j [N] Compile in parallel using N CPUs. Omit argument to compile
on all available CPUs.
-noclean Do not clean already compiled objects.

Example:

./build.sh -j 2

User Definable Compilation Options

ROMS has a growing list of choices the user must make about the compilation before starting the compile process. These user-defined variables can be set in the build script. Since we use gnu make, it is possible to set the value of these variables in the Unix environment, rather than inside the Makefile (see gmake). The user-definable variables understood by the ROMS makefile are:

ROMS_APPLICATION

CPP option defining a particular application (for example, UPWELLING). The makefile will include its associated header file which is located in either the ROMS/Include directory or the path specified in the MY_HEADER_DIR definition. The header file name is the lowercase value of ROMS_APPLICATION with the .h extension (for example, upwelling.h). Recall the CPP option defining an application must be unique.

NestedGrids

Number of ROMS nested/composed/mosaic grids. Currently, only one grid is supported.

MY_ROOT_DIR

This path is internal to the build script and has no equivalent in the makefile. It can be used to set the base path to your ROMS source code and/or your application input and output files. This variable is not strictly necessary but can make setting the necessary paths easier.

MY_PROJECT_DIR

This path is internal to the build script and has no equivalent in the makefile. MY_PROJECT_DIR is most often used to set the paths: MY_HEADER_DIR, MY_ANALYTICAL_DIR, and BINDIR and to set the base path to SCRATCH_DIR. This can be seen near the bottom of the build script.

MY_ROMS_SRC

This path is internal to the build script and has no equivalent in the makefile. It is the location of your ROMS source code and is necessary for the build script to work properly.

COMPILERS

Directory where make can find your system/compiler specific mk include files as described in FORT. In some situations you will need to edit the appropriate mk file in the Compilers directory (i.e. Compilers/Linux-gfortran.mk). Rather than editing the mk files released with ROMS and possibly creating conflicts during updates, we suggest that you copy the appropriate system specific mk file and the Compilers/make_macros.mk to another folder and set COMPILERS to that folder.

MY_CPP_FLAGS

Additional CPP options to activate. Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the -D syntax. For example, to write out time-averaged NetCDF file use -DAVERAGES.

USE_MPI

Compile your application using MPI libraries.

USE_MPIF90

If USE_MPI is activated, use this to turn on compilation via the script mpif90. This is needed in some Linux operating systems. In some systems with native MPI libraries the compilation does not require MPICH type scripts. This macro is also convenient when there are several Fortran compiliers (ifort, pgf90, pathf90) in the system that use mpif90. In this case, the user needs to select the desired compiler via the FORT variable and turn on both USE_MPI and USE_MPIF90 macros.

FORT

The ROMS make system will include a file with all the settings that depend on the system and the compiler. The name of this file is determined from the output of the uname command and the value of FORT (i.e. Linux-ifort.mk). Set your compiler here from the following list:

Operating System Compiler(s)

AIX: xlf
CYGWIN: df, g95, gfortran, ifort
Darwin: f90, ifort, pgi, xlf
IRIX64: f90
Linux: ftn, g95, gfortran, ifc, ifort, necsx, path, pgi
MINGW: g95, gfortran
OSF1: f90
SunOS: f95, ftn
UNICOS-mk: f90
UNICOS-mp: ftn
UNICOS-sk: f90

Feel free to send us additional rule files to include!

USE_OpenMP

Compile your application using OpenMP librairies and compiler options.

USE_DEBUG

Activate debugging compiler options.

USE_LARGE

If applicable, activate 64-bit compilation.

USE_NETCDF4

If applicable, link with NetCDF-4 library. Notice that the NetCDF-4 library needs both the HDF5 and, if you want to enable parallel I/O, MPI libraries.

MY_HEADER_DIR

This is the Directory where your application's header file is located. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that is included in cppdefs.h. Notice that if the header file is located in the ROMS/Include directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.

MY_ANALYTICAL_DIR

This is the directory containing the user's analytic files, if any. Templates of the analytic expressions are provided in the User/Functionals directory. If analytic expressions are activated, the makefile will first load the ones located in MY_ANALYTICAL_DIR and then the ones located in ROMS/Functionals. We recommended that you do not modify the analytical expressions in the ROMS/Functionals directory since these are the official versions of these files and updates will conflict with your changes. The user however can modify those in User/Functionals or some other directory and specify which directory with this variable.

BINDIR

Directory where your oceanG, oceanM, oceanO, or oceanS executable will be created.

SCRATCH_DIR

Directory where your temporary objects and files will be placed during the build process.

Library and Executable Paths

There is a section (the one from build.sh is shown below) in the the build scripts to make it easier to switch between different compilers and MPI implementations. This will only apply to MPI implementations that use the mpif90 wrapper script (MPICH, MPICH2, OpenMPI, etc.).

if ($?USE_MPIF90) then
switch ($FORT)

case "ifort"
# setenv PATH /opt/intelsoft/mpich/bin:$PATH
# setenv PATH /opt/intelsoft/mpich2/bin:$PATH
setenv PATH /opt/intelsoft/openmpi/bin:$PATH
breaksw

case "pgi"
setenv PATH /opt/pgisoft/mpich/bin:$PATH
# setenv PATH /opt/pgisoft/mpich2/bin:$PATH
# setenv PATH /opt/pgisoft/openmpi/bin:$PATH
breaksw

case "gfortran"
setenv PATH /opt/gfortransoft/mpich/bin:$PATH
# setenv PATH /opt/gfortransoft/mpich2/bin:$PATH
# setenv PATH /opt/gfortransoft/openmpi/bin:$PATH
breaksw

case "g95"
# setenv PATH /opt/g95soft/mpich2/bin:$PATH
setenv PATH /opt/g95soft/openmpi/bin:$PATH
breaksw

endsw
endif

Keep in mind that you need to adjust the paths to you MPI implementations for your particular system. You must also set USE_MPIF90 to on.

The path of the libraries required by ROMS can be set using environment variables which take precedence over the values specified in the makefile macro definitions file (Compilers/*.mk). If so desired, uncomment the local USE_MY_LIBS definition in the build script and edit the paths to your values. For most applications, only the location of the NetCDF library (NETCDF_LIBDIR) and include directory (NETCDF_INCDIR) are needed. Below is an except from this section.

setenv USE_MY_LIBS on

if ($?USE_MY_LIBS) then
switch ($FORT)

case "ifort"
setenv ESMF_DIR /opt/intelsoft/esmf
setenv ESMF_OS Linux
setenv ESMF_COMPILER ifort
setenv ESMF_BOPT O
setenv ESMF_ABI 64
setenv ESMF_COMM mpich
setenv ESMF_SITE default
setenv MCT_INCDIR /opt/intelsoft/mct/include
setenv MCT_LIBDIR /opt/intelsoft/mct/lib

setenv ARPACK_LIBDIR /opt/intelsoft/serial/ARPACK
if ($?USE_MPI) then
# setenv PARPACK_LIBDIR /opt/intelsoft/mpich/PARPACK
setenv PARPACK_LIBDIR /opt/intelsoft/mpich2/PARPACK
# setenv PARPACK_LIBDIR /opt/intelsoft/openmpi/PARPACK
endif

if ($?USE_NETCDF4) then
if ($?USE_MPI) then
# setenv NETCDF_INCDIR /opt/intelsoft/mpich/netcdf4/include
# setenv NETCDF_LIBDIR /opt/intelsoft/mpich/netcdf4/lib
# setenv HDF5_LIBDIR /opt/intelsoft/mpich/hdf5/lib

setenv NETCDF_INCDIR /opt/intelsoft/mpich2/netcdf4/include
setenv NETCDF_LIBDIR /opt/intelsoft/mpich2/netcdf4/lib
setenv HDF5_LIBDIR /opt/intelsoft/mpich2/hdf5/lib

# setenv NETCDF_INCDIR /opt/intelsoft/openmpi/netcdf4/include
# setenv NETCDF_LIBDIR /opt/intelsoft/openmpi/netcdf4/lib
# setenv HDF5_LIBDIR /opt/intelsoft/openmpi/hdf5/lib
else
setenv NETCDF_INCDIR /opt/intelsoft/serial/netcdf4/include
setenv NETCDF_LIBDIR /opt/intelsoft/serial/netcdf4/lib
setenv HDF5_LIBDIR /opt/intelsoft/serial/hdf5/lib
endif
else
setenv NETCDF_INCDIR /opt/intelsoft/serial/netcdf3/include
setenv NETCDF_LIBDIR /opt/intelsoft/serial/netcdf3/lib
endif
breaksw

. . .

endsw
endif

Notice that when the USE_NETCDF4 macro is activated, we need to choose between serial and parallel versions of the NetCDF-4/HDF5 libraries. The parallel library uses parallel I/O through MPI-I/O so we also need to compile with the MPI library. This is fine in ROMS distributed-memory applications. However, in serial or shared-memory ROMS applications we need to use the serial version of NetCDF-4/HDF5 to avoid conflicts with the compiler. Recall also that the MPI library comes in several flavors: MPICH, MPICH2, OpenMPI.

Notes and Recommendations

The new structure of ROMS (version 3.0 or higher) allows user specific CPP definitions and analytical options to be set in a small number of separate standalone header files. Therefore, users no longer have to modify small blocks of code within huge files as was necessary in previous versions (cppdefs.h and analytical.F). This makes it easy to distribute the configuration for a specific application via a small number of source code related files plus the inputs (grid, initial conditions, and forcing NetCDF files), while allowing users of the application to keep current with central code improvements via svn. This also takes care of all licensing issues. You are free to include your own statement of your contributions in the files you distribute. The license and copyright statements by the ROMS developer group remain in the source code distributed at www.myroms.org.

We recommend that users work with a local source code that is regularly updated with the official version at the ROMS svn repository using svn update. This will allow bug fixes and new code developments to be pushed out to users and minimize the risk of a user unwittingly working with code with known flaws. There are a few standalone applications in ROMS website Datasets menu. The user can download any of these set-ups. Note that these applications do not include the source code since the user can checkout the latest version of ROMS from the svn repository. The recommended priactice is to create a Projects root directory containing all applications. For example, the double gyre test case (double_gyre.tar.gz) has the following directory structure:

 Projects                               root directory 
         /double_gyre                   idealized double gyre test case 
                     /Data              input NetCDF files 
                     /Forward           nonlinear model, ensembles 
                     /IS4DVAR           incremental, strong contraint 4DVar 
                     /OBS               data assimilation observations

If you check the IS4DVAR sub-directory, you will find the following files:

 Projects 
         /double_gyre 
                     /IS4DVAR 
                             /build.sh
                             /double_gyre.h
                             /job_is4dvar.sh
                             /is4dvar.in
                             /ocean_double_gyre.in
                             /s4dvar.in

Notice that there is a local copy of the header file double_gyre.h containing the CPP options to use in this application. We highly recommend that the user have a local copy of the header file instead of editing the distributed files in the ROMS/Include directory. During the compilation build.sh will direct make to use the appropriate one.