Difference between revisions of "build roms"

From WikiROMS
Jump to navigationJump to search
(31 intermediate revisions by 3 users not shown)
Line 1: Line 1:
<div class="title">Build Script - <span class="red">build.sh</span></div>
<div class="title">Build Script - <span class="red">build_roms.sh, build_roms.bash</span></div>


An alternative way to compile and link ROMS is to use the <span class="red">build.sh</span> script located in the '''ROMS/Bin''' directory. Instead of editing the [[makefile]] often, the user may generate a build script for each of his/her applications. The build script uses environmental variables that overwrite the makefile targets defined with <span class="red">?=</span> syntax. This means that [[gmake | make]] will check whether an environment variable by that name is set in the script that calls make. The environmental variable take precedence and overwrites makefile default value. Its scope is local to the build script execution, allowing compiling different applications at the same time. There is not need to maintain separate makefiles, or frequently edit the makefile to run separated applications.
As mentioned in [[makefile]], you need to provide settings for some user-defined choices before you can compile ROMS. If you have more than one application (or more than one compiler), you may get tired of editing the [[makefile]]. One option is to have a makefile for each configuration. The recommended solution, however, is to use the ROMS build script.


  <div class="box">'''Usage:'''<br /><br /><span class="red" style="padding-left:2em">./build.sh [options]</span><br /><br />'''Options:'''<br /><br /><span class="red" style="padding-left:2em;padding-right:70px;">-j N</span>Compile in parallel using N CPUsOmit argument to compile on all availabe CPUs.<br /><span class="red" style="padding-left:2em;padding-right:20px;">-noclean</span>Do not clean already compiled objects.</div>
 
__TOC__
 
 
==Introduction==
There are two of these build scripts in the '''ROMS/Bin''' directory: '''build_roms.sh''' (which is surprisingly a '''csh''' script) and '''build_roms.bash'''.  The build scripts use environment variables to provide values for the user-defined [[make]] variables, overwriting those found in the ROMS '''makefile'''. Just as in the multiple '''makefile''' option, you will need as many copies of the build script as you have applications. However, the scope of these variables is local to the build script, allowing you to compile different applications at the same time from the same sources as long as each '''$([[#SCRATCH_DIR|SCRATCH_DIR]])''' is unique.
 
:<div class="box">'''Usage:'''<br /><br /><span class="red"> ./build_roms.sh [options]</span><br /><br />'''Options:'''<br /><br /><span class="red">  -j [N]</span>    Compile in parallel using <span class="red">N</span> CPUs<br />                omit argument for all available CPUs.<br /><br /><span class="red">  -p macro</span>  Prints any Makefile macro value. For example,<br /><br />                  build.sh -p FFLAGS<br /><br /><span class="red">  -noclean</span>  Do not clean already compiled objects.<br /><br />'''Example:'''<br /><br /><span class="green">  ./build.sh -j 2</span></div>
 
 
==User Definable Compilation Options==
ROMS has a growing list of choices the user must make about the compilation before starting the compile process. These user-defined variables can be set in the build script. Since we use '''gnu make''', it is possible to set the value of these variables in the Unix environment, rather than inside the '''Makefile''' (see [[gmake]]). The user-definable variables understood by the ROMS '''makefile''' are:
 
 
<section begin=ROMS_APPLICATION />;<span id="ROMS_APPLICATION"></span>[[ROMS_APPLICATION]]
:CPP option defining a particular application (for example, [[UPWELLING]]). The makefile will include its associated header file which is located in either the '''ROMS/Include''' directory or the path specified in the [[#MY_HEADER_DIR|MY_HEADER_DIR]] definition. The header file name is the lowercase value of [[ROMS_APPLICATION]] with the '''.h''' extension (for example, [[upwelling.h]]). Recall the CPP option defining an application must be unique.<section end=ROMS_APPLICATION />
 
 
<section begin=MY_ROOT_DIR />;<span id="MY_ROOT_DIR"></span>[[MY_ROOT_DIR]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. It can be used to set the base path to your ROMS source code and/or your application input and output files. This variable is not strictly necessary but can make setting the necessary paths easier.<section end=MY_ROOT_DIR />
 
 
<section begin=MY_PROJECT_DIR />;<span id="MY_PROJECT_DIR"></span>[[MY_PROJECT_DIR]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. [[MY_PROJECT_DIR]] is most often used to set the paths: [[#MY_HEADER_DIR|MY_HEADER_DIR]], [[#MY_ANALYTICAL_DIR|MY_ANALYTICAL_DIR]], and [[#BINDIR|BINDIR]] and to set the base path to [[#SCRATCH_DIR|SCRATCH_DIR]]. This can be seen near the bottom of the build script.<section end=MY_PROJECT_DIR />
 
 
<section begin=MY_ROMS_SRC />;<span id="MY_ROMS_SRC"></span>[[MY_ROMS_SRC]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. It is the location of your ROMS source code and is necessary for the build script to work properly.<section end=MY_ROMS_SRC />
 
 
<section begin=COMPILERS />;<span id="COMPILERS"></span>[[COMPILERS]]
: Directory where [[make]] can find your system/compiler specific mk include files as described in [[#FORT|FORT]]. In some situations you will need to edit the appropriate mk file in the Compilers directory (i.e. '''Compilers/Linux-gfortran.mk'''). Rather than editing the mk files released with ROMS and possibly creating conflicts during updates, we suggest that you copy the appropriate system specific mk file and the '''Compilers/make_macros.mk''' to another folder and set [[COMPILERS]] to that folder.<section end=COMPILERS />
 
 
<section begin=MY_CPP_FLAGS />;<span id="MY_CPP_FLAGS"></span>[[MY_CPP_FLAGS]]
: Additional CPP options to activate.  Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the '''-D''' syntax. Notice that you need to use your shell's quoting syntax to enclose the definitions. For example:
:<div class="box">setenv [[MY_CPP_FLAGS]] "-D[[Options#AVERAGES|AVERAGES]]"<br />setenv [[MY_CPP_FLAGS]] "${[[MY_CPP_FLAGS]]} -D[[Options#ANA_INITIAL|ANA_INITIAL]]"</div>
:can be use to write time-averaged fields and use analytical initial conditions. Notice that you can have as many definitions as you want by appending values.<section end=MY_CPP_FLAGS />
 
 
{{warning}}'''Warning:''' Be sure to leave environment variables meant to be off set to an empty string or commented out.  Any string value (including '''off''') will evaluate to '''TRUE''' in conditional if-statements.
 
 
<section begin=USE_MPI />;<span id="USE_MPI"></span>[[USE_MPI]]
: Compile your application using '''MPI''' (distributed-memory) libraries.<section end=USE_MPI />
 
 
<section begin=USE_MPIF90 />;<span id="USE_MPIF90"></span>[[USE_MPIF90]]
: If [[#USE_MPI|USE_MPI]] is activated, use this to turn on compilation via the script <span class="red">mpif90</span>. This is needed in most Linux operating systems. In some systems with native '''MPI''' libraries the compilation does not require '''MPICH''' type scripts. This macro is also convenient when there are several Fortran compiliers (<span class="red">ifort</span>, <span class="red">pgf90</span>, <span class="red">pathf90</span>) in the system that use <span class="red">mpif90</span>. In this case, the user needs to select the desired compiler via the [[#FORT|FORT]] variable and turn on both [[#USE_MPI|USE_MPI]] and [[USE_MPIF90]] macros.<section end=USE_MPIF90 />
 
 
<section begin=which_MPI />;<span id="which_MPI"></span>[[which_MPI]]
: If [[#USE_MPI|USE_MPI]] and [[#USE_MPIF90|USE_MPIF90]] are activated, this macro is used to determine which <span class="red">mpif90</span> to compile with. This macro is convenient when there are several Fortran compiliers (<span class="red">ifort</span>, <span class="red">pgf90</span>, etc.) in the system that use <span class="red">mpif90</span>. The choice of which <span class="red">mpif90</span> to use will be determined by the combination of this macro and the [[#FORT|FORT]] macro.<section end=which_MPI />
 
 
<section begin=USE_OpenMP />;<span id="USE_OpenMP"></span>[[USE_OpenMP]]
: Compile your application using OpenMP librairies and compiler options. '''Note:''' you cannot set [[USE_OpenMP]] and [[#USE_MPI|USE_MPI]] at the same time.<section end=USE_OpenMP />
 
 
<section begin=FORT />;<span id="FORT"></span>[[FORT]]
: The ROMS make system will include a file with all the settings that depend on the system and the compiler. The name of this file is determined from the output of the <span class="red">uname</span> command and the value of [[FORT]] (''i.e.'' '''Linux-ifort.mk'''). Set your compiler here from the following list:
::<div class="box">  Operating System        Compiler(s)<br><br>    AIX:                   xlf<br>    CYGWIN:                df, g95, gfortran, ifort<br>    Darwin:                f90, gfortran, ifort, pgi, xlf<br>    IRIX64:                f90<br>    Linux:                  ftn, g95, gfortran, ifc, ifort, necsx, path, pgi<br>    MINGW:                  g95, gfortran<br>    OSF1:                  f90<br>    SunOS:                  f95, ftn<br>    UNICOS-mk:              f90<br>    UNICOS-mp:              ftn<br>    UNICOS-sk:              f90</div>
: Feel free to send us additional rule files to include!<section end=FORT />
 
 
<section begin=USE_DEBUG />;<span id="USE_DEBUG"></span>[[USE_DEBUG]]
: Activate debugging compiler options.<section end=USE_DEBUG />
 
 
<section begin=USE_LARGE />;<span id="USE_LARGE"></span>[[USE_LARGE]]
: If applicable, activate 64-bit compilation.<section end=USE_LARGE />
 
 
<section begin=USE_NETCDF4 />;<span id="USE_NETCDF4"></span>[[USE_NETCDF4]]
: If applicable, link with '''NetCDF-4''' library. Notice that the '''NetCDF-4''' library needs both the '''HDF5''' and, if you want to enable parallel I/O, '''MPI''' libraries. In order to write the output files in '''NetCDF-4/HDF5''' format, you will need to set the [[Options#HDF5|HDF5]] CPP Option.<section end=USE_NETCDF4 />
 
 
<section begin=USE_PARALLEL_IO />;<span id="USE_PARALLEL_IO"></span>[[USE_PARALLEL_IO]]
: Activate parallel I/O using parallel enabled '''NetCDF-4/HDF5''' libraries. You will also need to set the [[Options#HDF5|HDF5]] and [[Options#PARALLEL_IO|PARALLEL_IO]] CPP Options to enable parallel I/O.<section end=USE_PARALLEL_IO />
 
 
<section begin=WRF_SRC_DIR />;<span id="WRF_SRC_DIR"></span>[[WRF_SRC_DIR]]
: WRF cannot be built in a directory specified by the user; it must be built in its own root directory and cannot be moved when debugging with tools like TotalView. If you wish to couple ROMS with the Earth Systems Model (ESM) WRF, ROMS needs to know the WRF root directory.<section end=WRF_SRC_DIR />
 
 
<section begin=USE_MY_LIBS />;<span id="USE_MY_LIBS"></span>[[USE_MY_LIBS]]
: Use custom library paths for '''ESMF''', '''HDF5''', '''NetCDF''' libraries, etc.<section end=USE_MY_LIBS />
 
 
<section begin=MY_PATHS />;<span id="MY_PATHS"></span>[[MY_PATHS]]
: Path to your <span class="red">my_build_paths.sh</span> (or .bash) file that contains custom library paths for your system.<section end=MY_PATHS />
 
 
<section begin=MY_HEADER_DIR />;<span id="MY_HEADER_DIR"></span>[[MY_HEADER_DIR]]
: This is the Directory where your application's header file is located. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that is included in [[cppdefs.h]]. Notice that if the header file is located in the '''ROMS/Include''' directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.<section end=MY_HEADER_DIR />
 
 
<section begin=MY_ANALYTICAL_DIR />;<span id="MY_ANALYTICAL_DIR"></span>[[MY_ANALYTICAL_DIR]]
: This is the directory containing the user's [[Functionals|analytic files]], if any. Templates of the analytic expressions are provided in the '''User/Functionals''' directory. If analytic expressions are activated, the makefile will first load the ones located in [[MY_ANALYTICAL_DIR]] and then the ones located in '''ROMS/Functionals'''. We recommended that you do not modify the analytical expressions in the '''ROMS/Functionals''' directory since these are the official versions of these files and updates will conflict with your changes.  The user however can modify those in '''User/Functionals''' or some other directory and specify which directory with this variable.<section end=MY_ANALYTICAL_DIR />
 
 
<section begin=BINDIR />;<span id="BINDIR"></span>[[BINDIR]]
: Directory where your <span class="red">oceanG</span>, <span class="red">oceanM</span>, <span class="red">oceanO</span>, or <span class="red">oceanS</span> executable will be created.<section end=BINDIR />
 
 
<section begin=SCRATCH_DIR />;<span id="SCRATCH_DIR"></span>[[SCRATCH_DIR]]
: Directory where your temporary objects and files will be placed during the build process.<section end=SCRATCH_DIR />
 
==Library and Executable Paths==
 
{{note}}'''Note:''' in SVN revision 933 (January 26, 2019) the custom libraries section was removed from the ROMS build script and placed in <span class="red">Compilers/my_build_paths.sh</span> and <span class="red">Compilers/my_build_paths.bash</span> to both streamline the ROMS build script and make setting up new applications less tedious. More information can be found in ROMS [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. Prior to ROMS release 933, the paths mentioned below were included in the ROMS build script.
 
 
The path of the libraries and MPI implementations required by ROMS can be set using environment variables which take precedence over the values specified in the makefile macro definitions file ('''Compilers/*.mk'''). If so desired, uncomment the local [[USE_MY_LIBS]] definition above and properly set [[MY_PATHS]]. There is a section (the one from <span class="red">my_build_paths.sh</span> is shown below) in the [[MY_PATHS]] file to make it easier to switch between different compilers and '''MPI''' implementations. This will only apply to '''MPI''' implementations that use the <span class="red">mpif90</span> wrapper script ('''MPICH''', '''MPICH2''', '''OpenMPI''', etc.).
 
:<div class="box">if ($?USE_MPIF90) then<br />  switch ($FORT)<br /><br />    case "ifort"<br />      if ($which_MPI == "mpich" ) then<br />        setenv MPI_ROOT /opt/intelsoft/mpich<br />      else if ($which_MPI == "mpich2" ) then<br />        setenv MPI_ROOT /opt/intelsoft/mpich2<br />      else if ($which_MPI == "openmpi" ) then<br />        setenv MPI_ROOT /opt/intelsoft/openmpi<br />      else if ($which_MPI == "mvapich2" ) then<br />        setenv MPI_ROOT /opt/intelsoft/mvapich2<br />      endif<br />      setenv PATH      ${MPI_ROOT}/bin:$PATH<br />      setenv MPI_INCDIR ${MPI_ROOT}/include<br />    breaksw<br /><br />    case "pgi"<br />      if ($which_MPI == "mpich" ) then<br />        setenv MPI_ROOT /opt/pgisoft/mpich<br />      else if ($which_MPI == "mpich2" ) then<br />        setenv MPI_ROOT /opt/pgisoft/mpich2<br />      else if ($which_MPI == "openmpi" ) then<br />        setenv MPI_ROOT /opt/pgisoft/openmpi<br />      else if ($which_MPI == "mvapich2" ) then<br />        setenv MPI_ROOT /opt/pgisoft/mvapich2<br />      endif<br />      setenv PATH      ${MPI_ROOT}/bin:$PATH<br />      setenv MPI_INCDIR ${MPI_ROOT}/include<br />    breaksw<br /><br />    case "gfortran"<br />      if ($which_MPI == "mpich2" ) then<br />        setenv MPI_ROOT /opt/gfortransoft/mpich2<br />      else if ($which_MPI == "openmpi" ) then<br />        setenv MPI_ROOT /opt/gfortransoft/openmpi<br />      else if ($which_MPI == "mvapich2" ) then<br />        setenv MPI_ROOT /opt/gfortransoft/mvapich2<br />      endif<br />      setenv PATH      ${MPI_ROOT}/bin:$PATH<br />      setenv MPI_INCDIR ${MPI_ROOT}/include<br />    breaksw<br /><br />  endsw<br />endif</div>
 
{{note}}Keep in mind that you need to adjust the paths to you '''MPI''' implementations for your particular system. You must also set '''USE_MPIF90''' to on.
 
For most applications, only the location of the '''NetCDF''' library ('''NETCDF_LIBDIR''') and include directory ('''NETCDF_INCDIR''') are needed. Notice that when the [[#USE_NETCDF4|USE_NETCDF4]] macro is activated, we need the serial or parallel version of the '''NetCDF-4/HDF5''' library. The configuration script '''NF_CONFIG''' (available since NetCDF 4.2) is used to set up all the required libraries according to the installed options ('''OPeNDAP''', '''NetCDF4/HDF5''' file format). The parallel library uses the '''MPI-I/O''' layer (usually available in '''MPICH2''' and '''OpenMPI''') requiring compiling with the selected '''MPI''' library.
 
In ROMS distributed-memory applications, you may use either the serial or parallel version of the '''NetCDF-4/HDF5''' library. The parallel version is required when parallel I/O is activated (ROMS cpp option [[Options#PARALLEL_IO|PARALLEL_IO]] and [[Options#HDF5|HDF5]]).
 
However, in serial or shared-memory ROMS applications, we need to use the serial version of the '''NetCDF-4/HDF5''' to avoid conflicts with the compiler. We cannot activate '''MPI''' constructs in serial or shared-memory ROMS code. Hybrid parallelism is not possible.
 
 
Below is an excerpt from the custom library section.
 
:<div class="box">setenv MPI_SOFT ""<br /><br />switch ($FORT)<br /><br /># Intel Compiler:<br /><br />  case "ifort"<br />    setenv ESMF_COMPILER        intelgcc<br />    if ($?USE_DEBUG) then<br />      setenv ESMF_BOPT          g<br />    else<br />      setenv ESMF_BOPT          O<br />    endif<br /><br />    setenv ESMF_ABI            64<br />    setenv ESMF_COMM            ${which_MPI}<br />    setenv ESMF_SITE            default<br /><br />    setenv ARPACK_LIBDIR        /opt/intelsoft/serial/ARPACK<br /><br />    if ($?USE_MPI) then<br />      if ($which_MPI == "mpich" ) then<br />        setenv MPI_SOFT        /opt/intelsoft/mpich<br />      else if ($which_MPI == "mpich2" ) then<br />        setenv MPI_SOFT        /opt/intelsoft/mpich2<br />      else if ($which_MPI == "openmpi" ) then<br />        setenv MPI_SOFT        /opt/intelsoft/openmpi<br />      else if ($which_MPI == "mvapich2" ) then<br />        setenv MPI_SOFT        /opt/intelsoft/mvapich2<br />      endif<br /><br />      setenv MCT_INCDIR        ${MPI_SOFT}/mct/include<br />      setenv MCT_LIBDIR        ${MPI_SOFT}/mct/lib<br />      setenv PARPACK_LIBDIR    ${MPI_SOFT}/PARPACK<br />    endif<br /><br />    if ($?USE_NETCDF4) then<br />      if ($?USE_PARALLEL_IO && $?USE_MPI) then<br />        setenv ESMF_DIR        ${MPI_SOFT}/esmf_nc4<br />        setenv NETCDF          ${MPI_SOFT}/netcdf4<br />        setenv NF_CONFIG        ${NETCDF}/bin/nf-config<br />        setenv NETCDF_INCDIR    ${NETCDF}/include<br />        setenv NETCDF4          1<br />      else<br />        setenv ESMF_DIR        ${MPI_SOFT}/esmf_nc4<br />        setenv NETCDF          /opt/intelsoft/serial/netcdf4<br />        setenv NF_CONFIG        ${NETCDF}/bin/nf-config<br />        setenv NETCDF_INCDIR    ${NETCDF}/include<br />        setenv NETCDF4          1<br />      endif<br />    else<br />      setenv ESMF_DIR          ${MPI_SOFT}/esmf_nc3<br />      setenv NETCDF            /opt/intelsoft/serial/netcdf3<br />      setenv NETCDF_INCDIR      ${NETCDF}/include<br />      setenv NETCDF_LIBDIR      ${NETCDF}/lib<br />    endif<br /><br />    if ($?USE_HDF5) then<br />      if ($?USE_PARALLEL_IO && $?USE_MPI) then<br />        setenv HDF5            ${MPI_SOFT}/hdf5<br />        setenv HDF5_LIBDIR      ${HDF5}/lib<br />        setenv HDF5_INCDIR      ${HDF5}/include<br />      else<br />        setenv HDF5            /opt/intelsoft/serial/hdf5<br />        setenv HDF5_LIBDIR      ${HDF5}/lib<br />        setenv HDF5_INCDIR      ${HDF5}/include<br />      endif<br />    endif<br /><br />  breaksw<br />. . .</div>
 
==Notes and Recommendations==
The new structure of ROMS (version 3.0 or higher) allows user specific CPP definitions and analytical options to be set in a small number of separate standalone header files. Therefore, users no longer have to modify small blocks of code within huge files as was necessary in previous versions ([[cppdefs.h]] and [[analytical.F]]). This makes it easy to distribute the configuration for a specific application via a small number of source code related files plus the inputs (grid, initial conditions, and forcing NetCDF files), while allowing users of the application to keep current with central code improvements via [[Subversion | svn]]. This also takes care of all licensing issues. You are free to include your own statement of your contributions in the files you distribute. The license and copyright statements by the ROMS developer group remain in the source code distributed at [https://www.myroms.org www.myroms.org].
 
We recommend that users work with a local source code that is regularly updated with the official version at the ROMS [https://www.myroms.org/projects/src svn repository] using '''svn update'''. This will allow bug fixes and new code developments to be pushed out to users and minimize the risk of a user unwittingly working with code with known flaws. The user can download any of these set-ups from the  [https://www.myroms.org/svn/src/test test svn repository]. Note that these applications do not include the source code since the user can checkout the latest version of ROMS from the '''svn''' repository. The recommended practice is to create a Projects root directory containing all applications.  For example, the WC13 root directory for the data assimilation tutorial has the following strcuture:
 
  <span class="twilightBlue">/WC13                    Root directory for the California System 4D-Var applications
      /ARRAYS_MODES        Stabilized representer matrix array modes and clipping
      /DATA                Input NetCDF files
      /Functionals        Analytical files
      /I4DVAR              Primal for of incremental, strong constraint 4D-Var  (I4D-Var)
      /I4DVAR_impact      I4D-Var observation impact
      /Normalization      4D-Var error covariance normalization coefficients
      /PSAS                Dual form of 4D-Var, Physical-space Statisitical Analysis System (4D-PSAS)
      /PSAS_impact        4D-PSAS observation impact
      /PSAS_sensitivity    4D-PSAS observation sensitivity (adjoint of 4D-PSAS)
      /R4DVAR              Dual form of 4D-Var, indirect representer method (R4D-Var)
      /R4DVAR_impact      R4D-Var observation impact
      /R4DVAR_sensitivity  R4D-Var observation sensitivity (adjoint of R4D-Var)
      /plotting            4D-Var plotting scripts (Matlab and ROMS plotting package)</span>
 
If you check the '''I4DVAR''' sub-directory, you will find the following files:
 
  <span class="twilightBlue"> /WC13
        /I4DVAR
              /Readme              Information and instructions
              /build_roms.bash          bash shell script to compile application
              /build_roms.sh            csh Unix script to compile application
              /job_i4dvar.sh      job configuration script
              /roms_wc13.in      ROMS standard input script for WC13
              /s4dvar.in          4D-Var standard input script template
              /wc13.h              WC13 header with CPP options</span>
 
Notice that there is a local copy of the header file <span class="red">wc13.h</span> containing the CPP options to use in this application. We highly recommend that the user have a local copy of the header file instead of editing the distributed files in the '''ROMS/Include''' directory. During the compilation <span class="red">build_roms.sh</span> will direct '''make''' to use the appropriate one.

Revision as of 13:26, 7 May 2019

Build Script - build_roms.sh, build_roms.bash

As mentioned in makefile, you need to provide settings for some user-defined choices before you can compile ROMS. If you have more than one application (or more than one compiler), you may get tired of editing the makefile. One option is to have a makefile for each configuration. The recommended solution, however, is to use the ROMS build script.



Introduction

There are two of these build scripts in the ROMS/Bin directory: build_roms.sh (which is surprisingly a csh script) and build_roms.bash. The build scripts use environment variables to provide values for the user-defined make variables, overwriting those found in the ROMS makefile. Just as in the multiple makefile option, you will need as many copies of the build script as you have applications. However, the scope of these variables is local to the build script, allowing you to compile different applications at the same time from the same sources as long as each $(SCRATCH_DIR) is unique.

Usage:

./build_roms.sh [options]

Options:

-j [N] Compile in parallel using N CPUs
omit argument for all available CPUs.

-p macro Prints any Makefile macro value. For example,

build.sh -p FFLAGS

-noclean Do not clean already compiled objects.

Example:

./build.sh -j 2


User Definable Compilation Options

ROMS has a growing list of choices the user must make about the compilation before starting the compile process. These user-defined variables can be set in the build script. Since we use gnu make, it is possible to set the value of these variables in the Unix environment, rather than inside the Makefile (see gmake). The user-definable variables understood by the ROMS makefile are:


ROMS_APPLICATION
CPP option defining a particular application (for example, UPWELLING). The makefile will include its associated header file which is located in either the ROMS/Include directory or the path specified in the MY_HEADER_DIR definition. The header file name is the lowercase value of ROMS_APPLICATION with the .h extension (for example, upwelling.h). Recall the CPP option defining an application must be unique.


MY_ROOT_DIR
This path is internal to the build script and has no equivalent in the makefile. It can be used to set the base path to your ROMS source code and/or your application input and output files. This variable is not strictly necessary but can make setting the necessary paths easier.


MY_PROJECT_DIR
This path is internal to the build script and has no equivalent in the makefile. MY_PROJECT_DIR is most often used to set the paths: MY_HEADER_DIR, MY_ANALYTICAL_DIR, and BINDIR and to set the base path to SCRATCH_DIR. This can be seen near the bottom of the build script.


MY_ROMS_SRC
This path is internal to the build script and has no equivalent in the makefile. It is the location of your ROMS source code and is necessary for the build script to work properly.


COMPILERS
Directory where make can find your system/compiler specific mk include files as described in FORT. In some situations you will need to edit the appropriate mk file in the Compilers directory (i.e. Compilers/Linux-gfortran.mk). Rather than editing the mk files released with ROMS and possibly creating conflicts during updates, we suggest that you copy the appropriate system specific mk file and the Compilers/make_macros.mk to another folder and set COMPILERS to that folder.


MY_CPP_FLAGS
Additional CPP options to activate. Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the -D syntax. Notice that you need to use your shell's quoting syntax to enclose the definitions. For example:
can be use to write time-averaged fields and use analytical initial conditions. Notice that you can have as many definitions as you want by appending values.


WarningWarning: Be sure to leave environment variables meant to be off set to an empty string or commented out. Any string value (including off) will evaluate to TRUE in conditional if-statements.


USE_MPI
Compile your application using MPI (distributed-memory) libraries.


USE_MPIF90
If USE_MPI is activated, use this to turn on compilation via the script mpif90. This is needed in most Linux operating systems. In some systems with native MPI libraries the compilation does not require MPICH type scripts. This macro is also convenient when there are several Fortran compiliers (ifort, pgf90, pathf90) in the system that use mpif90. In this case, the user needs to select the desired compiler via the FORT variable and turn on both USE_MPI and USE_MPIF90 macros.


which_MPI
If USE_MPI and USE_MPIF90 are activated, this macro is used to determine which mpif90 to compile with. This macro is convenient when there are several Fortran compiliers (ifort, pgf90, etc.) in the system that use mpif90. The choice of which mpif90 to use will be determined by the combination of this macro and the FORT macro.


USE_OpenMP
Compile your application using OpenMP librairies and compiler options. Note: you cannot set USE_OpenMP and USE_MPI at the same time.


FORT
The ROMS make system will include a file with all the settings that depend on the system and the compiler. The name of this file is determined from the output of the uname command and the value of FORT (i.e. Linux-ifort.mk). Set your compiler here from the following list:
Operating System Compiler(s)

AIX: xlf
CYGWIN: df, g95, gfortran, ifort
Darwin: f90, gfortran, ifort, pgi, xlf
IRIX64: f90
Linux: ftn, g95, gfortran, ifc, ifort, necsx, path, pgi
MINGW: g95, gfortran
OSF1: f90
SunOS: f95, ftn
UNICOS-mk: f90
UNICOS-mp: ftn
UNICOS-sk: f90
Feel free to send us additional rule files to include!


USE_DEBUG
Activate debugging compiler options.


USE_LARGE
If applicable, activate 64-bit compilation.


USE_NETCDF4
If applicable, link with NetCDF-4 library. Notice that the NetCDF-4 library needs both the HDF5 and, if you want to enable parallel I/O, MPI libraries. In order to write the output files in NetCDF-4/HDF5 format, you will need to set the HDF5 CPP Option.


USE_PARALLEL_IO
Activate parallel I/O using parallel enabled NetCDF-4/HDF5 libraries. You will also need to set the HDF5 and PARALLEL_IO CPP Options to enable parallel I/O.


WRF_SRC_DIR
WRF cannot be built in a directory specified by the user; it must be built in its own root directory and cannot be moved when debugging with tools like TotalView. If you wish to couple ROMS with the Earth Systems Model (ESM) WRF, ROMS needs to know the WRF root directory.


USE_MY_LIBS
Use custom library paths for ESMF, HDF5, NetCDF libraries, etc.


MY_PATHS
Path to your my_build_paths.sh (or .bash) file that contains custom library paths for your system.


MY_HEADER_DIR
This is the Directory where your application's header file is located. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that is included in cppdefs.h. Notice that if the header file is located in the ROMS/Include directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.


MY_ANALYTICAL_DIR
This is the directory containing the user's analytic files, if any. Templates of the analytic expressions are provided in the User/Functionals directory. If analytic expressions are activated, the makefile will first load the ones located in MY_ANALYTICAL_DIR and then the ones located in ROMS/Functionals. We recommended that you do not modify the analytical expressions in the ROMS/Functionals directory since these are the official versions of these files and updates will conflict with your changes. The user however can modify those in User/Functionals or some other directory and specify which directory with this variable.


BINDIR
Directory where your oceanG, oceanM, oceanO, or oceanS executable will be created.


SCRATCH_DIR
Directory where your temporary objects and files will be placed during the build process.

Library and Executable Paths

NoteNote: in SVN revision 933 (January 26, 2019) the custom libraries section was removed from the ROMS build script and placed in Compilers/my_build_paths.sh and Compilers/my_build_paths.bash to both streamline the ROMS build script and make setting up new applications less tedious. More information can be found in ROMS Trac ticket #794. Prior to ROMS release 933, the paths mentioned below were included in the ROMS build script.


The path of the libraries and MPI implementations required by ROMS can be set using environment variables which take precedence over the values specified in the makefile macro definitions file (Compilers/*.mk). If so desired, uncomment the local USE_MY_LIBS definition above and properly set MY_PATHS. There is a section (the one from my_build_paths.sh is shown below) in the MY_PATHS file to make it easier to switch between different compilers and MPI implementations. This will only apply to MPI implementations that use the mpif90 wrapper script (MPICH, MPICH2, OpenMPI, etc.).

if ($?USE_MPIF90) then
switch ($FORT)

case "ifort"
if ($which_MPI == "mpich" ) then
setenv MPI_ROOT /opt/intelsoft/mpich
else if ($which_MPI == "mpich2" ) then
setenv MPI_ROOT /opt/intelsoft/mpich2
else if ($which_MPI == "openmpi" ) then
setenv MPI_ROOT /opt/intelsoft/openmpi
else if ($which_MPI == "mvapich2" ) then
setenv MPI_ROOT /opt/intelsoft/mvapich2
endif
setenv PATH ${MPI_ROOT}/bin:$PATH
setenv MPI_INCDIR ${MPI_ROOT}/include
breaksw

case "pgi"
if ($which_MPI == "mpich" ) then
setenv MPI_ROOT /opt/pgisoft/mpich
else if ($which_MPI == "mpich2" ) then
setenv MPI_ROOT /opt/pgisoft/mpich2
else if ($which_MPI == "openmpi" ) then
setenv MPI_ROOT /opt/pgisoft/openmpi
else if ($which_MPI == "mvapich2" ) then
setenv MPI_ROOT /opt/pgisoft/mvapich2
endif
setenv PATH ${MPI_ROOT}/bin:$PATH
setenv MPI_INCDIR ${MPI_ROOT}/include
breaksw

case "gfortran"
if ($which_MPI == "mpich2" ) then
setenv MPI_ROOT /opt/gfortransoft/mpich2
else if ($which_MPI == "openmpi" ) then
setenv MPI_ROOT /opt/gfortransoft/openmpi
else if ($which_MPI == "mvapich2" ) then
setenv MPI_ROOT /opt/gfortransoft/mvapich2
endif
setenv PATH ${MPI_ROOT}/bin:$PATH
setenv MPI_INCDIR ${MPI_ROOT}/include
breaksw

endsw
endif

NoteKeep in mind that you need to adjust the paths to you MPI implementations for your particular system. You must also set USE_MPIF90 to on.

For most applications, only the location of the NetCDF library (NETCDF_LIBDIR) and include directory (NETCDF_INCDIR) are needed. Notice that when the USE_NETCDF4 macro is activated, we need the serial or parallel version of the NetCDF-4/HDF5 library. The configuration script NF_CONFIG (available since NetCDF 4.2) is used to set up all the required libraries according to the installed options (OPeNDAP, NetCDF4/HDF5 file format). The parallel library uses the MPI-I/O layer (usually available in MPICH2 and OpenMPI) requiring compiling with the selected MPI library.

In ROMS distributed-memory applications, you may use either the serial or parallel version of the NetCDF-4/HDF5 library. The parallel version is required when parallel I/O is activated (ROMS cpp option PARALLEL_IO and HDF5).

However, in serial or shared-memory ROMS applications, we need to use the serial version of the NetCDF-4/HDF5 to avoid conflicts with the compiler. We cannot activate MPI constructs in serial or shared-memory ROMS code. Hybrid parallelism is not possible.


Below is an excerpt from the custom library section.

setenv MPI_SOFT ""

switch ($FORT)

# Intel Compiler:

case "ifort"
setenv ESMF_COMPILER intelgcc
if ($?USE_DEBUG) then
setenv ESMF_BOPT g
else
setenv ESMF_BOPT O
endif

setenv ESMF_ABI 64
setenv ESMF_COMM ${which_MPI}
setenv ESMF_SITE default

setenv ARPACK_LIBDIR /opt/intelsoft/serial/ARPACK

if ($?USE_MPI) then
if ($which_MPI == "mpich" ) then
setenv MPI_SOFT /opt/intelsoft/mpich
else if ($which_MPI == "mpich2" ) then
setenv MPI_SOFT /opt/intelsoft/mpich2
else if ($which_MPI == "openmpi" ) then
setenv MPI_SOFT /opt/intelsoft/openmpi
else if ($which_MPI == "mvapich2" ) then
setenv MPI_SOFT /opt/intelsoft/mvapich2
endif

setenv MCT_INCDIR ${MPI_SOFT}/mct/include
setenv MCT_LIBDIR ${MPI_SOFT}/mct/lib
setenv PARPACK_LIBDIR ${MPI_SOFT}/PARPACK
endif

if ($?USE_NETCDF4) then
if ($?USE_PARALLEL_IO && $?USE_MPI) then
setenv ESMF_DIR ${MPI_SOFT}/esmf_nc4
setenv NETCDF ${MPI_SOFT}/netcdf4
setenv NF_CONFIG ${NETCDF}/bin/nf-config
setenv NETCDF_INCDIR ${NETCDF}/include
setenv NETCDF4 1
else
setenv ESMF_DIR ${MPI_SOFT}/esmf_nc4
setenv NETCDF /opt/intelsoft/serial/netcdf4
setenv NF_CONFIG ${NETCDF}/bin/nf-config
setenv NETCDF_INCDIR ${NETCDF}/include
setenv NETCDF4 1
endif
else
setenv ESMF_DIR ${MPI_SOFT}/esmf_nc3
setenv NETCDF /opt/intelsoft/serial/netcdf3
setenv NETCDF_INCDIR ${NETCDF}/include
setenv NETCDF_LIBDIR ${NETCDF}/lib
endif

if ($?USE_HDF5) then
if ($?USE_PARALLEL_IO && $?USE_MPI) then
setenv HDF5 ${MPI_SOFT}/hdf5
setenv HDF5_LIBDIR ${HDF5}/lib
setenv HDF5_INCDIR ${HDF5}/include
else
setenv HDF5 /opt/intelsoft/serial/hdf5
setenv HDF5_LIBDIR ${HDF5}/lib
setenv HDF5_INCDIR ${HDF5}/include
endif
endif

breaksw
. . .

Notes and Recommendations

The new structure of ROMS (version 3.0 or higher) allows user specific CPP definitions and analytical options to be set in a small number of separate standalone header files. Therefore, users no longer have to modify small blocks of code within huge files as was necessary in previous versions (cppdefs.h and analytical.F). This makes it easy to distribute the configuration for a specific application via a small number of source code related files plus the inputs (grid, initial conditions, and forcing NetCDF files), while allowing users of the application to keep current with central code improvements via svn. This also takes care of all licensing issues. You are free to include your own statement of your contributions in the files you distribute. The license and copyright statements by the ROMS developer group remain in the source code distributed at www.myroms.org.

We recommend that users work with a local source code that is regularly updated with the official version at the ROMS svn repository using svn update. This will allow bug fixes and new code developments to be pushed out to users and minimize the risk of a user unwittingly working with code with known flaws. The user can download any of these set-ups from the test svn repository. Note that these applications do not include the source code since the user can checkout the latest version of ROMS from the svn repository. The recommended practice is to create a Projects root directory containing all applications. For example, the WC13 root directory for the data assimilation tutorial has the following strcuture:

 /WC13                     Root directory for the California System 4D-Var applications
      /ARRAYS_MODES        Stabilized representer matrix array modes and clipping
      /DATA                Input NetCDF files
      /Functionals         Analytical files
      /I4DVAR              Primal for of incremental, strong constraint 4D-Var  (I4D-Var) 
      /I4DVAR_impact       I4D-Var observation impact
      /Normalization       4D-Var error covariance normalization coefficients
      /PSAS                Dual form of 4D-Var, Physical-space Statisitical Analysis System (4D-PSAS)
      /PSAS_impact         4D-PSAS observation impact
      /PSAS_sensitivity    4D-PSAS observation sensitivity (adjoint of 4D-PSAS)
      /R4DVAR              Dual form of 4D-Var, indirect representer method (R4D-Var)
      /R4DVAR_impact       R4D-Var observation impact
      /R4DVAR_sensitivity  R4D-Var observation sensitivity (adjoint of R4D-Var)
      /plotting            4D-Var plotting scripts (Matlab and ROMS plotting package)

If you check the I4DVAR sub-directory, you will find the following files:

  /WC13
       /I4DVAR
              /Readme              Information and instructions
              /build_roms.bash          bash shell script to compile application
              /build_roms.sh            csh Unix script to compile application
              /job_i4dvar.sh       job configuration script
              /roms_wc13.in       ROMS standard input script for WC13
              /s4dvar.in           4D-Var standard input script template
              /wc13.h              WC13 header with CPP options

Notice that there is a local copy of the header file wc13.h containing the CPP options to use in this application. We highly recommend that the user have a local copy of the header file instead of editing the distributed files in the ROMS/Include directory. During the compilation build_roms.sh will direct make to use the appropriate one.