ROMS UNSW2008

Installing and Running ROMS for First Time Users

A tutorial for new ROMS users will be held at the UNSW Computer Labs on Monday 30 March 2009, immediately prior to the ROMS Sydney 2009 User Workshop at the Sydney Institute of Marine Sciences, 31 March to 2 April 2009.

This tutorial is intended for complete newcomers to ROMS. It assumes basic knowledge of working in a UNIX environment, and that the essential components required to compile and execute ROMS are already installed on the host computer network. This wiki page borrows heavily from David's Robertson's excellent Installing ROMS under Cygwin tutorial where you will find more information about setting up the required computing environment (compilers, libraries etc.) for ROMS.

In this tutorial, we cover how to download the code, configure it for an application, and run the model. Error messages that arise during the configuration process will be explained so that these can better be debugged when users return to their home institutions and try to work through this process again.

A follow-in tutorial covering sediment will be presented on Friday 03 April.

An important resource you should use as you get started is the Frequently Asked Questions entry in WikiROMS.

Download ROMS

The disk space available on the UNSW Computer Lab machines is quite limited, so for the purposes of this tutorial we have downloaded the ROMS source code to /srv/ckpt/roms/shared/src on host matht001. Instructions below will explain how to point the build.bash script that compiles ROMS to this directory.

To download the code to your own machine, these are the steps you would follow:

• You must have already registered on the ROMS portal and obtained your ROMS username/password as indicated in the Register. If you are also interested in access to research branches of the sediment code, indicate an interest in the Community Sediment Transport Modeling System (CSTMS) while you are registering. Once your registration is accepted, or if you are already registered, ask Dave Robertson (robertson at marine.rutgers.edu) to add you as a CSTMS user.

• Create a src folder where you will keep the ROMS source code. You can place this wherever you wish in your directory tree (here we assume under your home directory "~") and name it whatever you like.

cd ~
mkdir src

• Check out the ROMS source code replacing bruce with the ROMS user name you registered with.

svn checkout --username bruce https://www.myroms.org/svn/src/trunk src

Note the target directory src at the end of the command. If your code ends up in the wrong place, you may have omitted this.

You will see many lines stream by indicating the files that are being added to your src directory. When it finishes, you can type ls src to see the contents of the directory.

To see the contents of the directory where the code is downloaded for this tutorial, type this:

cd /srv/ckpt/roms/shared
ls src

Customize the Build Script

The ROMS source code comes with a build script in the ROMS/Bin directory. Examples written with bash (build.bash) and csh (build.sh) are provided. The UNSW Computer Lab machines are configured to use bash as the default login shell, so we will work with build.bash. A full description of the build script can be found here.

• In your home directory (you can use some other directory to organize your ROMS projects if you wish) create a new folder named Projects and change into it.

cd ~
mkdir Projects
cd Projects

• Create a folder named upwelling and change into it. ROMS is distributed with several Test Cases and the Upwelling example is the default which we will compile and run here.

mkdir upwelling
cd upwelling

• Copy the build.bash file distributed with ROMS to your Projects/upwelling directory.

cp /srv/ckpt/roms/shared/src/ROMS/Bin/build.bash .

Next we need to configure a few options inside build.bash so that it finds the directories where the source code and your Project are located.

• Open the build.bash script you just copied into your upwelling directory using your preferred text editor, e.g. vi.

vi build.bash

• Scroll down until you find ROMS_APPLICATION. You will notice it is set as follows:

export ROMS_APPLICATION=UPWELLING

We do not need to change this. But this is the first thing you will alter when starting your own project. This tells ROMS the name of an include file that will contain all the directives to the C-PreProcessor to configure your application at compile time. ROMS' rule is change this string to lowercase and append a ".h", so this will search for a file called upwelling.h. It must be in the directory specified by MY_PROJECT_DIR:

• Scroll down until you find MY_PROJECT_DIR and set it as follows:

export MY_PROJECT_DIR=${HOME}/Projects/upwelling

This obviously assumes you put Projects/upwelling under your home directory.

If you frequently move your ROMS project between hosts where you have a different directory structure, e.g. a temporary scratch space, you can use the MY_ROOT_DIR variable to minimize the changes you make to build.bash.

• For example:

export MY_ROOT_DIR=/usr/scratch/bruce
export MY_PROJECT_DIR=${MY_ROOT_DIR}/Projects/upwelling

Next we tell build.bash where to find the ROMS source code downloaded from the svn repository (which you can keep up to date the svn update command - see more on this at LINK ). Note that most of the source code changes you make to customize ROMS will be made in your Projects space, and need not be made to the downloaded code directly. We will discuss exceptions to this during the tutorial, and how source code modifications interact with svn.

• Set MY_ROMS_SRC to the location of the source code:

export MY_ROMS_SRC=/srv/ckpt/roms/shared/src

In practise, you will probably do something more like this:

export MY_ROMS_SRC=${MY_ROOT_DIR}/src

assuming this is the relative path in which you keep your source code on the various machines you work on.

Make sure that MY_CPP_FLAGS is not set. Sometimes this is set in the distributed build.bash exmaple. Comment out options with the # symbol like so:

#export MY_CPP_FLAGS="-DAVERAGES"

The UNSW Computer Lab machines are single core, so we need to tell build.bash not to assume MPI parallel compilation.

• Comment out the options for USE_MPI and USE_MPIF90

#export USE_MPI=on
#export USE_MPIF90=on

• If you were compiling in parallel you would leave the default entries in build.bash.

export USE_MPI=
export USE_MPIF90=

• We leave the compiler option as the default because this says use the ifort (Intel FORTRAN) compiler which is what we want on the UNSW machines.

export FORT=ifort

• In the interests of speed for this tutorial, we turn off compiler optimization by activating the debug option:

export USE_DEBUG=on

On the UNSW Lab machines compiling with optimization on will take over 15 minutes, but with optimization off (USE_DEBUG=on) it will be less than 60 seconds.

Save and close the build.bash file.

Copy the input and CPPDEFS options files

We need three more files in Projects/upwelling to configure and run ROMS. We copy the versions downloaded with svn because these are files you will work with locally when you experiment with changes to the test case example configuration.

• Copy files ocean_upwelling.in, varinfo.dat and upwelling.h into the Projects/upwelling directory you just created.

cd ~/Projects/upwelling
cp /srv/ckpt/roms/shared/src/ROMS/External/ocean_upwelling.in .
cp /srv/ckpt/roms/shared/src/ROMS/External/varinfo.dat .
cp /srv/ckpt/roms/shared/src/ROMS/Include/upwelling.h .

View the file upwelling.h. It contains all the C-Pre-Processor (CPP) options that the compiler interprets to activate certain source code options within ROMS.

View the file ocean_upwelling.in. It contains the inputs options that ROMS reads from standard input at run time to set options that need not be fixed at compile time.

View varinfo.dat. The file varinfo.dat contains descriptions of the names and attributes of input and output variables that ROMS reads and writes from netcdf files. For most applications you will not need to change the entries in this file. If you need to know the default units assumed for different variables, those are noted in this file. (Before we run ROMS, we will need to tell it where to find this file).

Now we are ready to compile ROMS by executing the build.bash script.

Compile ROMS

Before you run ROMS, you need to compile it to create an executable oceanS file (S for serial or single processor computer), or oceanM file (if using MPI on a parallel computer).

• Go to your upwelling project directory:

cd ~/Projects/upwelling

• Then type:

./build.bash

• If lots of stuff comes on the screen then compilation is proceeding, and make take some time.

• If the build process ends quickly with an error, then it is likely that build.bash does not point to the correct location for the upwelling.h file, the FORTRAN compiler, or some libraries. We describe common getting started errors and solutions in the next section.

• You may give the option -j to the build command to distribute the compilation to multiple processors if your host supports this, e.g.:

./build.bash -j 8

to compile on 8 processor at once.

If your build was successful it will not have reported any errors, and there will be an executable file in your Projects/upwelling directory called oceanG. The "G" in the file name indicates build.bash activated the USE_DEBUG option.

If USE_DEBUG were not selected, the executable would be oceanS, where the "S" indicates "serial" or "single-processor" because we deactivated MPI.

If you had activated MPI with the USE_MPI option the executable would be named oceanM.

(See also FAQ: My build finished with no errors, where is the ROMS executable?).

Common getting started compile error messages

Getting past the first few errors with compilation is often tricky. Carefully read any error messages you get for clues on what might be wrong with your configuration. Here are some common difficulties new users encounter getting started when first executing the build.bash command.

• Compilers/../ROMS/Include/cppdefs.h:709:22:
  error: /student/0/a0000020/Projects/upwelling/upwelling.h: No such file or directory
  This says the file upwelling.h is not where Build expects it to be, which is in MY_PROJECT_DIR. You set this to ~/Projects/upwelling.

• cp: cannot stat `/opt/intelsoft/netcdf/include/netcdf.mod': No such file or directory
  This says that netcdf is not where build.bash expects to find it. Locate where the netcdf include and lib directories with steps something like:
  cd /usr
  find . -name netcdf.mod -print
  ./local/netcdf-3.6.2/include/netcdf.mod
  ./local/netcdf/intel/3.6.3/include/netcdf.mod
  This tells us the most recent (3.6.3) netcdf is in /usr/local/netcdf/intel/3.6.3. Direct ROMS to this location by making two changes to build.bash. First, advise ROMS to read your changes to the default library path by uncommenting the option for USE_MY_LIBS.

export USE_MY_LIBS=on

Then specifiy the correct location for netcdf:

export NETCDF_INCDIR=/usr/local/netcdf/intel/3.6.3/include
export NETCDF_LIBDIR=/usr/local/netcdf/intel/3.6.3/lib

Be careful where you make this change. You need to make it for the ifort compiler option, and NOT for the USE_NETCDF4 option (we are using netcdf-3). If you've done this correctly, your compilation with build.bash should now succeed.

• error
  error:
  Note here further errors we encounter during the tutorial.

Run ROMS

You run ROMS by executing the oceanG (or oceanS) binary, giving it the ocean_upwelling.in file as UNIX standard input.

./oceanS < ocean_upwelling.in

ROMS standard output will be typed to the screen. To save it a file instead, enter, e.g.:

./oceanS < ocean_upwelling.in > my_upwelling.log

If you have compiled a parallel (MPI) executable, the syntax for running the mode is slightly but critically different. The ocean_upwelling.in file is no longer read from UNIX standard input (it has handled by all the MPI threads) so the "<" disappears from the command, and you need the correct syntax on your UNIX host for running an MPI process. It is probably something like:

mpirun -np 8 ./oceanM ocean_upwelling.in > my_upwelling.log

where the "-np 8" indicates use 8 processors and this number of tiles must have been set by

(See also FAQ: What do I have to do to runs ROMS?).

Common getting started run error messages

bash: oceanG: command not found

The working directory is not in your UNIX path. That's why we type "dot-slash" in front of the commands above.

Successful execution

Standard Output

When ROMS runs it will type a lot of information to UNIX standard output. This is the "logfile" you named following the ">", or your terminal if you did not redirect stdout.

STDOUT shows the following:

• UNIX process info, run time, run TITLE

Process Information:
Thread # 0 (pid= 4449) is active.
Model Input Parameters: ROMS/TOMS version 3.2
Monday - March 23, 2009 - 10:02:39 AM
-----------------------------------------------------------
Wind-Driven Upwelling/Downwelling over a Periodic Channel

• OS, compiler information, SVN version, and your MY_ROMS_SRC, MY_HEADER_DIR and ROMS_APPLICATION settings

Operating system : Linux
CPU/hardware  : i686
Compiler system  : ifort
Compiler command : /usr/local/intel/fc/10.1.021/bin/ifort
Compiler flags  : -heap-arrays -ip -O3 -pc80 -xW -free

SVN Root URL  : https://www.myroms.org/svn/src/trunk
SVN Revision  : 333
Local Root  : /srv/ckpt/roms/shared/src
Header Dir  : /student/0/a0000020/srv/Projects/upwelling
Header file  : upwelling.h
Analytical Dir: /student/0/a0000020/srv/Projects/upwelling

Resolution, Grid 01: 0041x0080x016, Parallel Threads: 1, Tiling: 001x001

Check that these are what you intended. In last line above

• "Grid 01" pertains to future ROMS developments with multiple nested/connected grids,
• 0041x0080x016 shows the grid size is 41 x 80 x 16 grid points in the K,J,I directions
• The Parallel/Tiling message shows you are using a single process and a single domain tile. When using MPI, this message will describe how many tiles you are using and the MPI processes assigned.

• Input parameters set in ocean_upwelling.in

Physical Parameters, Grid: 01
=============================

288 ntimes Number of timesteps for 3-D equations.
300.000 dt Timestep size (s) for 3-D equations.
30 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
...
Output Averages File: ocean_avg.nc
Output Diagnostics File: ocean_dia.nc

• Then some more about the tiling when running in parallel

• The C-PreProcessor (CPP) flags set in upwelling.h but AS MODIFIED by ROMS when interpreting and checking the selected CPP options.

Activated C-preprocessing Options:

UPWELLING Wind-Driven Upwelling/Downwelling over a Periodic Channel
ANA_BSFLUX Analytical kinematic bottom salinity flux.
ANA_BTFLUX Analytical kinematic bottom temperature flux.
ANA_GRID Analytical grid set-up.
ANA_INITIAL Analytical initial conditions.
...

You should check that the CPP options displayed here agree with what you intended. For example, if you inadvertently specify more than one horizontal advection scheme option, ROMS will have chosen only one and reported that option here.

• The preamble in STDOUT continues with information about the space and time discretization: grid spacing, grid volume, Courant number (time step stability) and stiffness (related to s-coordinate accuracy).

• Then the model starts time stepping:

NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 00000001 - 00000288)

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME

0 0 00:00:00 0.000000E+00 6.579497E+02 6.579497E+02 3.884376E+11
DEF_HIS - creating history file: ocean_his.nc
WRT_HIS - wrote history fields (Index=1,1) into time record = 000000
DEF_AVG - creating average file: ocean_avg.nc
DEF_DIAGS - creating diagnostics file: ocean_dia.nc
1 0 00:05:00 3.268255E-13 6.579497E+02 6.579497E+02 3.884376E+11
2 0 00:10:00 6.503587E-12 6.579497E+02 6.579497E+02 3.884376E+11
3 0 00:15:00 4.592307E-11 6.579497E+02 6.579497E+02 3.884376E+11

...

This output indicates several things:

• the run is programmed to run from time step 1 to 288
• the run starts at time 00:00:0
• netcdf output HISTORY, AVERAGES and DIAGNOSTICS files are created. Every time ROMS creates a new netcdf file, and writes to an existing file, it reports this to STDOUT
• output is written to the HISTORY file
• then global quantities related to the model KE, PE and domain volume are reported on each time step

In 99% of situations, getting started problems with model set-up and configuration can be diagnosed by carefully reading the STDOUT above. Things to look for are:

• misconfigured CPP options (what you got is not what you thought you asked for)
• parameter errors (e.g. you activated horizontal mixing but left the coefficient as zero)
• misnamed output files (that's why the files from your last run got overwritten)
• irrational choices of grid spacing or time step
• initial/boundary/forcing data being read from the wrong file, or not read at all (because you selected analytical conditions)

At the conclusion of the run, ROMS reports information about run time:

Elapsed CPU time (seconds):

Thread # 0 CPU: 108.079
Total: 108.079

Nonlinear model elapsed time profile:

Initialization ................................... 0.016 ( 0.0148 %)
Processing of input data ......................... 0.028 ( 0.0259 %)
Processing of output time averaged data .......... 4.312 ( 3.9899 %)
...

• about the number of output records written to each file

ROMS/TOMS - Output NetCDF summary for Grid 01:
number of time records written in HISTORY file = 00000005
...

• and the analytical files included

Analytical header files used:

ROMS/Functionals/ana_btflux.h
ROMS/Functionals/ana_grid.h
...

If you used a modified analytical file in your MY_HEADER_DIR it will be reported here and is another thing you should check for consistency with your intentions.

Netcdf file output

As reported above, ROMS created 4 output netcdf files when it ran. There are ocean_his.nc, ocean_avg.nc, ocean_dia.nc, and coean_rst.nc. These are, respectively:

• history records or 'snapshots' of the model state a selected time intervals
• averages of the model state over selected intervals (not necessarily the same intervals as the history)
• diagnostics of the model state, the precise contents of which are controlled by CPP options
• a restart file with everything ROMS needs to restart a application. This is useful if your job crashes at some point and you want to recommence from a previous state without starting over. Typically the restart file is set to keep just 2 time records by continually over-writing the oldest as the run proceeds. This behaviour is controlled in ocean_upwelling.in. Also, when ROMS "blows up" it dumps the ocean state to a 3rd record in the restart file.

You can browse the contents of netcdf files at the UNIX command line with the command ncdump, e.g.

ncdump -h ocean_his.nc | more

Note the use of the "-h" option. This restricts the output from ncdump to be just the header information, or metadata, in the netcdf file. Without the "-h" option you will get the entire contents of the file converted to ascii.

Things to notice when you ncdump the ocean_his.nc file are that it contains all the input parameters (time step, mixing coefficients, s-coordinate parameters, etc.) from ocean.in, the model grid coordinates (x, y, lon, lat, depth, Coriolis parameter, etc.) which may have been computed by the ANA_GRID option or read from an input grid netcdf file, in addition to the actual model output (ocean_time, zeta, u, v, temp, salt).

There are netcdf global attributes that echo much of the information typed to STDOUT. This includes compiler, svn version, and project directory information, and all the CPP options. This is a valuable source of information when returning to a project and trying to figure out what you did! The global attributes metadata show precisely which options were activated when creating the output in this netcdf file.

A Realistic model example: LaTTE

This section of the tutorial assumes you have successfully compiled and run the UPWELLING example above. Key concepts you should be comfortable with to proceed are:

• you need a new directory where you will keep the files specific to the new application
• customize build.bash for the new application (copy the build.bash from upwelling because it has all the correct compiler and library settings)
• set MY_PROJECT_DIR in build.bash to point to the directory for the new application
• set ROMS_APPLICATION in build.bash to the correct name for the new application
• if you wish to customize any of the ana_*.h files, copy just the ones you need into the new project directory
• you don't need to make a copy the source code a new application

With these concepts in mind, we proceed by configuring ROMS to run a realistic coastal ocean application that includes open boundaries on 3 sides, open boundary tides and climatology velocity and tracer (temperature and salinity) conditions, surface meteorological forcing, and initial conditions, all provided by input netcdf files. The example is called LaTTE_C because it simulates ocean conditions during the Lagrangian Transport and Transformation Experiment conducted on the New Jersey inner shelf in the Spring of 2006. The '_C' denotes a coarse resolution configuration suitable for this training exercise.

Create a latte_c project directory

We have placed the CPP options file latte_c.h, standand input ocean_lattec.in, and a modified varinfo.dat in /srv/ckpt/roms/shared/latte_c/Forward. Make a new Project directory for this new application and copy these 3 files. DO NOT copy all the netcdf files from /srv/ckpt/roms/shared/latte_c/in .

Edit build.bash

Set the correct entries for ROMS_APPLICATION and MY_PROJECT_DIR

Edit ocean_lattec.in

Change the entries for GRDNAME etc so that ROMS finds the shared input files.

Change the path for HISNAME and other outputs so that your output is written to your own work space.

Before we run latte_c

Browse input netcdf files

Use ncdump -h