WikiROMS - User contributions [en]

4DVar Tutorial Introduction

2025-10-17T16:26:50Z

Robertson:

<div class="title">4-Dimensional Variational (4D-Var) Data Assimilation Tutorial</div>__NOTOC__

This tutorial shows several examples of ROMS 4D-Var data assimilation algorithms in the California Current System (CCS), 1/3 degree resolution ([[Options#WC13|WC13]]).


<div style="float: left;margin: 0 20px 0 0;">{{4DVar Tut TOC}}</div>__TOC__

<div style="clear: both"></div>

<section begin=setup />
==Model Set-up==
The [[Options#WC13|WC13]] model domain is shown in Fig. 1 and has open boundaries along the northern, western, and southern edges of the model domain.

[[Image:hraw_coarse_transect.png|500px|thumb|center|<center>'''Fig. 1:''' Model Bathymetry with 37°N Transect and Target Area</center>]]
<div style="clear: both"></div>

In the tutorial, you will perform a 4D-Var data assimilation cycle that spans the period 3-6 January, 2004. The 4D-Var control vector ''δ'''''z''' is comprised of increments to the initial conditions, ''δ'''''x'''(''t<sub>0</sub>''), surface forcing, δ'''f'''(''t''), and open boundary conditions, δ'''b'''(''t''). The ''prior'' initial conditions, '''x<sub>b</sub>'''(''t<sub>0</sub>''), are taken from the sequence of 4D-Var experiments described by [[Bibliography#MooreAM_2011b|Moore ''et al.'' (2011b)]] in which data were assimilated every 7 days during the period July 2002- December 2004. The ''prior'' surface forcing, '''f<sub>b</sub>'''(''t''), takes the form of surface wind stress, heat flux, and a freshwater flux computed using the ROMS bulk flux formulation, and using near surface air data from COAMPS ([[Bibliography#DoyleJD_2009|Doyle ''et al.'', 2009]]). Clamped open boundary conditions are imposed on (''u'',''v'') and tracers, and the ''prior'' boundary conditions, '''b<sub>b</sub>'''(''t''), are taken from the global ECCO product ([[Bibliography#WunschC_2007|Wunsch and Heimbach, 2007]]). The free-surface height and vertically integrated velocity components are subject to the usual Chapman and Flather radiation conditions at the open boundaries. The ''prior'' surface forcing and open boundary conditions are provided daily and linearly interpolated in time. Similarly, the increments ''δ'''''f'''(''t'') and ''δ'''''b'''(''t'') are also computed daily and linearly interpolated in time.

The observations assimilated into the model are satellite SST, satellite SSH in the form of a gridded product from Aviso, and hydrographic observations of temperature and salinity collected from Argo floats and during the GLOBEC/LTOP and CalCOFI cruises off the coast of Oregon and southern California, respectively. The observation locations are illustrated in Fig. 2.

{|align="center"
|+style="caption-side:bottom;"|'''Figure 2:''' WC13 Observations
|-
|[[Image:aviso_ssh_2019.png|thumb|400px|<center>'''a)''' Aviso SSH</center>]]
|[[Image:blended_sst_2019.png|thumb|400px|<center>'''b)''' Blended SST</center>]]
|-
|[[Image:in_situ_T_2019.png|thumb|400px|<center>'''c)''' In Situ Temperature</center>]]
|[[Image:in_situ_S_2019.png|thumb|400px|<center>'''d)''' In Situ Salinity</center>]]
|-
|}<section end=setup />

==Download Lectures and Exercises==

The workshop Agenda can be found [https://www.myroms.org/Workshops/4DVAR2019/Agenda.pdf here]. Links to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop] lectures are provided below. These lectures are meant to supplement this 4D-Var tutorial.

*'''Lecture 1:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_1_2019.pdf PDF]</span>   <span class="twilightBlue">4D-Var: Some Basics</span>
*'''Lecture 2:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_2_2019.pdf PDF]</span>   <span class="twilightBlue">The Mechanics of 4D-Var</span>
*'''Lecture 3:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_3_2019.pdf PDF]</span>   <span class="twilightBlue">Dual 4D-Var</span>
*'''Lecture 4:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_4_2019.pdf PDF]</span>   <span class="twilightBlue">Observing System Simulation Experiments (OSSEs)</span>
*'''Lecture 5:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_5_2019.pdf PDF]</span>   <span class="twilightBlue">Observation Impact & Observation Sensitivity</span>
*'''Lecture 6:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_6_2019.pdf PDF]</span>   <span class="twilightBlue">The MARACOOS analysis-forecast system</span>
*'''Lecture 7:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_7_2019.pdf PDF]</span>   <span class="twilightBlue">Array Modes</span>
*'''Lecture 8:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">The West Coast Ocean Forecasting System (WCOFS)</span>
*'''Lecture 9:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">The CeNCOOS and PacIOOS analysis-forecast systems</span>

Below are the tutorials pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Tutorial 01:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_01_2019.pdf PDF]</span>   <span class="twilightBlue">Explanation of cpp options, ocean.in, s4dvar.in (I4DVAR)</span>
*'''Tutorial 02:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_02_2019.pdf PDF]</span>   <span class="twilightBlue">Multiple outer loops</span>
*'''Tutorial 03:'''   <span style="font-size: 75%"></span>            <span class="twilightBlue">Discussion of exercises 1 and 2</span>
*'''Tutorial 04:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_04_2019.pdf PDF]</span>   <span class="twilightBlue">Calculation of prior error standard deviations</span>
*'''Tutorial 05:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_05_2019.pdf PDF]</span>   <span class="twilightBlue">Explanation of cpp options, ocean.in, s4dvar.in (RBL4D-Var)</span>
*'''Tutorial 06:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_06_2019.pdf PDF]</span>   <span class="twilightBlue">Semi-variograms</span>
*'''Tutorial 07:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_07_2019.pdf PDF]</span>   <span class="twilightBlue">ERDDAP data server and management tools</span>
*'''Tutorial 08:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">Discussion of exercises 3 & 4</span>
*'''Tutorial 09:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_09_2019.pdf PDF]</span>   <span class="twilightBlue">Computing normalization coefficients for covariance models</span>
*'''Tutorial 10:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_10_2019.pdf PDF_Arango] | [https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_10a_2019.pdf PDF_Wilkin]</span>   <span class="twilightBlue">Building your observation files</span>
*'''Tutorial 11:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_11_2019.pdf PDF]</span>   <span class="twilightBlue">Observation Impact & Observation Sensitivity</span>
*'''Tutorial 12:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">Discussion of exercise 5</span>
*'''Tutorial 13:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_13_2019.pdf PDF]</span>   <span class="twilightBlue">Using ERDDAP to view observation impact information</span>
*'''Tutorial 14:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">Discussion of exercise 6 & 7</span>
*'''Tutorial 15:'''   <span style="font-size: 75%"></span>             <span class="twilightBlue">Putting it all together</span>

Below are the exercises pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Exercise 1:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_1_2019.pdf PDF]</span>   <span class="twilightBlue">Incremental, Strong Constraint 4D-Var</span>
*'''Exercise 2:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_2_2019.pdf PDF]</span>   <span class="twilightBlue">I4D-Var with Multiple Outer-loops</span>
*'''Exercise 3:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_3_2019.pdf PDF]</span>   <span class="twilightBlue">Dual Formulation 4D-Var - RBL4D-Var</span>
*'''Exercise 4:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_4_2019.pdf PDF]</span>   <span class="twilightBlue">Weak Constraint Dual Formulation 4D-Var</span>
*'''Exercise 5:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_5_2019.pdf PDF]</span>   <span class="twilightBlue">Analysis Cycle Observation Impacts</span>
*'''Exercise 6:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_6_2019.pdf PDF]</span>   <span class="twilightBlue">Analysis Cycle Observation Sensitivity</span>
*'''Exercise 7:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_7_2019.pdf PDF]</span>   <span class="twilightBlue">Reduced-Rank Array Modes</span>
*'''Exercise 8:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_8_2019.pdf PDF]</span>   <span class="twilightBlue">Forecast Cycle Observation Impacts</span>
*'''Exercise 9:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_9_2019.pdf PDF]</span>   <span class="twilightBlue">Forecast Cycle Observation Sensitivities</span>

Below are the homeworks pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Homework 1:'''     <span class="twilightBlue">Building the standard deviation file for user model configuration</span>
*'''Homework 2:'''     <span class="twilightBlue">Building the file of normalization coefficients for the prior error covariance matrix for user model configuration</span>
*'''Homework 3:'''     <span class="twilightBlue">Build observation file for user model configuration</span>

==Download Tutorial Files==
All data files and scripts necessary to run the 4D-Var tests discussed in this tutorial are available from the ROMS [https://github.com/myroms GitHub] repository. To download, execute the following command:
<div class="box"><span class="red"> git clone https://github.com/myroms/roms_test.git</div>

==Tutorial Directory Structure==
When the checkout is complete, you will have the following directory structure under your <span class="twilightBlue">my_test</span> directory.

<span class="twilightBlue">/WC13 Main California Current System 4D-Var applications
/ARRAY_MODES Stabilized representer matrix array modes and clipping
/Data Input data directory
/Functionals Analytical expressions header files
/I4DVAR Primal form of incremental, strong constraint 4D-Var, I4D-Var
/Normalization 4D-Var error covariance normalization coefficients
/plotting 4D-Var plotting scripts (Matlab and ROMS plotting package)
/RBL4DVAR Dual form of 4D-Var, Restricted B-preconditioned Lanczos Analysis System, RBL4D-Var
/RBL4DVAR_analysis_impact RBL4D-Var analysis observation impact
/RBL4DVAR_analysis_sensitivity RBL4D-Var analysis observation sensitivity (adjoint of RBL4D-Var)
/RBL4DVAR_forecast_impact RBL4D-Var forecast observation impact
/RBL4DVAR_forecast_sensitivity RBL4D-Var forecast observation sensitivity (adjoint of RBL4D-Var)
</span>
Mostly all the directories have a <span class="twilightBlue">Readme</span> file with detailed instructions for configuring, compiling, running, and plotting the results.

==References==

The technical description of the algorithms and application used in this tutorial are described in Moore ''et al.'' ([[Bibliography#MooreAM_2011a|2011a]], [[Bibliography#MooreAM_2011b|b]], [[Bibliography#MooreAM_2011c|c]]).

4DVar Tutorial Introduction

2025-10-17T16:19:46Z

Robertson:

<div class="title">4-Dimensional Variational (4D-Var) Data Assimilation Tutorial</div>__NOTOC__

This tutorial shows several examples of ROMS 4D-Var data assimilation algorithms in the California Current System (CCS), 1/3 degree resolution ([[Options#WC13|WC13]]).


<div style="float: left;margin: 0 20px 0 0;">{{4DVar Tut TOC}}</div>__TOC__

<div style="clear: both"></div>

<section begin=setup />
==Model Set-up==
The [[Options#WC13|WC13]] model domain is shown in Fig. 1 and has open boundaries along the northern, western, and southern edges of the model domain.

[[Image:hraw_coarse_transect.png|500px|thumb|center|<center>'''Fig. 1:''' Model Bathymetry with 37°N Transect and Target Area</center>]]
<div style="clear: both"></div>

In the tutorial, you will perform a 4D-Var data assimilation cycle that spans the period 3-6 January, 2004. The 4D-Var control vector ''δ'''''z''' is comprised of increments to the initial conditions, ''δ'''''x'''(''t<sub>0</sub>''), surface forcing, δ'''f'''(''t''), and open boundary conditions, δ'''b'''(''t''). The ''prior'' initial conditions, '''x<sub>b</sub>'''(''t<sub>0</sub>''), are taken from the sequence of 4D-Var experiments described by [[Bibliography#MooreAM_2011b|Moore ''et al.'' (2011b)]] in which data were assimilated every 7 days during the period July 2002- December 2004. The ''prior'' surface forcing, '''f<sub>b</sub>'''(''t''), takes the form of surface wind stress, heat flux, and a freshwater flux computed using the ROMS bulk flux formulation, and using near surface air data from COAMPS ([[Bibliography#DoyleJD_2009|Doyle ''et al.'', 2009]]). Clamped open boundary conditions are imposed on (''u'',''v'') and tracers, and the ''prior'' boundary conditions, '''b<sub>b</sub>'''(''t''), are taken from the global ECCO product ([[Bibliography#WunschC_2007|Wunsch and Heimbach, 2007]]). The free-surface height and vertically integrated velocity components are subject to the usual Chapman and Flather radiation conditions at the open boundaries. The ''prior'' surface forcing and open boundary conditions are provided daily and linearly interpolated in time. Similarly, the increments ''δ'''''f'''(''t'') and ''δ'''''b'''(''t'') are also computed daily and linearly interpolated in time.

The observations assimilated into the model are satellite SST, satellite SSH in the form of a gridded product from Aviso, and hydrographic observations of temperature and salinity collected from Argo floats and during the GLOBEC/LTOP and CalCOFI cruises off the coast of Oregon and southern California, respectively. The observation locations are illustrated in Fig. 2.

{|align="center"
|+style="caption-side:bottom;"|'''Figure 2:''' WC13 Observations
|-
|[[Image:aviso_ssh_2019.png|thumb|400px|<center>'''a)''' Aviso SSH</center>]]
|[[Image:blended_sst_2019.png|thumb|400px|<center>'''b)''' Blended SST</center>]]
|-
|[[Image:in_situ_T_2019.png|thumb|400px|<center>'''c)''' In Situ Temperature</center>]]
|[[Image:in_situ_S_2019.png|thumb|400px|<center>'''d)''' In Situ Salinity</center>]]
|-
|}<section end=setup />

==Download Lectures and Exercises==

The workshop Agenda can be found [https://www.myroms.org/Workshops/4DVAR2019/Agenda.pdf here]. Links to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop] lectures are provided below. These lectures are meant to supplement this 4D-Var tutorial.

*'''Lecture 1:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_1_2019.pdf PDF]</span>   <span class="twilightBlue">4D-Var: Some Basics</span>
*'''Lecture 2:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_2_2019.pdf PDF]</span>   <span class="twilightBlue">The Mechanics of 4D-Var</span>
*'''Lecture 3:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_3_2019.pdf PDF]</span>   <span class="twilightBlue">Dual 4D-Var</span>
*'''Lecture 4:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_4_2019.pdf PDF]</span>   <span class="twilightBlue">Observing System Simulation Experiments (OSSEs)</span>
*'''Lecture 5:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_5_2019.pdf PDF]</span>   <span class="twilightBlue">Observation Impact & Observation Sensitivity</span>
*'''Lecture 6:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_6_2019.pdf PDF]</span>       <span class="twilightBlue">The MARACOOS analysis-forecast system</span>
*'''Lecture 7:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Lectures/Lecture_7_2019.pdf PDF]</span>   <span class="twilightBlue">Array Modes</span>
*'''Lecture 8:'''   <span style="font-size: 75%">PDF</span>       <span class="twilightBlue">The West Coast Ocean Forecasting System (WCOFS)</span>
*'''Lecture 9:'''   <span style="font-size: 75%">PDF</span>       <span class="twilightBlue">The CeNCOOS and PacIOOS analysis-forecast systems</span>

Below are the tutorials pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Tutorial 01:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_01_2019.pdf PDF]</span>   <span class="twilightBlue">Explanation of cpp options, ocean.in, s4dvar.in (I4DVAR)</span>
*'''Tutorial 02:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_02_2019.pdf PDF]</span>   <span class="twilightBlue">Multiple outer loops</span>
*'''Tutorial 03:'''   <span style="font-size: 75%"></span>   <span class="twilightBlue">Discussion of exercises 1 and 2</span>
*'''Tutorial 04:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_04_2019.pdf PDF]</span>   <span class="twilightBlue">Calculation of prior error standard deviations</span>
*'''Tutorial 05:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_05_2019.pdf PDF]</span>   <span class="twilightBlue">Explanation of cpp options, ocean.in, s4dvar.in (RBL4D-Var)</span>
*'''Tutorial 06:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_06_2019.pdf PDF]</span>   <span class="twilightBlue">Semi-variograms</span>
*'''Tutorial 07:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_07_2019.pdf PDF]</span>   <span class="twilightBlue">ERDDAP data server and management tools</span>
*'''Tutorial 08:'''   <span style="font-size: 75%"></span>   <span class="twilightBlue">Discussion of exercises 3 & 4</span>
*'''Tutorial 09:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_09_2019.pdf PDF]</span>   <span class="twilightBlue">Computing normalization coefficients for covariance models</span>
*'''Tutorial 10:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_10_2019.pdf PDF_Arango] | [https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_10a_2019.pdf PDF_Wilkin]</span>   <span class="twilightBlue">Building your observation files</span>
*'''Tutorial 11:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_11_2019.pdf PDF]</span>   <span class="twilightBlue">Observation Impact & Observation Sensitivity</span>
*'''Tutorial 12:'''   <span style="font-size: 75%"></span>   <span class="twilightBlue">Discussion of exercise 5</span>
*'''Tutorial 13:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Tutorials/Tutorial_13_2019.pdf PDF]</span>   <span class="twilightBlue">Using ERDDAP to view observation impact information</span>
*'''Tutorial 14:'''   <span style="font-size: 75%"></span>   <span class="twilightBlue">Discussion of exercise 6 & 7</span>
*'''Tutorial 15:'''   <span style="font-size: 75%"></span>   <span class="twilightBlue">Putting it all together</span>

Below are the exercises pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Exercise 1:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_1_2019.pdf PDF]</span>   <span class="twilightBlue">Incremental, Strong Constraint 4D-Var</span>
*'''Exercise 2:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_2_2019.pdf PDF]</span>   <span class="twilightBlue">I4D-Var with Multiple Outer-loops</span>
*'''Exercise 3:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_3_2019.pdf PDF]</span>   <span class="twilightBlue">Dual Formulation 4D-Var - RBL4D-Var</span>
*'''Exercise 4:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_4_2019.pdf PDF]</span>   <span class="twilightBlue">Weak Constraint Dual Formulation 4D-Var</span>
*'''Exercise 5:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_5_2019.pdf PDF]</span>   <span class="twilightBlue">Analysis Cycle Observation Impacts</span>
*'''Exercise 6:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_6_2019.pdf PDF]</span>   <span class="twilightBlue">Analysis Cycle Observation Sensitivity</span>
*'''Exercise 7:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_7_2019.pdf PDF]</span>   <span class="twilightBlue">Reduced-Rank Array Modes</span>
*'''Exercise 8:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_8_2019.pdf PDF]</span>   <span class="twilightBlue">Forecast Cycle Observation Impacts</span>
*'''Exercise 9:'''   <span style="font-size: 75%">[https://www.myroms.org/Workshops/4DVAR2019/Exercises/EXERCISE_9_2019.pdf PDF]</span>   <span class="twilightBlue">Forecast Cycle Observation Sensitivities</span>

Below are the homeworks pertinent to the [https://www.myroms.org/index.php?page=4DVAR_2019_info 2019 ROMS 4D-Var workshop]:

*'''Homework 1:'''     <span class="twilightBlue">Building the standard deviation file for user model configuration</span>
*'''Homework 2:'''     <span class="twilightBlue">Building the file of normalization coefficients for the prior error covariance matrix for user model configuration</span>
*'''Homework 3:'''     <span class="twilightBlue">Build observation file for user model configuration</span>

==Download Tutorial Files==
All data files and scripts necessary to run the 4D-Var tests discussed in this tutorial are available from the ROMS [https://github.com/myroms GitHub] repository. To download, execute the following command:
<div class="box"><span class="red"> git clone https://github.com/myroms/roms_test.git</div>

==Tutorial Directory Structure==
When the checkout is complete, you will have the following directory structure under your <span class="twilightBlue">my_test</span> directory.

<span class="twilightBlue">/WC13 Main California Current System 4D-Var applications
/ARRAY_MODES Stabilized representer matrix array modes and clipping
/Data Input data directory
/Functionals Analytical expressions header files
/I4DVAR Primal form of incremental, strong constraint 4D-Var, I4D-Var
/Normalization 4D-Var error covariance normalization coefficients
/plotting 4D-Var plotting scripts (Matlab and ROMS plotting package)
/RBL4DVAR Dual form of 4D-Var, Restricted B-preconditioned Lanczos Analysis System, RBL4D-Var
/RBL4DVAR_analysis_impact RBL4D-Var analysis observation impact
/RBL4DVAR_analysis_sensitivity RBL4D-Var analysis observation sensitivity (adjoint of RBL4D-Var)
/RBL4DVAR_forecast_impact RBL4D-Var forecast observation impact
/RBL4DVAR_forecast_sensitivity RBL4D-Var forecast observation sensitivity (adjoint of RBL4D-Var)
</span>
Mostly all the directories have a <span class="twilightBlue">Readme</span> file with detailed instructions for configuring, compiling, running, and plotting the results.

==References==

The technical description of the algorithms and application used in this tutorial are described in Moore ''et al.'' ([[Bibliography#MooreAM_2011a|2011a]], [[Bibliography#MooreAM_2011b|b]], [[Bibliography#MooreAM_2011c|c]]).

External Libraries

2025-06-18T18:42:51Z

Robertson:

<div class="title">External Libraries</div>

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

==ARPACK (ARnoldi PACKage)==
Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method ('''IRAM''') for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the '''BLAS''' and '''LAPACK''' libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

To obtain '''ARPACK''', execute the following:
<div class="box">> git clone https://github.com/myroms/roms_libs.git<br />> cd roms_libs/ARPACK</div>
To compile, customize '''ARmake.inc''' for the desired compiler and its flags for the serial ('''FC''' and '''FFLAGS''') and parallel ('''PFC''' and '''PFFLAGS''') versions of the library. Once configured type:<div class="box">> make lib plib</div> to build '''libarpack.a''' and '''libparpack.a'''. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the '''[[my_build_paths]]''' script.

{{note}} '''Note:''' Prior to SVN revision 1080 (July 23, 2021), '''ARPACK''' was distributed in the ROMS '''Lib/ARPACK''' subdirectory. For more information about why this was changed see TRAC ticket [https://www.myroms.org/projects/src/ticket/891 #891]

==MCT (Model Coupling Toolkit)==

Open-source library distributed as a set of Fortran-90 modules for constructing a coupled model system from Earth System Model (ESM) components ([http://www-unix.mcs.anl.gov/mct Larson et al., 2004]). Each component model has its own grid and runs on its own set of processors. The MCT library provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred.

The MCT library is used to couple ROMS atmosphere models (like WRF) and wave models (like SWAN and WW3). For example, SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

MCT is available via github:

<div class="box">> git clone https://github.com/MCSclimate/MCT<br />> cd MCT<br />> ./configure<br />> make</div>

==NetCDF (Network Common Data Form)==

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides '''API''' in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses '''NetCDF''' for all its input and output data management. Its '''NetCDF3''' or '''NetCDF4''' type files can be processed using the standard library developed and distributed [http://doi.org/10.5065/D6H70CW6 UCAR/Unidata], the Parallel-IO ('''PIO''') developed at ANL/NCAR ([https://doi.org/10.1177/1094342011428143 Dennis ''et al''., 2012]; [https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE Hartnett and Edwards, 2021]). We have migrated to the Fortran-90 '''API''' interface, which consists of a library and one or two module files. See the [[makefile]] and [[build_roms| build script]] for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

===Building NetCDF-4/HDF5===
NetCDF-4 uses HDF5 for it's underlying data format so you first need to build HDF5. We recommend 1.10.6 because 1.10.7 causes errors in the NetCDF-C test suite. Here we will describe how to build parallel versions for use on an HPC cluster. We will note what changes are necessary if you are building for a system that doesn't have parallel I/O.

====Building HDF5====
* First, download '''HDF5''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://support.hdfgroup.org/releases/hdf5/v1_14/v1_14_3/downloads/hdf5-1.14.3.tar.gz</div>or<div class="box">> git clone https://github.com/HDFGroup/hdf5<br />> cd hdf5<br />> git checkout tags/hdf5-1_14_3<br />> autoconf -i</div>

* Configure '''HDF5''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort \<br /> RUNPARALLEL="srun --nodes=1 --ntasks=6 --cpus-per-task=1 --time=00:13:00 --export=ALL " \<br /> --prefix=/path/to/parallel/hdf5/1.14.3 --enable-static-exec --enable-shared=no \<br /> --enable-parallel --disable-silent-rules --enable-fortran</div>{{note}} '''Note:''' ''RUNPARALLEL'' is only needed if your system uses a scheduler to run MPI jobs. The example here is for a SLURM system but you will need to adjust this for your cluster.<br /><br />{{note}} '''Note:''' To compile '''serial''', remove ''RUNPARALLEL=...'' and ''--enable-parallel'' and change the CC, CXX, and FC to the corresponding serial compilers (e.g. gcc, g++, and gfortran).<br /><br />{{note}} '''Note:''' If you would like '''shared libraries''' to be built, also remove ''--enable-shared=no''.<br /><br />

* Build, test, and install '''HDF5''':<div class="box">> make -j 5<br />> make all_tests<br />> make install</div>

====Building NetCDF-C====
The C and Fortran APIs for NetCDF are in separate packages and the C version must be built first.

* Download '''NetCDF-C''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://downloads.unidata.ucar.edu/netcdf-c/4.9.2/netcdf-c-4.9.2.tar.gz</div>or<div class="box">> git clone https://github.com/Unidata/netcdf-c<br />> cd netcdf-c<br />> git checkout tags/v4.9.2<br />> autoconf -i</div>

* Configure '''NetCDF-C''':<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 CPPFLAGS="-I/path/to/parallel/hdf5/1.14.3/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.14.3/lib" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf-c/4.9.2 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>{{note}} '''Note:''' To compile '''serial''', remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

====Building NetCDF-Fortran====
Now we can build the Fortran API.

* Download '''NetCDF-Fortran''' source code as a gzipped tarball or via git clone:<div class="box">> wget http://downloads.unidata.ucar.edu/netcdf-fortran/4.6.1/netcdf-fortran-4.6.1.tar.gz</div>or<div class="box">git clone https://github.com/Unidata/netcdf-fortran<br />> cd netcdf-fortan<br />> git checkout tags/v4.6.1<br />> autoconf -i</div>

* Configure '''NetCDF-Fortran''':<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 FFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/parallel/hdf5/1.14.3/include -I/path/to/parallel/netcdf-c/4.9.2/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.14.3/lib -L/path/to/parallel/netcdf-c/4.9.2/lib" \<br /> LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl" \<br /> --with-mpiexec=/path/to/job/submission/script \<br /> --prefix=/path/to/parallel/netcdf-fortran/4.6.1 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' should point to the submision script you created while building '''NetCDF-C''' and is only needed for running test on clusters that use scheduling systems like SLURM.<br /><br />{{note}} '''Note:''' To compile serial remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* If you are running on clusters that use scheduling systems like SLURM, you will need to edit 4 testing scripts replacing all occurrences of '''mpiexec''' with the full path to the submission script you created for '''NetCDF-C''':<div class="box">> vi examples/F90/run_f90_par_examples.sh<br />> vi nf03_test4/run_f90_par_test.sh<br />> vi nf_test4/run_f77_par_test_03.sh<br />> vi nf_test4/run_f77_par_test.sh</div>

* Build, test, and install '''NetCDF-fortran''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

==PIO (Parallel I/O)==

Open-source parallel library for reading and writing distributed arrays to several scientific data formats like '''pNetCDF''', '''NetCDF3''', and '''NetCDF4/HDF5'''. '''PIO''' provides the flexibility to regulate the number of I/O tasks through data rearrangement between computational and I/O processes to improve performance and memory usage. It allows both '''synchronous''' and '''asynchronous''' I/O. In '''synchronous mode''', a subset of processes performs both I/O and computations (MPI intra-communications). Alternatively, in '''asynchronous mode''', the I/O is carried out by a set of disjointed and dedicated processes (MPI inter-communications).

The '''PIO''' library is intended for ROMS Message Passing Interface (MPI) applications running on a large number of processes in an HPC computer with a '''Parallel File System''' (Lustre, GPFS, and so on). It uses the '''MPI-IO''' interface to facilitate the partitioning of data across computational or dedicated I/O processes. [https://github.com/NCAR/ParallelIO PIO] will work with ROMS [https://www.myroms.org/projects/src/ticket/884 svn revision 1064] (May 10, 2021) or newer. We recommend you read the manuals for all the build options, but below we show example build processes for '''PIO'''.

'''PIO''' can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for '''PIO''', PnetCDF is optional. {{note}}However, the ROMS build system assumes that '''PIO''' is built with PnetCDF support and will need customization if '''PIO''' is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

'''PIO''' recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the '''PIO''' library but ROMS Parallel I/O is not currently implemented this way. In order to build '''PIO''' with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The '''PIO''' library can be built with the classic Autotools or Cmake. Both are shown below.

===Building PIO With Autotools===

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/archive/refs/tags/pio2_6_2.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_6_2<br />> autoconf -i</div>

*If you downloaded the gzipped tarball, you should run ''make clean'' because some Fortran .mod files may be included and need to be removed:<div class="box">> make clean</div>

*Configure '''PIO''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/netcdf-c/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \<br /> LDFLAGS="-L/path/to/netcdf-c/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \<br /> LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz -lbz2 -lzstd -lxml2" \<br /> --with-mpiexec=/path/to/cluster/submission/script \<br /> --prefix=/pio/install/path --enable-fortran --disable-timing \<br /> --disable-silent-rules --disable-shared --enable-netcdf-integration</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install '''PIO''':<div class="box">> make -j 5<br />> make check<br />> make install</div>{{warning}} '''Warning:''' If you are using the Intel compilers, you will probably need to remove "'''test_darray_async_many'''" from line 50 of '''test/cunit/run_tests.sh''' for ''make check'' to complete without errors. Details [https://github.com/NCAR/ParallelIO/issues/1867 here].

===Building PIO With Cmake===

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_5_4</div>

*Configure '''PIO''':<div class="box">> cd ..<br />> mkdir build_pio<br />> cd build_pio<br />> CC=mpicc CXX=mpicxx FC=mpifort cmake \<br /> -DNetCDF_PATH=/path/to/netcdf \<br /> -DPnetCDF_PATH=/path/to/pnetcdf \<br /> -DMPIEXEC=/path/submission/scipt \<br /> -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \<br /> -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \<br /> -DCMAKE_INSTALL_PREFIX=/pio/install/path ../ParallelIO</div>{{note}} Note that ''-DMPIEXEC'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Compile, test, and install:<div class="box">> make -j 6<br />> make -j 6 tests<br />> ctests<br />> make install</div>

==ESMF (Earth System Modeling Framework)==
High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model ('''ESM''') components. It includes the '''NUOPC''' (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The '''NUOPC''' ''cap'' file is a Fortran module layer that sits on top of each '''ESM''' component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the '''NUOPC''' layer includes the native CoupleSets for '''ESM''' applications with nested grids. Various types of connections are possible, like '''nest-to-nest''', '''fine-to-coarse''', or '''coarse-to-fine'''. The user decides which nested grids to connect.

*Download from github:<div class="box">> git clone https://github.com/esmf-org/esmf.git esmf<br />> cd esmf<br />> git checkout tags/ESMF_8_6_0</div>

*Configure with environment variables:<div class="box">> export ESMF_DIR=/path/to/esmf/source<br />> export ESMF_COMM=openmpi<br />> export ESMF_COMPILER=intel<br />> export ESMF_BOPT=O<br />> export ESMF_OPTLEVEL=3<br />> export ESMF_ABI=64<br />> export ESMF_INSTALL_PREFIX=/ESMF/install/dir<br />> export ESMF_NETCDF="/path/to/netcdf/bin/nc-config"<br />> export ESMF_PNETCDF="/path/to/pnetcdf/bin/pnetcdf-config"<br />> export ESMF_PIO="external""<br />> export ESMF_PIO_LIBPATH="/path/to/pio/lib""<br />> export ESMF_PIO_INCLUDE="/path/to/pio/include" <br />> export ESMF_MPIRUN=/path/to/cluster/submission/script</div>{{note}} Note that ''ESMF_MPIRUN'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-np''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -np N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-np" ]<br />then<br /> echo "Usage: mpirun -np # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install:<div class="box">> make -j 10<br />> make all_tests<br />> make install</div>

build roms

2025-05-05T18:37:42Z

Robertson:

<div class="title">GNU Make Build Script - <span class="red">build_roms.csh, build_roms.sh</span></div>

As mentioned in [[makefile]], you need to provide settings for some user-defined choices before you can compile ROMS. If you have more than one application (or more than one compiler), you may get tired of editing the [[makefile]]. One option is to have a makefile for each configuration. The recommended solution, however, is to use the ROMS build script.

{{note}} '''Note:''' in SVN revision 1030 (April 30, 2020) the ROMS build scripts (and other shell scripts) were renamed to follow accepted naming conventions; <span class="red">*.sh</span> scripts were changed to <span class="red">*.csh</span> and <span class="red">*.bash</span> where changed to <span class="red">*.sh</span>.

__TOC__

==Introduction==
There are two of these build scripts in the '''ROMS/Bin''' directory: '''build_roms.csh''' and '''build_roms.sh'''. The build scripts use environment variables to provide values for the user-defined [[make]] variables, overwriting those found in the ROMS '''makefile'''. Just as in the multiple '''makefile''' option, you will need as many copies of the build script as you have applications. However, the scope of these variables is local to the build script, allowing you to compile different applications at the same time from the same sources as long as each '''$([[#SCRATCH_DIR|SCRATCH_DIR]])''' is unique.

:<div class="box">'''Usage:'''<br /><br /><span class="red"> ./build_roms.csh [options]</span><br /><br />'''Options:'''<br /><br /><span class="red"> -j [N]</span> Compile in parallel using <span class="red">N</span> CPUs<br /> omit argument for all available CPUs.<br /><br /><span class="red"> -p macro</span> Prints any [[makefile]] variable value. For example,<br /><br /> build_roms.sh -p FFLAGS<br /><br /><span class="red"> -noclean</span> Do not clean already compiled objects.<br /><br />'''Example:'''<br /><br /><span class="green"> ./build_roms.sh -j 2</span></div>

==User Definable Configuation Options==
ROMS has a growing list of choices the user must make about the compilation before starting the compile process. These user-defined variables can be set in the build script. Since we use '''gnu make''', it is possible to set the value of these variables in the Unix environment, rather than inside the '''Makefile''' (see [[gmake]]). This page describes the user-definable variables understood by the ROMS '''makefile'''.

===Application and File Locations===

<section begin=ROMS_APPLICATION />;<span id="ROMS_APPLICATION"></span>[[ROMS_APPLICATION]]
:CPP option defining a particular application (for example, [[UPWELLING]]). The makefile will include its associated header file which is located in either the '''ROMS/Include''' directory or the path specified in the [[#MY_HEADER_DIR|MY_HEADER_DIR]] definition. The header file name is the lowercase value of [[ROMS_APPLICATION]] with the '''.h''' extension (for example, [[upwelling.h]]). Recall the CPP option defining an application must be unique.<section end=ROMS_APPLICATION />

<section begin=MY_ROOT_DIR />;<span id="MY_ROOT_DIR"></span>[[MY_ROOT_DIR]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. It can be used to set the base path to your ROMS source code and/or your application input and output files. This variable is not strictly necessary but can make setting the necessary paths easier.<section end=MY_ROOT_DIR />

<section begin=MY_PROJECT_DIR />;<span id="MY_PROJECT_DIR"></span>[[MY_PROJECT_DIR]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. [[MY_PROJECT_DIR]] is most often used to set the paths: [[#MY_HEADER_DIR|MY_HEADER_DIR]], [[#MY_ANALYTICAL_DIR|MY_ANALYTICAL_DIR]], and [[#BINDIR|BINDIR]] and to set the base path to [[#SCRATCH_DIR|SCRATCH_DIR]]. This can be seen near the bottom of the build script.<section end=MY_PROJECT_DIR />

<section begin=MY_ROMS_SRC />;<span id="MY_ROMS_SRC"></span>[[MY_ROMS_SRC]]
: This path is internal to the build script and has no equivalent in the [[makefile]]. It is the location of your ROMS source code and is necessary for the build script to work properly.<section end=MY_ROMS_SRC />

<section begin=COMPILERS />;<span id="COMPILERS"></span>[[COMPILERS]]
: Directory where [[make]] can find your system/compiler specific mk include files as described in [[#FORT|FORT]]. In some situations you will need to edit the appropriate mk file in the Compilers directory (i.e. '''Compilers/Linux-gfortran.mk'''). Rather than editing the mk files released with ROMS and possibly creating conflicts during updates, we suggest that you copy the appropriate system specific mk file and the '''Compilers/make_macros.mk''' to another folder and set [[COMPILERS]] to that folder.<section end=COMPILERS />

===Dynamic CPP Option Tuning===

<section begin=MY_CPP_FLAGS />;<span id="MY_CPP_FLAGS"></span>[[MY_CPP_FLAGS]]
: Additional CPP options to activate. Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the '''-D''' syntax. Notice that you need to use your shell's quoting syntax to enclose the definitions. For example:
:<div class="box">setenv [[MY_CPP_FLAGS]] "-D[[Options#AVERAGES|AVERAGES]]"<br />setenv [[MY_CPP_FLAGS]] "${[[MY_CPP_FLAGS]]} -D[[Options#ANA_INITIAL|ANA_INITIAL]]"</div>
:can be used to write time-averaged fields and use analytical initial conditions. Notice that you can have as many definitions as you want by appending values.<section end=MY_CPP_FLAGS />

===Compiling Options===

{{warning}}'''Warning:''' Be sure to leave environment variables meant to be off set to an empty string or commented out. Any string value (including '''off''') will evaluate to '''TRUE''' in conditional if-statements.

====Parallel Execution Options====

<section begin=USE_MPI />;<span id="USE_MPI"></span>[[USE_MPI]]
: Compile your application using '''MPI''' (distributed-memory) libraries.<section end=USE_MPI />

<section begin=USE_MPIF90 />;<span id="USE_MPIF90"></span>[[USE_MPIF90]]
: If [[#USE_MPI|USE_MPI]] is activated, use this to turn on compilation via the script <span class="red">mpif90</span>. This is needed in most Linux operating systems. In some systems with native '''MPI''' libraries the compilation does not require '''MPICH''' type scripts. This macro is also convenient when there are several Fortran compiliers (<span class="red">ifort</span>, <span class="red">pgf90</span>, <span class="red">pathf90</span>) in the system that use <span class="red">mpif90</span>. In this case, the user needs to select the desired compiler via the [[#FORT|FORT]] variable and turn on both [[#USE_MPI|USE_MPI]] and [[USE_MPIF90]] macros.<section end=USE_MPIF90 />

<section begin=which_MPI />;<span id="which_MPI"></span>[[which_MPI]]
: If [[#USE_MPI|USE_MPI]] and [[#USE_MPIF90|USE_MPIF90]] are activated, this macro is used to determine which <span class="red">mpif90</span> to compile with. This macro is convenient when there are several Fortran compiliers (<span class="red">ifort</span>, <span class="red">pgf90</span>, etc.) in the system that use <span class="red">mpif90</span>. The choice of which <span class="red">mpif90</span> to use will be determined by the combination of this macro and the [[#FORT|FORT]] macro.<section end=which_MPI />

<section begin=USE_OpenMP />;<span id="USE_OpenMP"></span>[[USE_OpenMP]]
: Compile your application using OpenMP librairies and compiler options. '''Note:''' you cannot set [[USE_OpenMP]] and [[#USE_MPI|USE_MPI]] at the same time.<section end=USE_OpenMP />

====Compiler and Options====

<section begin=FORT />;<span id="FORT"></span>[[FORT]]
: The ROMS make system will include a file with all the settings that depend on the system and the compiler. The name of this file is determined from the output of the <span class="red">uname</span> command and the value of [[FORT]] (''i.e.'' '''Linux-ifort.mk'''). Set your compiler here from the following list:
::<div class="box"> Operating System Compiler(s)<br><br> AIX: xlf<br> CYGWIN: df, g95, gfortran, ifort<br> Darwin: f90, gfortran, ifort, pgi, xlf<br> IRIX64: f90<br> Linux: ftn, g95, gfortran, ifc, ifort, necsx, path, pgi<br> MINGW: g95, gfortran<br> OSF1: f90<br> SunOS: f95, ftn<br> UNICOS-mk: f90<br> UNICOS-mp: ftn<br> UNICOS-sk: f90</div>
: Feel free to send us additional rule files to include!<section end=FORT />

<section begin=USE_DEBUG />;<span id="USE_DEBUG"></span>[[USE_DEBUG]]
: Activate debugging compiler options.<section end=USE_DEBUG />

<section begin=USE_LARGE />;<span id="USE_LARGE"></span>[[USE_LARGE]]
: If applicable, activate 64-bit compilation.<section end=USE_LARGE />

====Library Types and Executable====

The options in this section allow you to choose whether to build ROMS using a shared or static (or both) library. If you enable both SHARED, STATIC and EXEC, the ROMS executable will automatically choose the shared library (except in cygwin). Building the ROMS executable using the shared library <span class="red">is not recommended</span> because it requires keeping track of the matching libROMS.{so|dylib} which is located in the Build_roms or Build_romsG directory and will be lost and/or replaced with each new build. The option to build the shared version of libROMS is for use in model coupling systems.

{{note}}'''NOTE:''' If both SHARED and STATIC are unset but EXEC is set, ROMS will automatically build with the static library. If all three are unset, ROMS will default to building the roms executable with the static library.

<section begin=SHARED />;<span id="SHARED"></span>[[SHARED]]
: Build the shared library version of libROMS.<section end=SHARED />

<section begin=STATIC />;<span id="STATIC"></span>[[STATIC]]
: Build the static library version of libROMS.<section end=STATIC />

<section begin=EXEC />;<span id="EXEC"></span>[[EXEC]]
: Build the ROMS executable. In most cases you will want this on. The only time this should be off is if you are building the ROMS library for inclusion in a model coupling systems.<section end=EXEC/>

====ROMS I/O Choices====
A more complete description of the available options can be found on the [[IO]] page. Most users will want to enable at least USE_NETCDF4 because that will instruct the ROMS build system to use nf-config to determine the necessary libraries and paths to link into the ROMS executable.

<section begin=USE_NETCDF4 />;<span id="USE_NETCDF4"></span>[[USE_NETCDF4]]
: If applicable, link with '''NetCDF-4''' library. Notice that the '''NetCDF-4''' library needs both the '''HDF5''' and, if you want to enable parallel I/O, '''MPI''' libraries. In order to write the output files in '''NetCDF-4/HDF5''' format, you will need to set the [[Options#HDF5|HDF5]] CPP Option.<section end=USE_NETCDF4 />

<section begin=USE_PARALLEL_IO />;<span id="USE_PARALLEL_IO"></span>[[USE_PARALLEL_IO]]
: Activate parallel I/O using parallel enabled '''NetCDF-4/HDF5''' libraries. You will also need to set the [[Options#HDF5|HDF5]] and [[Options#PARALLEL_IO|PARALLEL_IO]] CPP Options to enable parallel I/O.<section end=USE_PARALLEL_IO />

<section begin=USE_PIO />;<span id="USE_PIO"></span>[[USE_PIO]]
: Activate parallel I/O using Parallel IO (PIO) libraries developed at NCAR. More information can be found on on the [[IO|ROMS I/O]] page.<section end=USE_PIO />

<section begin=USE_HDF5 />;<span id="USE_HDF5"></span>[[USE_HDF5]]
: If any of the coupling components use the HDF5 Fortran API for primary I/O, we need to compile the main driver with the HDF5 Fortran library.<section end=USE_HDF5 />

====Earth Systems Models (ESM) coupling====
If coupling Earth Systems Models (ESM), set the location of the ESM component libraries and modules. The strategy is to compile and link each ESM component separately first, and then ROMS since it is driving the coupled system. Only the ESM components activated are considered and the rest are ignored. Some components like WRF cannot be built in a directory specified by the user but in its own root directory, and cannot be moved when debugging with tools like TotalView.

<section begin=WRF_SRC_DIR />;<span id="WRF_SRC_DIR"></span>[[WRF_SRC_DIR]]
: WRF cannot be built in a directory specified by the user; it must be built in its own root directory and cannot be moved when debugging with tools like TotalView. If you wish to couple ROMS with the Earth Systems Model (ESM) WRF, ROMS needs to know the WRF root directory.<section end=WRF_SRC_DIR />

====Various Path options====

<section begin=USE_MY_LIBS />;<span id="USE_MY_LIBS"></span>[[USE_MY_LIBS]]
: Use custom library paths for '''ESMF''', '''HDF5''', '''NetCDF''' libraries, etc.<section end=USE_MY_LIBS />

<section begin=MY_PATHS />;<span id="MY_PATHS"></span>[[MY_PATHS]]
: Path to your <span class="red">my_build_paths.csh</span> (or .sh) file that contains custom library paths for your system.<section end=MY_PATHS />

<section begin=MY_HEADER_DIR />;<span id="MY_HEADER_DIR"></span>[[MY_HEADER_DIR]]
: This is the Directory where your application's header file is located. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that is included in [[cppdefs.h]]. Notice that if the header file is located in the '''ROMS/Include''' directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.<section end=MY_HEADER_DIR />

<section begin=MY_ANALYTICAL_DIR />;<span id="MY_ANALYTICAL_DIR"></span>[[MY_ANALYTICAL_DIR]]
: This is the directory containing the user's [[Functionals|analytic files]], if any. Templates of the analytic expressions are provided in the '''User/Functionals''' directory. If analytic expressions are activated, the makefile will first load the ones located in [[MY_ANALYTICAL_DIR]] and then the ones located in '''ROMS/Functionals'''. We recommended that you do not modify the analytical expressions in the '''ROMS/Functionals''' directory since these are the official versions of these files and updates will conflict with your changes. The user however can modify those in '''User/Functionals''' or some other directory and specify which directory with this variable.<section end=MY_ANALYTICAL_DIR />

<section begin=BINDIR />;<span id="BINDIR"></span>[[BINDIR]]
: Directory where your <span class="red">romsG</span>, <span class="red">romsM</span>, <span class="red">romsO</span>, or <span class="red">romsS</span> executable will be created.<section end=BINDIR />

<section begin=SCRATCH_DIR />;<span id="SCRATCH_DIR"></span>[[SCRATCH_DIR]]
: Directory where your temporary objects and files will be placed during the build process.<section end=SCRATCH_DIR />

==Build Script Debugging==

{{note}}'''Note:''' The build script has the <span class="red">-p</span> option to inquire and report about any of the environmental and local variables used in the [[makefile]] or build scripts. Usefull build commands are:
:<div class="box"><span class="red">build_roms.sh -p</span> '''LIBS'''<br /><span class="red">build_roms.sh -p</span> '''libraries'''<br /><span class="red">build_roms.sh -p</span> '''modules'''<br /><span class="red">build_roms.sh -p</span> '''includes'''<br /><span class="red">build_roms.sh -p</span> '''sources'''<br /><span class="red">build_roms.sh -p</span> '''CPPFLAGS'''<br /><span class="red">build_roms.sh -p</span> '''MY_CPP_FLAGS'''<br /><span class="red">build_roms.sh -p</span> '''MY_ROOT_DIR'''<br /><span class="red">build_roms.sh -p</span> '''MY_ROMS_SRC'''<br /><span class="red">build_roms.sh -p</span> '''MKFILE'''<br /><span class="red">build_roms.sh -p</span> '''FFLAGS'''<br /></div>
:We highly recommend to use this option to inquire your environment when having errors during compiling and linking. For Example:
:<div class="box"><span class="red">build_roms.sh -p</span> '''MKFILE'''<br /><br /><br /><span class="twilightBlue">Using customized library paths from:</span> <span class="hotPink">/Users/arango/Compilers/ROMS/my_build_paths.sh</span><br /><br /><span class="red">MKFILE</span> = /Users/arango/Compilers/ROMS/Darwin-ifort.mk</div>
:Notice that there is not a compilation, but a report about the value of local [[makefile]] variable <span class="red">MKFILE</span>.

==Library and Executable Paths==

{{note}}'''Note:''' in SVN revision 933 (January 26, 2019) the custom libraries section was removed from the ROMS build script and placed in <span class="red">Compilers/my_build_paths.csh</span> and <span class="red">Compilers/my_build_paths.sh</span> to both streamline the ROMS build script and make setting up new applications less tedious. More information can be found in ROMS [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. Prior to ROMS release 933, library and executable paths where set in ROMS build script.

The path of the libraries and MPI implementations required by ROMS can be set using environment variables which take precedence over the values specified in the makefile macro definitions file ('''Compilers/*.mk'''). If so desired, set [[USE_MY_LIBS]] to '''yes''' and properly set [[MY_PATHS]]. Visit [[my build paths]] for more details.

roms.in

2025-05-05T18:35:40Z

Robertson:

<div class="title">Standard Input Script - <span class="red">roms.in</span></div>

{{note}} '''Notice:''' In SVN revision 933 (January 26, 2019), all "<span class="red">ocean_*.in</span>" files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794].

File <span class="red">roms.in</span> is the ROMS standard input file to any model run. This file sets the application spatial dimensions and many of the parameters that are not specified at compile time, including parallel tile decomposition, timestepping, physical coefficients and constants, vertical coordinate set-up, logical switches and flags to control the frequency of output, the names of input and output NetCDF files, and additional input scripts names for [[s4dvar.in|data assimilation]], [[stations.in|stations]], [[floats.in| floats trajectories]], [[biology.in|ecosystem models]], and [[sediment.in|sediment model]].

This standard input ASCII file is organized in several sections as shown below, with links to more detailed explanation where required.

{{note}} '''Notice:''' A detailed information about ROMS input script file syntax can be found [[Input Parameter Files|here]].

{{note}} '''Notice:''' A default <span class="red">roms.in</span> input script is provided in the '''User/External''' subdirectory. Also there are several standard input scripts in the '''ROMS/External''' subdirectory which are used in the distributed [[Test Cases|test cases]]. They are usually named <span class="red">roms_</span><span class="blue">app</span><span class="red">.in</span> where <span class="blue">app</span> is the lowercase of the test case [[Options|cpp option]].

== Configuration Parameters ==

* Application title. This string will be saved in the output NetCDF files. <div class="box"> [[Variables#title|TITLE]] = Wind-Driven Upwelling/Downwelling over a Periodic Channel</div>

* C-preprocessing Flag to define the specific configuration.<div class="box"> MyAppCPP = UPWELLING</div>Though this is set by [[makefile#ROMS_APPLICATION | ROMS_APPLICATION]] in the [[makefile]] or [[build Script]], ROMS is also compiled with <span style="white-space:nowrap;">'''-D$(ROMS_APPLICATION)'''</span>, which allows the use of<div class="box">#ifdef UPWELLING</div>for instance. The net result of both<div class="box">-D$(ROMS_APPLICATION)=UPWELLING -DUPWELLING</div>is that '''ROMS_APPLICATION''' becomes '''1''' in the source code. ROMS therefore needs to be told the application name here as well in order to report it to the output file.

* Input variable information file name. This file needs to be processed first so all information arrays can be initialized properly. Notice that we need an absolute or relative path for input metadata file [[varinfo.dat]]. There are many posts in the ROMS Forum of new users that fail to specify the correct location of this file. Expert users usually have the own modified copy of this file for a particular application.<div class="box"> [[Variables#varname|VARNAME]] = ROMS/External/[[varinfo.dat]]</div>

:{{note}}'''NOTE:''' Starting with '''revision 460''' file names can be up to 256 characters long. Previously only 80 characters were allowed.

* Number of nested grids.<div class="box"> [[Ngrids]] = 1</div>

* Number of grid nesting layers. This parameter is used to allow refinement and composite grid combinations.<div class="box"> [[Variables#NestLayers|NestLayers]] = 1</div>

* Number of grids in each nesting layer, [1:[[Variables#NestLayers|NestLayers]]] values are expected.<div class="box">[[Variables#GridsInLayer|GridsInLayer]] = 1</div>

* Grid dimension parameters. These are used to dynamically allocate all model state variables upon execution.<div class="box"> [[Variables#Lm|Lm]] == 41 ! Number of I-direction INTERIOR RHO-points<br /> [[Variables#Mm|Mm]] == 80 ! Number of J-direction INTERIOR RHO-points<br /> [[Variables#N|N]] == 16 ! Number of vertical levels<br /><br /> [[Variables#Nbed|Nbed]] = 0 ! Number of sediment bed layers<br /><br /> [[Variables#NAT|NAT]] = 2 ! Number of active tracers (usually, 2)<br /> [[Variables#NPT|NPT]] = 0 ! Number of inactive passive tracers<br /> [[Variables#NCS|NCS]] = 0 ! Number of cohesive (mud) sediment tracers<br /> [[Variables#NNS|NNS]] = 0 ! Number of non-cohesive (sand) sediment tracers</div>

* Domain decomposition parameters for serial, distributed-memory or shared-memory configurations used to determine tile horizontal range indices ([[Istr]],[[Iend]]) and ([[Jstr]],[[Jend]]), [1:[[Ngrids]]] values are expected.<div class="box"> [[NtileI]] == 1 ! I-direction partition<br /> [[NtileJ]] == 1 ! J-direction partition</div>

==Tracer Advection Schemes==

Set horizontal and vertical advection schemes for active and inert tracers. A different advection scheme is allowed for each tracer. For example, a positive-definite (monotonic) algorithm can be activated for salinity and inert tracers, while a different one is set for temperature.

It is more advantageous to set the horizontal and vertical advection schemes for each tracer with switches instead of a single CPP flag for all of them. Positive-definite and monotonic algorithms (i.e., MPDATA and HSIMT) are appropriate and useful for positive fields like salinity, inert, biological, and sediment tracers. However, since the temperature has a dynamic range with negative and positive values in the ocean, other advection schemes are more appropriate.

Currently, the following tracer advection schemes are available and are activated using the associated Keyword:
:<div class="box">Keyword Advection Algorithm<br /><br />A4 4th-order Akima (horizontal/vertical)<br />C2 2nd-order centered differences (horizontal/vertical)<br />C4 4th-order centered differences (horizontal/vertical)<br />HSIMT 3th-order HSIMT with TVD limiter (horizontal/vertical)<br />MPDATA recursive flux corrected MPDATA (horizontal/vertical)<br />SPLINES parabolic splines reconstruction (only vertical)<br />SU3 split third-order upstream (horizontal/vertical)<br />U3 3rd-order upstresm-bias (only horizontal)</div>

The user has the option of specifying the full Keyword or the first two letters, regardless if using uppercase or lowercase.

If using either HSIMT ([[Bibliography#WuH_2010a|Wu and Zhu, 2010]]) or MPDATA ([[Bibliography#SmolarkiewiczPK_1998a|Smolarkiewicz and Margolin, 1998]]) options, the user needs to set the same scheme for both horizontal and vertical advection to preserve monotonicity.

*Horizontal and vertical advection for each active (temperature and salinity) and inert tracer, [1:[[Variables#NAT|NAT]]+[[Variables#NPT|NPT]],[[Variables#Ngrids|Ngrids]]] values are expected.<div class="box"> [[Variables#Hadvection|Hadvection]] == U3 \ ! temperature<br /> U3 \ ! salinity<br /> HSIMT \ ! dye_01, inert(1)<br /> HSIMT ! dy2_02, inert(2)<br /><br /> [[Variables#Vadvection|Vadvection]] == C4 \ ! temperature<br /> C4 \ ! salinity<br /> HSIMT \ ! dye_01, inert(1)<br /> HSIMT ! dy2_02, inert(2)</div>

*Horizontal and vertical advection for each active (temperature and salinity) and inert tracer for adjoint-based algorithms can have different horizontal schemes, [1:[[Variables#NAT|NAT]]+[[Variables#NPT|NPT]],[[Variables#Ngrids|Ngrids]]] values are expected.<div class="box">[[Variables#ad_Hadvection|ad_Hadvection]] == U3 \ ! temperature<br /> U3 \ ! salinity<br /> HSIMT \ ! dye_01, inert(1)<br /> HSIMT ! dy2_02, inert(2)<br /><br />[[Variables#ad_Vadvection|ad_Vadvection]] == C4 \ ! temperature<br /> C4 \ ! salinity<br /> HSIMT \ ! dye_01, inert(1)<br /> HSIMT ! dy2_02, inert(2)</div>

==Lateral Open Boundary Conditions Parameters==

*The lateral boundary conditions are now specified with logical switches instead of CPP flags to allow nested grid configurations. Their values are loaded into the structured array:<div class="box">[[Variables#LBC|LBC]](1:4, nLBCvar, [[Ngrids]])</div>where '''1:4''' are the numbered boundary edges, '''nLBCvar''' are the number LBC state variables, and [[Ngrids]] is the number of nested grids. For example, to apply gradient boundary conditions for free-surface we use:<div class="box">[[Variables#LBC|LBC]](iwest, isFsur, ng) % gradient<br />[[Variables#LBC|LBC]](ieast, ... , ng) % gradient<br />[[Variables#LBC|LBC]](isouth, ... , ng) % gradient<br />[[Variables#LBC|LBC]](inorth, ... , ng) % gradient</div>The lateral boundary conditions are entered with a keyword. A value is expected for each boundary segment per nested grid for each state variable. Each tracer variable requires [1:4,1:[[NAT]]+[[NPT]],[[Ngrids]]] values. [1:4,1:[[Ngrids]]] values are expected for other variables. The boundary order is: '''1'''=west, '''2'''=south, '''3'''=east, and '''4'''=north. That is, anticlockwise starting at the western boundary.<br /><br />The keyword is case insensitive and usually has three characters. However, it is possible to have compound keywords, if applicable. For example, the keyword '''RadNud''' implies radiation boundary condition with nudging. This combination is usually used in active/passive radiation conditions.<div class="box"><span class="twilightBlue">! Keyword Lateral Boundary Condition Type<br />!<br />! Cha Chapman<br />! Cla Clamped<br />! Clo Closed<br />! Fla Flather _____N_____ j=Mm<br />! Gra Gradient | 4 |<br />! Nes Nested | |<br />! Nud Nudging 1 W E 3<br />! Per Periodic | |<br />! Rad Radiation |_____S_____|<br />! Red Reduced Physics 2 j=1<br />! i=1 i=Lm<br />! W S E N<br />! e o a o<br />! s u s r<br />! t t t t<br />! h h<br />!<br />! 1 2 3 4</span><br /><br /> [[Variables#LBC|LBC(isFsur)]] == Per Clo Per Clo ! free-surface<br /> [[Variables#LBC|LBC(isUbar)]] == Per Clo Per Clo ! 2D U-momentum<br /> [[Variables#LBC|LBC(isVbar)]] == Per Clo Per Clo ! 2D V-momentum<br /> [[Variables#LBC|LBC(isUvel)]] == Per Clo Per Clo ! 3D U-momentum<br /> [[Variables#LBC|LBC(isVvel)]] == Per Clo Per Clo ! 3D V-momentum<br /> [[Variables#LBC|LBC(isMtke)]] == Per Clo Per Clo ! mixing TKE<br /><br /> [[Variables#LBC|LBC(isTvar)]] == Per Clo Per Clo \ ! temperature<br /> Per Clo Per Clo ! salinity</div>

*Adjoint-based algorithms can have different lateral boundary conditions keywords.<div class="box">[[Variables#ad_LBC|ad_LBC(isFsur)]] == Per Clo Per Clo ! free-surface<br />[[Variables#ad_LBC|ad_LBC(isUbar)]] == Per Clo Per Clo ! 2D U-momentum<br />[[Variables#ad_LBC|ad_LBC(isVbar)]] == Per Clo Per Clo ! 2D U-momentum<br />[[Variables#ad_LBC|ad_LBC(isUvel)]] == Per Clo Per Clo ! 3D U-momentum<br />[[Variables#ad_LBC|ad_LBC(isVvel)]] == Per Clo Per Clo ! 3D V-momentum<br />[[Variables#ad_LBC|ad_LBC(isMtke)]] == Per Clo Per Clo ! mixing TKE<br /><br />[[Variables#ad_LBC|ad_LBC(isTvar)]] == Per Clo Per Clo \ ! temperature<br /> Per Clo Per Clo ! salinity</div>

*Lateral open boundary edge volume conservation switch for nonlinear model and adjoint-based algorithm. This is usually activated with radiation boundary conditions to enforce global mass conservation. Notice that these switches should not be activated if tidal forcing, [1:[[Ngrids]]] values are expected.<div class="box"> [[Variables#VolCons|VolCons(west)]] == F ! western boundary<br /> [[Variables#VolCons|VolCons(east)]] == F ! eastern boundary<br /> [[Variables#VolCons|VolCons(south)]] == F ! southern boundary<br /> [[Variables#VolCons|VolCons(north)]] == F ! northern boundary<br /><br />[[Variables#ad_VolCons|ad_VolCons(west)]] == F ! western boundary<br />[[Variables#ad_VolCons|ad_VolCons(east)]] == F ! eastern boundary<br />[[Variables#ad_VolCons|ad_VolCons(south)]] == F ! southern boundary<br />[[Variables#ad_VolCons|ad_VolCons(north)]] == F ! northern boundary</div>

==Timestepping and Iterations Parameters==

* Timestepping parameters.<div class="box"> [[Variables#ntimes|NTIMES]] = 1440 ! Number of timesteps<br /> [[Variables#dt|DT]] == 300.0d0 ! Timestep size (seconds)<br /> [[Variables#ndtfast|NDTFAST]] == 30 ! Number of barotropic steps</div>

* Total number of timesteps for computing observations impacts interval during the analysis ([[Variables#ntimes_ana|NTIMES_ANA]]) or forecast ([[Variables#ntimes_fct|NTIMES_FCT]]) cycle.<div class="box"> [[Variables#ntimes_ana|NTIMES_ANA]] = 1440 ! analysis interval<br /> [[Variables#ntimes_fct|NTIMES_FCT]] = 1440 ! forecast interval</div>

* Model iteration loops parameters.<div class="box"> [[Variables#ERstr|ERstr]] = 1 ! Starting perturbation or iteration<br /> [[Variables#ERend|ERend]] = 1 ! Ending perturbation or iteration<br /> [[Variables#Nouter|Nouter]] = 1 ! Maximum number of 4DVar outer loop iterations<br /> [[Variables#Ninner|Ninner]] = 1 ! Maximum number of 4DVar inner loop iterations<br /> [[Variables#Nintervals|Nintervals]] = 1 ! Number of stochastic optimals interval divisions</div>

* Number of eigenvalues ([[Variables#NEV|NEV]]) and eigenvectors ([[Variables#NCV|NCV]]) to compute for the Lanczos/Arnoldi problem in the Generalized Stability Theory (GST) analysis. [[Variables#NCV|NCV]] must be greater than [[Variables#NEV|NEV]].<div class="box"> [[Variables#NEV|NEV]] = 2 ! Number of eigenvalues<br /> [[Variables#NCV|NCV]] = 10 ! Number of eigenvectors</div>{{note}} '''Notice:''' At present, there is no ''apriori'' analysis to guide the selection of <span class="twilightBlue">NCV</span> relative to <span class="twilightBlue">NEV</span>. The only formal requirement is that <span class="twilightBlue">NCV > NEV</span>. However in optimal perturbations, it is recommended to have <span class="twilightBlue">NCV ≥ 2*NEV</span>. In Finite Time Eigenmodes (FTE) and Adjoint Finite Time Eigenmodes (AFTE) the requirement is to have <span class="twilightBlue">NCV ≥ 2*NEV+1</span>. The efficiency of calculations depends critically on the combination of <span class="twilightBlue">NEV</span> and <span class="twilightBlue">NCV</span>. If <span class="twilightBlue">NEV</span> is large (greater than 10 say), you can use <span class="twilightBlue">NCV=2*NEV+1</span> but for <span class="twilightBlue">NEV</span> small (less than 6) it will be inefficient to use <span class="twilightBlue">NCV=2*NEV+1</span>. In complicated applications, you can start with <span class="twilightBlue">NEV=2</span> and <span class="twilightBlue">NCV=10</span>. Otherwise, it will iterate for very long time.

==Output Frequency Parameters==

* Flags controlling the frequency of output.<div class="box"> [[Variables#nrrec|NRREC]] = 0 ! Model restart flag<br /> [[Variables#LcycleRST|LcycleRST]] == T ! Switch to recycle restart time records<br /> [[Variables#nRST|NRST]] == 288 ! Number of timesteps between writing restart records<br /> [[Variables#nSTA|NSTA]] == 1 ! Number of timesteps between stations records<br /> [[Variables#nFLT|NFLT]] == 1 ! Number of timesteps between floats records<br /> [[Variables#ninfo|NINFO]] == 1 ! Number of timesteps between printing information diagnostics</div>

* Output history, average, diagnostic files parameters.<div class="box"> [[Variables#ldefout|LDEFOUT]] == T ! File creation/append switch<br /> [[Variables#nHIS|NHIS]] == 72 ! Number of timesteps between writing history records<br /> [[Variables#ndefHIS|NDEFHIS]] == 0 ! Number of timesteps between creation of new history file<br /> [[Variables#nQCK|NQCK]] == 0 ! Number of timesteps between writing quicksave records<br /> [[Variables#ndefQCK|NDEFQCK]] == 0 ! Number of timesteps between creation of new quicksave file<br /> [[Variables#ntsAVG|NTSAVG]] == 1 ! Starting averages timestep<br /> [[Variables#nAVG|NAVG]] == 72 ! Number of timesteps between writing averages records<br /> [[Variables#ndefAVG|NDEFAVG]] == 0 ! Number of timesteps between creation of new averages file<br /> [[Variables#ntsDIA|NTSDIA]] == 1 ! Starting diagnostics timestep<br /> [[Variables#nDIA|NDIA]] == 72 ! Number of timesteps between writing diagnostics records<br /> [[Variables#ndefDIA|NDEFDIA]] == 0 ! Number of timesteps between creation of new diagnostics file</div>

* Output tangent linear and adjoint models parameters.<div class="box"> [[Variables#LcycleTLM|LcycleTLM]] == F ! Switch to recycle TLM time records<br /> [[Variables#nTLM|NTLM]] == 72 ! Number of timesteps writing between TLM records<br /> [[Variables#ndefTLM|NDEFTLM]] == 0 ! Number of timesteps between creation of new TLM file<br> [[Variables#LcycleADJ|LcycleADJ]] == F ! Switch to recycle ADM time records<br> [[Variables#nADJ|NADJ]] == 72 ! Number of timesteps between writing ADM records<br> [[Variables#ndefADJ|NDEFADJ]] == 0 ! Number of timesteps between creation of new ADM file<br> [[Variables#nSFF|NSFF]] == 72 ! Number of timesteps between 4DVAR adjustment of<br> ! surface forcing fluxes<br> [[Variables#nOBC|NOBC]] == 72 ! Number of timesteps between 4DVAR adjustment of<br> ! open boundary fields</div>

* Output check pointing GST restart parameters.<div class="box"> [[Variables#LmultiGST|LmultiGST]] = F ! one eigenvector per history file<br /> [[Variables#LrstGST|LrstGST]] = F ! GST restart switch<br /> [[Variables#MaxIterGST|MaxIterGST]] = 500 ! maximum number of iterations<br /> [[Variables#nGST|NGST]] = 10 ! check pointing interval</div>

== Physical and Numerical Parameters ==

* Relative accuracy of the Ritz values computed in the GST analysis.<div class="box"> [[Variables#Ritz_tol|Ritz_tol]] = 1.0d-15</div>

* Harmonic/biharmonic horizontal diffusion of all active and passive (dye) tracers for the nonlinear model and adjoint-based algorithms: [1:[[NAT]]+[[NPT]],[[Ngrids]]] values are expected. Diffusion coefficients for biology and sediment tracers are set in their respective input scripts.<div class="box"> [[Variables#tnu2|TNU2]] == 0.0d0 0.0d0 ! m2/s<br /> [[Variables#tnu4|TNU4]] == 2*0.0d0 ! m4/s<br /><br /> [[Variables#ad_tnu2|ad_TNU2]] == 0.0d0 0.0d0 ! m2/s<br /> [[Variables#ad_tnu4|ad_TNU4]] == 0.0d0 0.0d0 ! m4/s</div>

* Harmonic/biharmonic, horizontal viscosity coefficient for the nonlinear model and adjoint-based algorithms: [1:[[Ngrids]] values are expected. Only used if the appropriate CPP options are defined.<div class="box"> [[Variables#visc2|VISC2]] == 0.0d0 ! m2/s<br /> [[Variables#visc4|VISC4]] == 0.0d0 ! m4/s<br /><br /> [[Variables#ad_visc2|ad_VISC2]] == 0.0d0 ! m2/s<br /> [[Variables#ad_visc4|ad_VISC4]] == 0.0d0 ! m4/s</div>

* Logical switches ('''TRUE'''/'''FALSE''') to increase/decrease horizontal viscosity and/or diffusivity in specific areas of the application domain (like sponge areas) for the desired application grid.<div class="box"> [[Variables#LuvSponge|LuvSponge]] == F ! horizontal momentum<br />[[Variables#LtracerSponge|LtracerSponge]] == F F ! temperature, salinity, inert</div>

* Background vertical mixing coefficients for active ([[NAT]]) and inert ([[NPT]]) tracers for the nonlinear model and basic state scale factor in adjoint-based algorithms: [1:[[NAT]]+[[NPT]],[[Ngrids]]] values are expected.<div class="box"> [[Variables#Akt_bak|AKT_BAK]] == 1.0d-6 1.0d-6 ! m2/s<br /><br /> [[Variables#ad_Akt_fac|ad_AKT_fac]] == 1.0d0 1.0d0 !nondimensional</div>

* Background vertical mixing coefficient for momentum for the nonlinear model and basic state scale factor in the adjoint-based algorithms: [1:[[Ngrids]]] values are expected.<div class="box"> [[Variables#Akv_bak|AKV_BAK]] == 1.0d-5 ! m2/s<br /><br /> [[Variables#ad_Akv_fac|ad_AKV_fac]] == 1.0d0 !nondimensional</div>

* Upper threshold values to limit vertical mixing coefficients computed from vertical mixing parameterizations. Although this is an engineering fix, the vertical mixing values inferred from ocean observations are rarely higher than this upper limit value.<div class="box"> [[Variables#Akt_limit|AKT_LIMIT]] == 1.0d-3 1.0d-3 ! m2/s<br /><br /> [[Variables#Akv_limit|AKV_LIMIT]] == 1.0d-3 ! m2/s</div>

* Turbulent closures parameters.<div class="box"> [[Variables#Akk_bak|AKK_BAK]] == 5.0d-6 ! m2/s<br /> [[Variables#Akp_bak|AKP_BAK]] == 5.0d-6 ! m2/s<br /> [[Variables#tkene2|TKENU2]] == 0.0d0 ! m2/s<br /> [[Variables#tkenu4|TKENU4]] == 0.0d0 ! m4/s</div>

* [[Vertical Mixing Parameterizations#Generic Length Scale|Generic length-scale]] turbulence closure parameters. These parameters are used when [[GLS_MIXING]] is activated.<div class="box"> [[Variables#gls_p|GLS_P]] == 3.0d0 ! <span class="red">K-epsilon</span><br /> [[Variables#gls_m|GLS_M]] == 1.5d0 ! Turbulent kinetic energy exponent<br /> [[Variables#gls_n|GLS_N]] == -1.0d0 ! Turbulent length scale exponent<br /> [[Variables#gls_Kmin|GLS_Kmin]] == 7.6d-6 ! Minimum value of specific turbulent energy<br /> [[Variables#gls_Pmin|GLS_Pmin]] == 1.0d-12 ! Minimum Value of dissipation<br /><br />'''! Closure independent constraint parameters:'''<br><br> [[Variables#gls_cmu0|GLS_CMU0]] == 0.5477d0 ! Stability coefficient<br /> [[Variables#gls_c1|GLS_C1]] == 1.44d0 ! Shear production coefficient<br /> [[Variables#gls_c2|GLS_C2]] == 1.92d0 ! Dissipation coefficient<br /> [[Variables#gls_c3m|GLS_C3M]] == -0.4d0 ! Buoyancy production coefficient (minus)<br /> [[Variables#gls_c3p|GLS_C3P]] == 1.0d0 ! Buoyancy production coefficient (plus)<br /> [[Variables#gls_sigp|GLS_SIGK]] == 1.0d0 ! Constant Schmidt number for turbulent<br> ! kinetic energy diffusivity<br /> [[Variables#gls_sigp|GLS_SIGP]] == 1.30d0 ! Constant Schmidt number for turbulent<br> ! generic statistical field, "psi"</div>

* Constants used in surface turbulent kinetic energy flux computation.<div class="box"> [[Variables#charnok_alpha|CHARNOK_ALPHA]] == 1400.0d0 ! Charnok surface roughness<br /> [[Variables#zoz_hsig_alpha|ZOS_HSIG_ALPHA]] == 0.5d0 ! Roughness from wave amplitude<br /> [[Variables#sz_alpha|SZ_ALPHA]] == 0.25d0 ! roughness from wave dissipation<br /> [[Variables#crgban_cw|CRGBAN_CW]] == 100.0d0 ! Craig and Banner wave breaking</div>

* Constants used in momentum stress computation.<div class="box"> [[Variables#rdrg|RDRG]] == 3.0d-04 ! m/s<br /> [[Variables#rdrg2|RDRG2]] == 3.0d-03 ! nondimensional<br /> [[Variables#Zob|Zob]] == 0.02d0 ! m<br /> [[Variables#Zos|Zos]] == 0.02d0 ! m</div>

* Height (m) of atmospheric measurements for Bulk fluxes parameterization.<div class="box"> [[Variables#blk_ZQ|BLK_ZQ]] == 10.0d0 ! air humidity<br /> [[Variables#blk_ZT|BLK_ZT]] == 10.0d0 ! air temperature<br /> [[Variables#blk_ZW|BLK_ZW]] == 10.0d0 ! winds</div>

* Minimum depth for wetting and drying.<div class="box"> [[Variables#Dcrit|DCRIT]] == 0.10d0 ! m</div>

* Jerlov water type used to set vertical depth scale for shortwave radiation absorption.<div class="box"> [[Variables#lmd_Jwt|WTYPE]] == 1</div>

* Deepest and shallowest levels to apply surface momentum stress as a [[BODYFORCE|body-force]].<div class="box"> [[Variables#levsfrc|LEVSFRC]] == 15<br /> [[Variables#levbfrc|LEVBFRC]] == 1</div>

* Mean Density and Brunt-Vaisala frequency.<div class="box"> [[Variables#rho0|RHO0]] = 1025.0d0 ! kg/m3<br /> [[Variables#bvf_bak|BVF_BAK]] = 1.0d-4 ! 1/s2</div>

* Timestamp assigned for model initialization, reference time origin for tidal forcing, and model reference time for output NetCDF units attribute.<div class="box"> [[Variables#dstart|DSTART]] = 0.0d0 ! days<br /> [[Variables#tide_start|TIDE_START]] = 0.0d0 ! days<br /> [[Variables#time_ref|TIME_REF]] = 0.0d0 ! yyyymmdd.dd</div>

* Nudging/relaxation time scales, inverse scales will be computed internally, [1:[[Ngrids]]] values are expected. These values are used for two purposes.
# When climatology nudging is active throughout the domain because the logical flags LtracerCLM, Lm3CLM, Lm2CLM etc. are TRUE, these values are the default nudging time scales set in Functionals/ana_nudgcoef.h. Since the user can choose to customize ana_nudgcoef.h, or provide 3-D climatology nudging time scales in an external file, these parameters might not be used
# When nudging is applied in the lateral open boundary conditions because the LBC logical flags are set to "RadNud" the values here set the nudging time scale when the Orlanski radiation scheme detects outflow conditions. When the Orlanski scheme detects inflow conditions, the nudging time scale is TNUDG/OBCFAC (see OBCFAC below).
<div class="box"> [[Variables#Tnudg|TNUDG]] == 2*0.0d0 ! days<br /> [[Variables#Znudg|ZNUDG]] == 0.0d0 ! days<br /> [[Variables#M2nudg|M2NUDG]] == 0.0d0 ! days<br /> [[Variables#M3nudg|M3NUDG]] == 0.0d0 ! days</div>

* Factor between passive (outflow) and active (inflow) (in the Orlanksi radiation sense) open boundary condition nudging time scales, [1:Ngrids]. If [[Variables#obcfac|OBCFAC]] > 1, nudging on inflow is stronger than on outflow (recommended) because the inflow time scale TNUDG/OBCFAC is less than the outflow timescale TNUDG (see above). The passive/active radiation conditions in ROMS follow the method proposed by [[Bibliography#MarchesielloP_2001a | Marchesiello et al. (2001)]]: <math display="block">\frac{\partial \phi}{\partial t} + C_x \frac{\partial \phi}{\partial x} + C_y \frac{\partial \phi}{\partial y} = - \frac{1}{\tau}(\phi - \phi^{ext})</math> with <math display="block">\begin{align} \tau &= \tau_{out} &\text{if} \;\;\; &C_x > 0, \\ \tau &= \tau_{in} &\text{if} \;\;\; &C_x < 0 \;\;\; \text{and} \;\;\; C_y = 0 \end{align}</math> where <math>\phi^{ext}</math> represents the external boundary data and <math>\tau</math> is the nudging time scale with <math>\tau_{out}</math> for outflow, <math>\tau_{in}</math> for inflow, and <math>\tau_{out} \ll \tau_{in}</math>. At outflow, a weak nudging is used to prevent a numerical drift in the solution while avoiding over-specification of the boundary data. During inflow, a strong nudging is applied to avoid data-shock in the solution. The nudging time scales provided above are for the outflow (passive) conditions, <math>\tau_{out}</math>, in days. The inflow nudging factor in the above equation is <math>\frac{1}{\tau_{in}} = \frac{\text{obcfac}}{\tau_{out}}</math>. <div class="box"> [[Variables#obcfac|OBCFAC]] == 10.0d0 ! nondimensional</div>

* Linear equation of State parameters, [1:[[Ngrids]]] values are expected.<div class="box"> [[Variables#R0|R0]] == 1027.0d0 ! kg/m3<br /> [[Variables#T0|T0]] == 10.0d0 ! Celsius<br /> [[Variables#S0|S0]] == 35.0d0 ! nondimensional<br /> [[Variables#Tcoef|TCOEF]] == 1.7d-4 ! 1/Celsius<br /> [[Variables#Scoef|SCOEF]] == 7.6d-4 ! 1/nondimensional</div>

* Slipperiness parameter: 1.0 (free slip) or -1.0 (no slip).<div class="box"> [[Variables#gamma2|GAMMA2]] = 1.0d0</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate horizontal momentum transport point Sources/Sinks (like river runoff) and mass point Sources/Sinks (like volume vertical influx): [1:[[Ngrids]]] values are expected. These switches replace obsolete CPP options [[Options#UV_PSOURCE|UV_PSOURCE]] and [[Options#Q_PSOURCE|Q_PSOURCE]], respectively. In nesting, a particular grid may or may not have Sources/Sinks forcing.<div class="box"> [[Variables#LuvSrc|LuvSrc]] == F ! horizontal momentum transport<br /> [[Variables#LwSrc|LwSrc]] == F ! volume vertical influx</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate tracers point Sources/Sinks (like river runoff) and to specify which tracer variables to consider: [1:[[NAT]]+[[NPT]],[[Ngrids]]] values are expected. Other biological and sediment tracer switches are activated in their respective input scripts. This switch replaces obsolete CPP option [[Options#TS_PSOURCE|TS_PSOURCE]]. In nesting, a particular grid may or may not have tracers Sources/Sinks forcing.<div class="box"> [[LtracerSrc]] == F F ! temperature, salinity, inert</div>

* Logical switches ('''TRUE'''/'''FALSE''') to read and process climatology fields.<div class="box"> [[Variables#LsshCLM|LsshCLM]] == F ! sea-surface height<br/> [[Variables#Lm2CLM|Lm2CLM]] == F ! 2D momentum<br/> [[Variables#Lm3CLM|Lm3CLM]] == F ! 3D momentum<br /><br /> [[Variables#LtracerCLM|LtracerCLM]] == F F ! temperature, salinity, inert</div>

* Logical switches ('''TRUE'''/'''FALSE''') to nudge the desired climatology field(s). If not analytical climatology fields, users need to turn ON the logical switches above to process the fields from the climatology NetCDF file that are needed for nudging.<div class="box"> [[Variables#LnudgeM2CLM|LnudgeM2CLM]] == F ! 2D momentum<br /> [[Variables#LnudgeM3CLM|LnudgeM3CLM]] == F ! 3D momentum<br /><br /> [[Variables#LnudgeTCLM|LnudgeTCLM]] == F F ! temperature, salinity, inert</div>

== Vertical Coordinates Parameters ==

* Set vertical, terrain-following coordinates transformation equation and stretching function (see [[Vertical S-coordinate]] for more details).<div class="box"> [[Variables#Vtransform|Vtransform]] == 2 ! transformation equation<br /> [[Variables#Vstretching|Vstretching]] == 4 ! stretching function</div>

* S-coordinate surface control parameter, [1:[[Ngrids]]] values are expected. The range of optimal values depends on the vertical stretching function.<div class="box"> [[Variables#theta_s|THETA_S]] == 3.0d0 ! surface stretching parameter</div>

* S-coordinate bottom control parameter, [1:[[Ngrids]]] values are expected. The range of optimal values depends on the vertical stretching function.<div class="box"> [[Variables#theta_b|THETA_B]] == 0.0d0 ! bottom stretching parameter</div>

* Critical depth (hc) in meters (positive) controlling the stretching. It can be interpreted as the width of surface or bottom boundary layer in which higher vertical resolution (levels) is required during stretching.<div class="box"> [[Variables#tcline|TCLINE]] == 25.0d0 ! critical depth (m)</div>

== Adjoint Sensitivity Parameters ==

* Starting ([[Variables#DstrS|DstrS]]) and ending ([[Variables#DendS|DendS]]) day for adjoint sensitivity forcing. [[Variables#DstrS|DstrS]] must be less or equal to [[Variables#DendS|DendS]]. If both values are zero, their values are reset internally to the full range of the adjoint integration.<div class="box"> [[Variables#DstrS|DstrS]] == 0.0d0 ! starting day<br /> [[Variables#DendS|DendS]] == 0.0d0 ! ending day</div>

* Starting and ending vertical levels of the 3D adjoint state variables whose sensitivity is required.<div class="box"> [[Variables#KstrS|KstrS]] == 1 ! starting level<br /> [[Variables#KendS|KendS]] == 1 ! ending level</div>

* Logical switches ('''TRUE'''/'''FALSE''') to specify the adjoint state variables whose sensitivity is required.<div class="box">[[Variables#Lstate|Lstate(isFsur)]] == F ! free-surface<br />[[Variables#Lstate|Lstate(isUbar)]] == F ! 2D U-momentum<br />[[Variables#Lstate|Lstate(isVbar)]] == F ! 2D V-momentum<br />[[Variables#Lstate|Lstate(isUvel)]] == F ! 3D U-momentum<br />[[Variables#Lstate|Lstate(isVvel)]] == F ! 3D V-momentum<br />[[Variables#Lstate|Lstate(isWvel)]] == F ! 3D W-momentum</div>

* Logical switches ('''TRUE'''/'''FALSE''') to specify the adjoint state tracer variables whose sensitivity is required, [1:[[NT]],1:[[Ngrids]]] values are expected.<div class="box">[[Variables#Lstate|Lstate(isTvar)]] == F F ! NT tracers</div>

== Stochastic Optimals Parameters ==

* Logical switches ('''TRUE'''/'''FALSE''') to specify the state variables required by Forcing Singular Vectors or Stochastic Optimals.<div class="box">[[Variables#Fstate|Fstate(isFsur)]] == F ! free-surface<br />[[Variables#Fstate|Fstate(isUbar)]] == F ! 2D U-momentum<br />[[Variables#Fstate|Fstate(isVbar)]] == F ! 2D V-momentum<br />[[Variables#Fstate|Fstate(isUvel)]] == F ! 3D U-momentum<br />[[Variables#Fstate|Fstate(isVvel)]] == F ! 3D V-momentum<br />[[Variables#Fstate|Fstate(isTvar)]] == F F ! NT tracers<br /><br />[[Variables#Fstate|Fstate(isUstr)]] == F ! surface U-stress<br />[[Variables#Fstate|Fstate(isVstr)]] == F ! surface V-stress<br />[[Variables#Fstate|Fstate(isTsur)]] == F F ! NT surface tracers flux</div>

* Stochastic optimals time decorrelation scale (days) assumed for red noise processes.<div class="box"> [[Variables#SO_decay|SO_decay]] == 2.0d0 ! days</div>

* Stochastic Optimals surface forcing standard deviation for dimensionalization.<div class="box">[[Variables#SO_sdev|SO_sdev(isFsur)]] == 1.0d0 ! free-surface<br />[[Variables#SO_sdev|SO_sdev(isUbar)]] == 1.0d0 ! 2D U-momentum<br />[[Variables#SO_sdev|SO_sdev(isVbar)]] == 1.0d0 ! 2D V-momentum<br />[[Variables#SO_sdev|SO_sdev(isUvel)]] == 1.0d0 ! 3D U-momentum<br />[[Variables#SO_sdev|SO_sdev(isVvel)]] == 1.0d0 ! 3D V-momentum<br />[[Variables#SO_sdev|SO_sdev(isTvar)]] == 1.0d0 1.0d0 ! NT tracers<br /><br />[[Variables#SOstate|SOstate(isUstr)]] == 1.0d0 ! surface u-stress<br />[[Variables#SOstate|SOstate(isVstr)]] == 1.0d0 ! surface v-stress<br />[[Variables#SO_sdev|SO_sdev(isTsur)]] == 1.0d0 1.0d0 ! NT surface tracer flux</div>

== History Output Variables Switches ==

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of fields into history output file.<div class="box">[[Variables#Hout|Hout(idUvel)]] == T ! u 3D U-velocity<br />[[Variables#Hout|Hout(idVvel)]] == T ! v 3D V-velocity<br />[[Variables#Hout|Hout(idu3dE)]] == F ! u_eastward 3D U-eastward at RHO-points<br />[[Variables#Hout|Hout(idv3dN)]] == F ! v_northward 3D V-northward at RHO-points<br />[[Variables#Hout|Hout(idWvel)]] == T ! w 3D W-velocity<br />[[Variables#Hout|Hout(idOvel)]] == T ! omega omega vertical velocity<br />[[Variables#Hout|Hout(idUbar)]] == T ! ubar 2D U-velocity<br />[[Variables#Hout|Hout(idVbar)]] == T ! vbar 2D V-velocity<br />[[Variables#Hout|Hout(idu2dE)]] == F ! ubar_eastward 2D U-eastward at RHO-points<br />[[Variables#Hout|Hout(idv2dN)]] == F ! vbar_northward 2D V-northward at RHO-points<br />[[Variables#Hout|Hout(idFsur)]] == T ! zeta free-surface<br />[[Variables#Hout|Hout(idBath)]] == T ! bath time-dependent bathymetry<br /><br />[[Variables#Hout|Hout(idTvar)]] == T T ! temp, salt temperature and salinity<br /><br />[[Variables#Hout|Hout(idpthR)]] == F ! z_rho time-varying depths of RHO-points<br />[[Variables#Hout|Hout(idpthU)]] == F ! z_u time-varying depths of U-points<br />[[Variables#Hout|Hout(idpthV)]] == F ! z_v time-varying depths of V-points<br />[[Variables#Hout|Hout(idpthW)]] == F ! z_w time-varying depths of W-points<br /><br />[[Variables#Hout|Hout(idUsms)]] == F ! sustr surface U-stress<br />[[Variables#Hout|Hout(idVsms)]] == F ! svstr surface V-stress<br />[[Variables#Hout|Hout(idUbms)]] == F ! bustr bottom U-stress<br />[[Variables#Hout|Hout(idVbms)]] == F ! bvstr bottom V-stress<br /><br />[[Variables#Hout|Hout(idUbrs)]] == F ! bustrc bottom U-current stress<br />[[Variables#Hout|Hout(idVbrs)]] == F ! bvstrc bottom V-current stress<br />[[Variables#Hout|Hout(idUbws)]] == F ! bustrw bottom U-wave stress<br />[[Variables#Hout|Hout(idVbws)]] == F ! bvstrw bottom V-wave stress<br />[[Variables#Hout|Hout(idUbcs)]] == F ! bustrcwmax bottom max wave-current U-stress<br />[[Variables#Hout|Hout(idVbcs)]] == F ! bvstrcwmax bottom max wave-current V-stress<br /><br />[[Variables#Hout|Hout(idUbot)]] == F ! Ubot bed wave orbital U-velocity<br />[[Variables#Hout|Hout(idVbot)]] == F ! Vbot bed wave orbital V-velocity<br />[[Variables#Hout|Hout(idUbur)]] == F ! Ur bottom U-velocity above bed<br />[[Variables#Hout|Hout(idVbvr)]] == F ! Vr bottom V-velocity above bed<br /><br />[[Variables#Hout|Hout(idW2xx)]] == F ! Sxx_bar 2D radiation stress, Sxx component<br />[[Variables#Hout|Hout(idW2xy)]] == F ! Sxy_bar 2D radiation stress, Sxy component<br />[[Variables#Hout|Hout(idW2yy)]] == F ! Syy_bar 2D radiation stress, Syy component<br />[[Variables#Hout|Hout(idU2rs)]] == F ! Ubar_Rstress 2D radiation U-stress<br />[[Variables#Hout|Hout(idV2rs)]] == F ! Vbar_Rstress 2D radiation V-stress<br />[[Variables#Hout|Hout(idU2Sd)]] == F ! ubar_stokes 2D U-Stokes velocity<br />[[Variables#Hout|Hout(idV2Sd)]] == F ! vbar_stokes 2D V-Stokes velocity<br /><br />[[Variables#Hout|Hout(idW3xx)]] == F ! Sxx 3D radiation stress, Sxx component<br />[[Variables#Hout|Hout(idW3xy)]] == F ! Sxy 3D radiation stress, Sxy component<br />[[Variables#Hout|Hout(idW3yy)]] == F ! Syy 3D radiation stress, Syy component<br />[[Variables#Hout|Hout(idW3zx)]] == F ! Szx 3D radiation stress, Szx component<br />[[Variables#Hout|Hout(idW3zy)]] == F ! Szy 3D radiation stress, Szy component<br />[[Variables#Hout|Hout(idU3rs)]] == F ! u_Rstress 3D U-radiation stress<br />[[Variables#Hout|Hout(idV3rs)]] == F ! v_Rstress 3D V-radiation stress<br />[[Variables#Hout|Hout(idU3Sd)]] == F ! u_stokes 3D U-Stokes velocity<br />[[Variables#Hout|Hout(idV3Sd)]] == F ! v_stokes 3D V-Stokes velocity<br /><br />[[Variables#Hout|Hout(idWamp)]] == F ! Hwave wave height<br />[[Variables#Hout|Hout(idWlen)]] == F ! Lwave wave length<br />[[Variables#Hout|Hout(idWdir)]] == F ! Dwave wave direction<br />[[Variables#Hout|Hout(idWptp)]] == F ! Pwave_top wave surface period<br />[[Variables#Hout|Hout(idWpbt)]] == F ! Pwave_bot wave bottom period<br />[[Variables#Hout|Hout(idWorb)]] == F ! Ub_swan wave bottom orbital velocity<br />[[Variables#Hout|Hout(idWdis)]] == F ! Wave_dissip wave dissipation<br /><br />[[Variables#Hout|Hout(idPair)]] == F ! Pair surface air pressure<br />[[Variables#Hout|Hout(idUair)]] == F ! Uair surface U-wind component<br />[[Variables#Hout|Hout(idVair)]] == F ! Vair surface V-wind component<br /><br />[[Variables#Hout|Hout(idTsur)]] == F F ! shflux, ssflux surface net heat and salt flux<br />[[Variables#Hout|Hout(idLhea)]] == F ! latent latent heat flux<br />[[Variables#Hout|Hout(idShea)]] == F ! sensible sensible heat flux<br />[[Variables#Hout|Hout(idLrad)]] == F ! lwrad longwave radiation flux<br />[[Variables#Hout|Hout(idSrad)]] == F ! swrad shortwave radiation flux<br />[[Variables#Hout|Hout(idEmPf)]] == F ! EminusP E-P flux <br />[[Variables#Hout|Hout(idevap)]] == F ! evaporation evaporation rate<br />[[Variables#Hout|Hout(idrain)]] == F ! rain precipitation rate<br /><br />[[Variables#Hout|Hout(idDano)]] == F ! rho density anomaly<br />[[Variables#Hout|Hout(idVvis)]] == F ! AKv vertical viscosity<br />[[Variables#Hout|Hout(idTdif)]] == F ! AKt vertical T-diffusion<br />[[Variables#Hout|Hout(idSdif)]] == F ! AKs vertical Salinity diffusion<br />[[Variables#Hout|Hout(idHsbl)]] == F ! Hsbl depth of surface boundary layer<br />[[Variables#Hout|Hout(idHbbl)]] == F ! Hbbl depth of bottom boundary layer<br />[[Variables#Hout|Hout(idMtke)]] == F ! tke turbulent kinetic energy<br />[[Variables#Hout|Hout(idMtls)]] == F ! gls turbulent length scale</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of extra inert passive tracers other than biological and sediment tracers. An inert passive tracer is one that it is only advected and diffused. Other processes are ignored. These tracers include, for example, dyes, pollutants, oil spills, etc. [1:[[NPT]]] values are expected. However, these switches can be activated using compact parameter specification.<div class="box">[[Variables#Hout|Hout(inert)]] == T ! dye_01, ... inert passive tracers</div>



== Quicksave Output Variables Switches ==

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of fields into quicksave output file.<div class="box">[[Variables#Qout|Qout(idUvel)]] == F ! u 3D U-velocity<br />[[Variables#Qout|Qout(idVvel)]] == F ! v 3D V-velocity<br />[[Variables#Qout|Qout(idu3dE)]] == F ! u_eastward 3D U-eastward at RHO-points<br />[[Variables#Qout|Qout(idv3dN)]] == F ! v_northward 3D V-northward at RHO-points<br />[[Variables#Qout|Qout(idWvel)]] == F ! w 3D W-velocity<br />[[Variables#Qout|Qout(idOvel)]] == F ! omega omega vertical velocity<br />[[Variables#Qout|Qout(idUbar)]] == T ! ubar 2D U-velocity<br />[[Variables#Qout|Qout(idVbar)]] == T ! vbar 2D V-velocity<br />[[Variables#Qout|Qout(idu2dE)]] == T ! ubar_eastward 2D U-eastward at RHO-points<br />[[Variables#Qout|Qout(idv2dN)]] == T ! vbar_northward 2D V-northward at RHO-points<br />[[Variables#Qout|Qout(idFsur)]] == T ! zeta free-surface<br />[[Variables#Qout|Qout(idBath)]] == T ! bath time-dependent bathymetry<br /><br />[[Variables#Qout|Qout(idTvar)]] == F F ! temp, salt temperature and salinity<br /><br />[[Variables#Qout|Qout(idUsur)]] == T ! u_sur surface U-velocity<br />[[Variables#Qout|Qout(idVsur)]] == T ! v_sur surface V-velocity<br />[[Variables#Qout|Qout(idUsuE)]] == T ! u_sur_eastward surface U-eastward velocity<br />[[Variables#Qout|Qout(idVsuN)]] == T ! v_sur_northward surface V-northward velocity<br /><br />[[Variables#Qout|Qout(idsurT)]] == T T ! temp_sur, salt_sur surface temperature and salinity<br /><br />[[Variables#Qout|Qout(idpthR)]] == F ! z_rho time-varying depths of RHO-points<br />[[Variables#Qout|Qout(idpthU)]] == F ! z_u time-varying depths of U-points<br />[[Variables#Qout|Qout(idpthV)]] == F ! z_v time-varying depths of V-points<br />[[Variables#Qout|Qout(idpthW)]] == F ! z_w time-varying depths of W-points<br /><br />[[Variables#Qout|Qout(idUsms)]] == F ! sustr surface U-stress<br />[[Variables#Qout|Qout(idVsms)]] == F ! svstr surface V-stress<br />[[Variables#Qout|Qout(idUbms)]] == F ! bustr bottom U-stress<br />[[Variables#Qout|Qout(idVbms)]] == F ! bvstr bottom V-stress<br /><br />[[Variables#Qout|Qout(idUbrs)]] == F ! bustrc bottom U-current stress<br />[[Variables#Qout|Qout(idVbrs)]] == F ! bvstrc bottom V-current stress<br />[[Variables#Qout|Qout(idUbws)]] == F ! bustrw bottom U-wave stress<br />[[Variables#Qout|Qout(idVbws)]] == F ! bvstrw bottom V-wave stress<br />[[Variables#Qout|Qout(idUbcs)]] == F ! bustrcwmax bottom max wave-current U-stress<br />[[Variables#Qout|Qout(idVbcs)]] == F ! bvstrcwmax bottom max wave-current V-stress<br /><br />[[Variables#Qout|Qout(idUbot)]] == F ! Ubot bed wave orbital U-velocity<br />[[Variables#Qout|Qout(idVbot)]] == F ! Vbot bed wave orbital V-velocity<br />[[Variables#Qout|Qout(idUbur)]] == F ! Ur bottom U-velocity above bed<br />[[Variables#Qout|Qout(idVbvr)]] == F ! Vr bottom V-velocity above bed<br /><br />[[Variables#Qout|Qout(idW2xx)]] == F ! Sxx_bar 2D radiation stress, Sxx component<br />[[Variables#Qout|Qout(idW2xy)]] == F ! Sxy_bar 2D radiation stress, Sxy component<br />[[Variables#Qout|Qout(idW2yy)]] == F ! Syy_bar 2D radiation stress, Syy component<br />[[Variables#Qout|Qout(idU2rs)]] == F ! Ubar_Rstress 2D radiation U-stress<br />[[Variables#Qout|Qout(idV2rs)]] == F ! Vbar_Rstress 2D radiation V-stress<br />[[Variables#Qout|Qout(idU2Sd)]] == F ! ubar_stokes 2D U-Stokes velocity<br />[[Variables#Qout|Qout(idV2Sd)]] == F ! vbar_stokes 2D V-Stokes velocity<br /><br />[[Variables#Qout|Qout(idW3xx)]] == F ! Sxx 3D radiation stress, Sxx component<br />[[Variables#Qout|Qout(idW3xy)]] == F ! Sxy 3D radiation stress, Sxy component<br />[[Variables#Qout|Qout(idW3yy)]] == F ! Syy 3D radiation stress, Syy component<br />[[Variables#Qout|Qout(idW3zx)]] == F ! Szx 3D radiation stress, Szx component<br />[[Variables#Qout|Qout(idW3zy)]] == F ! Szy 3D radiation stress, Szy component<br />[[Variables#Qout|Qout(idU3rs)]] == F ! u_Rstress 3D U-radiation stress<br />[[Variables#Qout|Qout(idV3rs)]] == F ! v_Rstress 3D V-radiation stress<br />[[Variables#Qout|Qout(idU3Sd)]] == F ! u_stokes 3D U-Stokes velocity<br />[[Variables#Qout|Qout(idV3Sd)]] == F ! v_stokes 3D V-Stokes velocity<br /><br />[[Variables#Qout|Qout(idWamp)]] == F ! Hwave wave height<br />[[Variables#Qout|Qout(idWlen)]] == F ! Lwave wave length<br />[[Variables#Qout|Qout(idWdir)]] == F ! Dwave wave direction<br />[[Variables#Qout|Qout(idWptp)]] == F ! Pwave_top wave surface period<br />[[Variables#Qout|Qout(idWpbt)]] == F ! Pwave_bot wave bottom period<br />[[Variables#Qout|Qout(idWorb)]] == F ! Ub_swan wave bottom orbital velocity<br />[[Variables#Qout|Qout(idWdis)]] == F ! Wave_dissip wave dissipation<br /><br />[[Variables#Qout|Qout(idPair)]] == F ! Pair surface air pressure<br />[[Variables#Qout|Qout(idUair)]] == F ! Uair surface U-wind component<br />[[Variables#Qout|Qout(idVair)]] == F ! Vair surface V-wind component<br /><br />[[Variables#Qout|Qout(idTsur)]] == F F ! shflux, ssflux surface net heat and salt flux<br />[[Variables#Qout|Qout(idLhea)]] == F ! latent latent heat flux<br />[[Variables#Qout|Qout(idShea)]] == F ! sensible sensible heat flux<br />[[Variables#Qout|Qout(idLrad)]] == F ! lwrad longwave radiation flux<br />[[Variables#Qout|Qout(idSrad)]] == F ! swrad shortwave radiation flux<br />[[Variables#Qout|Qout(idEmPf)]] == F ! EminusP E-P flux <br />[[Variables#Qout|Qout(idevap)]] == F ! evaporation evaporation rate<br />[[Variables#Qout|Qout(idrain)]] == F ! rain precipitation rate<br /><br />[[Variables#Qout|Qout(idDano)]] == F ! rho density anomaly<br />[[Variables#Qout|Qout(idVvis)]] == F ! AKv vertical viscosity<br />[[Variables#Qout|Qout(idTdif)]] == F ! AKt vertical T-diffusion<br />[[Variables#Qout|Qout(idSdif)]] == F ! AKs vertical Salinity diffusion<br />[[Variables#Qout|Qout(idHsbl)]] == F ! Hsbl depth of surface boundary layer<br />[[Variables#Qout|Qout(idHbbl)]] == F ! Hbbl depth of bottom boundary layer<br />[[Variables#Qout|Qout(idMtke)]] == F ! tke turbulent kinetic energy<br />[[Variables#Qout|Qout(idMtls)]] == F ! gls turbulent length scale</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of extra inert passive tracers other than biological and sediment tracers into the quicksave output file. An inert passive tracer is one that it is only advected and diffused. Other processes are ignored. These tracers include, for example, dyes, pollutants, oil spills, etc. [1:[[NPT]]] values are expected. However, these switches can be activated using compact parameter specification.<div class="box">[[Variables#Qout|Qout(inert)]] == F ! dye_01, ... inert passive tracers<br />[[Variables#Qout|Qout(Snert)]] == F ! dye_01, ... surface inert passive tracers</div>

== Time-averaged Output Variables Switches ==

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of fields into time-averaged output file.<div class="box">[[Variables#Aout|Aout(idUvel)]] == T ! u 3D U-velocityy<br />[[Variables#Aout|Aout(idVvel)]] == T ! v 3D V-velocity<br />[[Variables#Aout|Aout(idu3dE)]] == F ! u_eastward 3D U-eastward at RHO-points<br />[[Variables#Aout|Aout(idv3dN)]] == F ! v_northward 3D V-northward at RHO-points<br />[[Variables#Aout|Aout(idWvel)]] == T ! w 3D W-velocity<br />[[Variables#Aout|Aout(idOvel)]] == T ! omega omega vertical velocity<br />[[Variables#Aout|Aout(idUbar)]] == T ! ubar 2D U-velocity<br />[[Variables#Aout|Aout(idVbar)]] == T ! vbar 2D V-velocity<br />[[Variables#Aout|Aout(idu2dE)]] == F ! ubar_eastward 2D U-eastward at RHO-points<br />[[Variables#Aout|Aout(idv2dN)]] == F ! vbar_northward 2D V-northward at RHO-points<br />[[Variables#Aout|Aout(idFsur)]] == T ! zeta free-surface<br />[[Variables#Aout|Aout(idTvar)]] == T T ! temp, salt temperature and salinity<br /><br />[[Variables#Aout|Aout(idUsms)]] == F ! sustr surface U-stress<br />[[Variables#Aout|Aout(idVsms)]] == F ! svstr surface V-stress<br />[[Variables#Aout|Aout(idUbms)]] == F ! bustr bottom U-stress<br />[[Variables#Aout|Aout(idVbms)]] == F ! bvstr bottom V-stress<br /><br />[[Variables#Aout|Aout(idW2xx)]] == F ! Sxx_bar 2D radiation stress, Sxx component<br />[[Variables#Aout|Aout(idW2xy)]] == F ! Sxy_bar 2D radiation stress, Sxy component<br />[[Variables#Aout|Aout(idW2yy)]] == F ! Syy_bar 2D radiation stress, Syy component<br />[[Variables#Aout|Aout(idU2rs)]] == F ! Ubar_Rstress 2D radiation U-stress<br />[[Variables#Aout|Aout(idV2rs)]] == F ! Vbar_Rstress 2D radiation V-stress<br />[[Variables#Aout|Aout(idU2Sd)]] == F ! ubar_stokes 2D U-Stokes velocity<br />[[Variables#Aout|Aout(idV2Sd)]] == F ! vbar_stokes 2D V-Stokes velocity<br /><br />[[Variables#Aout|Aout(idW3xx)]] == F ! Sxx 3D radiation stress, Sxx component<br />[[Variables#Aout|Aout(idW3xy)]] == F ! Sxy 3D radiation stress, Sxy component<br />[[Variables#Aout|Aout(idW3yy)]] == F ! Syy 3D radiation stress, Syy component<br />[[Variables#Aout|Aout(idW3zx)]] == F ! Szx 3D radiation stress, Szx component<br />[[Variables#Aout|Aout(idW3zy)]] == F ! Szy 3D radiation stress, Szy component<br />[[Variables#Aout|Aout(idU3rs)]] == F ! u_Rstress 3D U-radiation stress<br />[[Variables#Aout|Aout(idV3rs)]] == F ! v_Rstress 3D V-radiation stress<br />[[Variables#Aout|Aout(idU3Sd)]] == F ! u_stokes 3D U-Stokes velocity<br />[[Variables#Aout|Aout(idV3Sd)]] == F ! v_stokes 3D V-Stokes velocity<br /><br />[[Variables#Aout|Aout(idPair)]] == F ! Pair surface air pressure<br />[[Variables#Aout|Aout(idUair)]] == F ! Uair surface U-wind component<br />[[Variables#Aout|Aout(idVair)]] == F ! Vair surface V-wind component<br /><br />[[Variables#Aout|Aout(idTsur)]] == F F ! shflux, ssflux surface net heat and salt flux<br />[[Variables#Aout|Aout(idLhea)]] == F ! latent latent heat flux<br />[[Variables#Aout|Aout(idShea)]] == F ! sensible sensible heat flux<br />[[Variables#Aout|Aout(idLrad)]] == F ! lwrad longwave radiation flux<br />[[Variables#Aout|Aout(idSrad)]] == F ! swrad shortwave radiation flux<br />[[Variables#Aout|Aout(idevap)]] == F ! evaporation evaporation rate<br />[[Variables#Aout|Aout(idrain)]] == F ! rain precipitation rate<br /><br />[[Variables#Aout|Aout(idDano)]] == F ! rho density anomaly<br />[[Variables#Aout|Aout(idVvis)]] == F ! AKv vertical viscosity<br />[[Variables#Aout|Aout(idTdif)]] == F ! AKt vertical T-diffusion<br />[[Variables#Aout|Aout(idSdif)]] == F ! AKs vertical Salinity diffusion<br />[[Variables#Aout|Aout(idHsbl)]] == F ! Hsbl depth of surface boundary layer<br />[[Variables#Aout|Aout(idHbbl)]] == F ! Hbbl depth of bottom boundary layer<br /><br />[[Variables#Aout|Aout(id2dRV)]] == F ! pvorticity_bar 2D relative vorticity<br />[[Variables#Aout|Aout(id3dRV)]] == F ! pvorticity 3D relative vorticity<br />[[Variables#Aout|Aout(id2dPV)]] == F ! rvorticity_bar 2D potential vorticity<br />[[Variables#Aout|Aout(id3dPV)]] == F ! rvorticity 3D potential vorticity<br /><br />[[Variables#Aout|Aout(idu3dD)]] == F ! u_detided detided 3D U-velocity<br />[[Variables#Aout|Aout(idv3dD)]] == F ! v_detided detided 3D V-velocity<br />[[Variables#Aout|Aout(idu2dD)]] == F ! ubar_detided detided 2D U-velocity<br />[[Variables#Aout|Aout(idu3dD)]] == F ! vbar_detided detided 2D V-velocity<br />[[Variables#Aout|Aout(idFsuD)]] == F ! zeta_detided detided free-surface<br /><br />[[Variables#Aout|Aout(idTrcD)]] == F F ! temp_detided, ... detided temperature and salinity<br /><br />[[Variables#Aout|Aout(idHUav)]] == F ! Huon u-volume flux, Huon<br />[[Variables#Aout|Aout(idHVav)]] == F ! Hvom v-volume flux, Hvom<br />[[Variables#Aout|Aout(idUUav)]] == F ! uu quadratic <u*u> term<br />[[Variables#Aout|Aout(idUVav)]] == F ! uv quadratic <u*v> term<br />[[Variables#Aout|Aout(idVVav)]] == F ! vv quadratic <v*v> term<br />[[Variables#Aout|Aout(idU2av)]] == F ! ubar2 quadratic <ubar*ubar> term<br />[[Variables#Aout|Aout(idV2av)]] == F ! vbar2 quadratic <vbar*vbar> term<br />[[Variables#Aout|Aout(idZZav)]] == F ! zeta2 quadratic <zeta*zeta> term<br /><br />[[Variables#Aout|Aout(idTTav)]] == F F ! temp2, ... quadratic <t*t> tracer terms<br />[[Variables#Aout|Aout(idUTav)]] == F F ! utemp, ... quadratic <u*t> tracer terms<br />[[Variables#Aout|Aout(idVTav)]] == F F ! vtemp, ... quadratic <v*t> tracer terms<br />[[Variables#Aout|Aout(iHUTav)]] == F F ! Huontemp, ... tracer volume flux, <Huon*t><br />[[Variables#Aout|Aout(iHVTav)]] == F F ! Hvomtemp, ... tracer volume flux, <Hvom*t></div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of extra inert passive tracers other than biological and sediment tracers into the time-averaged output file. An inert passive tracer is one that it is only advected and diffused. Other processes are ignored. These tracers include, for example, dyes, pollutants, oil spills, etc. [1:[[NPT]],1:[[Ngrids]]] values are expected. However, these switches can be activated using compact parameter specification.<div class="box">[[Variables#Aout|Aout(inert)]] == T ! dye_01, ... inert passive tracers</div>

== Time-averaged Diagnostic Output Variables Switches ==

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing time-averaged. 2D momentum (ubar, vbar) diagnostic terms into the diagnostics output file.<div class="box">[[Variables#Dout|Dout(M2rate)]] == T ! ubar_accel, ... acceleration<br />[[Variables#Dout|Dout(M2pgrd)]] == T ! ubar_prsgrd, ... pressure gradient<br />[[Variables#Dout|Dout(M2fcor)]] == T ! ubar_cor, ... Coriolis force<br />[[Variables#Dout|Dout(M2hadv)]] == T ! ubar_hadv, ... horizontal total advection<br />[[Variables#Dout|Dout(M2xadv)]] == T ! ubar_xadv, ... horizontal XI-advection<br />[[Variables#Dout|Dout(M2yadv)]] == T ! ubar_yadv, ... horizontal ETA-advection<br />[[Variables#Dout|Dout(M2hrad)]] == T ! ubar_hrad, ... horizontal total radiation stress<br />[[Variables#Dout|Dout(M2hvis)]] == T ! ubar_hvisc, ... horizontal total viscosity<br />[[Variables#Dout|Dout(M2xvis)]] == T ! ubar_xvisc, ... horizontal XI-viscosity<br />[[Variables#Dout|Dout(M2yvis)]] == T ! ubar_yvisc, ... horizontal ETA-viscosity<br />[[Variables#Dout|Dout(M2sstr)]] == T ! ubar_sstr, ... surface stress<br />[[Variables#Dout|Dout(M2bstr)]] == T ! ubar_bstr, ... bottom stress</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of time-averaged, 3D momentum (u,v) diagnostic terms into the diagnostics output file. <div class="box">[[Variables#Dout|Dout(M3rate)]] == T ! u_accel, ... acceleration<br />[[Variables#Dout|Dout(M3pgrd)]] == T ! u_prsgrd, ... pressure gradient<br />[[Variables#Dout|Dout(M3fcor)]] == T ! u_cor, ... Coriolis force<br />[[Variables#Dout|Dout(M3hadv)]] == T ! u_hadv, ... horizontal total advection<br />[[Variables#Dout|Dout(M3xadv)]] == T ! u_xadv, ... horizontal XI-advection<br />[[Variables#Dout|Dout(M3yadv)]] == T ! u_yadv, ... horizontal ETA-advection<br />[[Variables#Dout|Dout(M3vadv)]] == T ! u_vadv, ... vertical advection<br />[[Variables#Dout|Dout(M3hrad)]] == T ! u_hrad, ... horizontal total radiation stress<br />[[Variables#Dout|Dout(M3vrad)]] == T ! u_vrad, ... vertical radiation stress<br />[[Variables#Dout|Dout(M3hvis)]] == T ! u_hvisc, ... horizontal total viscosity<br />[[Variables#Dout|Dout(M3xvis)]] == T ! u_xvisc, ... horizontal XI-viscosity<br />[[Variables#Dout|Dout(M3yvis)]] == T ! u_yvisc, ... horizontal ETA-viscosity<br />[[Variables#Dout|Dout(M3vvis)]] == T ! u_vvisc, ... vertical viscosity</div>

* Logical switches ('''TRUE'''/'''FALSE''') to activate writing of time-averaged, active (temperature and salinity) and passive (inert) tracer diagnostic terms into the diagnostics output file. [1:[[NAT]]+[[NPT]],1:[[Ngrids]]] values are expected.<div class="box">[[Variables#Dout|Dout(iTrate)]] == T T ! temp_rate, ... time rate of change<br />[[Variables#Dout|Dout(iThadv)]] == T T ! temp_hadv, ... horizontal total advection<br />[[Variables#Dout|Dout(iTxadv)]] == T T ! temp_xadv, ... horizontal XI-advection<br />[[Variables#Dout|Dout(iTyadv)]] == T T ! temp_yadv, ... horizontal ETA-advection<br />[[Variables#Dout|Dout(iTvadv)]] == T T ! temp_vadv, ... vertical advection<br />[[Variables#Dout|Dout(iThdif)]] == T T ! temp_hdiff, ... horizontal total diffusion<br />[[Variables#Dout|Dout(iTxdif)]] == T T ! temp_xdiff, ... horizontal XI-diffusion<br />[[Variables#Dout|Dout(iTydif)]] == T T ! temp_ydiff, ... horizontal ETA-diffusion<br />[[Variables#Dout|Dout(iTsdif)]] == T T ! temp_sdiff, ... horizontal S-diffusion<br />[[Variables#Dout|Dout(iTvdif)]] == T T ! temp_vdiff, ... vertical diffusion</div>

== Generic User Parameters ==
<section begin=user />
* [[Variables#Nuser|NUSER]] is the number (integer) of user parameters to consider. [[Variables#user|USER]] is a vector containing [[Variables#Nuser|NUSER]] user parameters (real array).<div class="box"> [[Variables#Nuser|NUSER]] = 0<br /> [[Variables#user|USER]] = 0.d0</div>This array is primarily used with the [[Options#SANITY_CHECK|SANITY_CHECK]] to test the correctness of the tangent linear adjoint models. It contains the model variable and grid point to perturb:<div class="box">! INT([[Variables#user|user]](1)): tangent state variable to perturb<br>! INT([[Variables#user|user]](2)): adjoint state variable to perturb<br>! [ [[Variables#isFsur|isFsur]] = 1 ] free-surface<br>! [ [[Variables#isUbar|isUbar]] = 2 ] 2D U-momentum<br>! [ [[Variables#isVbar|isVbar]] = 3 ] 2D V-momentum<br>! [ [[Variables#isUvel|isUvel]] = 4 ] 3D U-momentum<br>! [ [[Variables#isVvel|isVvel]] = 5 ] 3D V-momentum<br>! [ [[Variables#isTvar|isTvar]] = 6 ] First tracer (temperature)<br>! [ ... ] ...<br>! [ [[Variables#isTvar|isTvar]] = ? ] Last tracer<br>! <br>! INT([[Variables#user|user]](3)): I-index of tangent variable to perturb<br>! INT([[Variables#user|user]](4)): I-index of adjoint variable to perturb<br>! INT([[Variables#user|user]](5)): J-index of tangent variable to perturb<br>! INT([[Variables#user|user]](6)): J-index of adjoint variable to perturb<br>! INT([[Variables#user|user]](7)): K-index of tangent variable to perturb, if 3D<br>! INT([[Variables#user|user]](8)): K-index of adjoint variable to perturb, if 3D</div>Set tangent and adjoint parameters to the same values if perturbing and reporting the same variable.

* This parameter could also be used to adjust constants in analytical functions at run time.
<section end=user />

==Parallel I/O (PIO) Parameters==
<section begin=PIO />
* Choose the input and output NetCDF library to use. For example, the user could choose to use the PIO library for writing but still use the standard library for reading. <span class="twilightBlue">To use this Parallel I/O strategy, the '''PIO''' library must be linked to ROMS at compile time and the [[PIO_LIB]] CPP option needs to be activated. It is only available in distributed-memory applications since it uses MPI-IO.</span> <div class="box">! [1] Standard NetCDF-3 or NetCDF-4 library<br />! [2] Parallel-IO from the PIO library (MPI, MPI-IO applications)<br /><br /> [[Variables#INP_LIB|INP_LIB]] = 2<br /> [[Variables#OUT_LIB|OUT_LIB]] = 2</div>

* PIO offers several methods for reading/writing NetCDF files. Depending on the build of the PIO libraries, not all the I/O types are available. If the NetCDF library does not support parallel I/O, methods 3 and 4 are not available. Currently, NetCDF4/HDF5 data compression is possible with method 3 during serial write.<div class="box">! [0] parallel read and parallel write of PnetCDF (CDF-5 type files, not recommended because of post-processing)<br />! [1] parallel read and parallel write of NetCDF3 (64-bit offset)<br />! [2] serial read and serial write of NetCDF3 (64-bit offset)<br />! [3] parallel read and serial write of NetCDF4/HDF5<br />! [4] parallel read and parallel write of NETCDF4/HDF5<br /><br /> [[Variables#PIO_METHOD|PIO_METHOD]] = 2</div>

* Parallel-IO tasks control parameters. Typically, it is advantageous and highly recommended to define the I/O decomposition in smaller number of processes for efficiency and to avoid MPI communications bottlenecks.<div class="box"> [[Variables#PIO_IOTASKS|PIO_IOTASKS]] = 1 ! number of I/O processes to define<br /> [[Variables#PIO_STRIDE|PIO_STRIDE]] = 1 ! stride in the MPI-rank between I/O processes<br /> [[Variables#PIO_BASE|PIO_BASE]] = 0 ! offset for the first I/O process<br /> [[Variables#PIO_AGGREG|PIO_AGGREG]] = 1 ! number of MPI-aggregators to use</div>

* Parallel-IO rearranger methods for moving data between computational and I/O processes. It provides the ability to rearrange data between computational and parallel I/O decompositions. Usually the Box rearrangement is more efficient.<div class="box">! [1] Box rearrangement<br />! [2] Subset rearrangement<br /><br /> [[Variables#PIO_REARR|PIO_REARR]] = 1</div>
**In the '''box''' method, data is rearranged from computational to I/O processes in a continuous manner to the data ordering in the file. Since the ordering of data between computational and I/O partitions may be different, the rearrangement will require all-to-all MPI communications. Also, notice that each computing tile may transfer data to one or more I/O processes.
**In the '''subset''' method, each I/O process is associated with a subset of computing processes. The computing tile sends its data to a unique I/O process. The data on I/O processes may be more fragmented to the ordering on disk, which may increase the communications to the storage medium. However, the rearrangement scales better since all-to-all MPI communications are not required.

* Parallel-IO rearranger flag for MPI communication between computational and I/O processes. In some systems, the Point-to-Point communications is more efficient. <div class="box">! [0] Point-to-Point communications<br />! [1] Collective communications<br /><br />[[Variables#PIO_REARRCOM|PIO_REARRCOM]] = 0</div>

* Parallel-IO rearranger flow-control direction flag for MPI communications between computational and I/O processes. The flow algorithm controls the rate and volume of messages sent to any destination MPI process. Optimally, the MPI communications should be designed to send a modest number of messages evenly distributed across a number of processes. An excessive number of messages to a single MPI process can exhaust the buffer space which can affect efficiency or failure due to the slowdown in the retransmitting of dropped messages. It only sends messages (Isend) when the receiver is ready and has sufficient resources.<div class="box">! [0] Enable computational to I/O processes, and vice versa<br />! [1] Enable computational to I/O processes only<br />! [2] Enable I/O to computational processes only<br />! [3] Disable flow control<br /><br />[[Variables#PIO_REARRDIR|PIO_REARRDIR]] = 0</div>

* Parallel-IO rearranger options for MPI communications from computational to I/O processes (C2I). <div class="box"> [[Variables#PIO_C2I_HS|PIO_C2I_HS]] = T ! Enable C2I handshake (T/F)<br />[[Variables#PIO_C2I_Send|PIO_C2I_Send]] = F ! Enable C2I Isends (T/F)<br /> [[Variables#PIO_I2C_HS|PIO_I2C_HS]] = 64 ! Maximum pending C2I requests</div>

* Parallel-IO rearranger options for MPI communications from I/O to computational processes (I2C). <div class="box"> [[Variables#PIO_I2C_HS|PIO_I2C_HS]] = F ! Enable I2C handshake (T/F)<br />[[Variables#PIO_I2C_Send|PIO_I2C_Send]] = T ! Enable I2C Isends (T/F)<br />[[Variables#PIO_I2C_Preq|PIO_I2C_Preq]] = 65 ! Maximum pending I2C requests</div><section end=PIO />

==NetCDF-4/HDF5 Compression Parameters==

* NetCDF-4/HDF5 compression parameters for output files. This capability is used when both [[Options#HDF5|HDF5]] and [[Options#DEFLATE|DEFLATE]] C-preprocessing options are activated. The user needs to compile with the NetCDF-4/HDF5 and MPI libraries. File deflation cannot be used in parallel I/O for writing libraries. File deflation cannot be used in parallel I/O for writing to exactly map the data to the disk location. For more information, check [http://www.unidata.ucar.edu/software/netcdf NetCDF official website].<div class="box"> [[Variables#shuffle|NC_SHUFFLE]] = 1 ! if non-zero, turn on shuffle filter<br> [[Variables#deflate|NC_DEFLATE]] = 1 ! if non-zero, turn on deflate filter<br> [[Variables#dlevel|NC_DLEVEL]] = 1 ! deflate level [0-9]</div>

== Input NetCDF Files ==

{{note}}'''NOTE:''' Starting with '''revision 460''' file names can be up to 256 characters long. Previously only 80 characters were allowed.

* Input NetCDF file names, [1:Ngrids] values are expected.<div class="box"> [[Variables#GRD|GRDNAME]] == roms_grd.nc ! Grid<br /> [[Variables#INI|ININAME]] == roms_ini.nc ! NLM initial conditions<br /> [[Variables#ITL|ITLNAME]] == roms_itl.nc ! TLM initial conditions<br /> [[Variables#IRP|IRPNAME]] == roms_irp.nc ! RPM initial conditions<br /> [[Variables#IAD|IADNAME]] == roms_iad.nc ! ADM initial conditions<br /> [[Variables#FWD|FWDNAME]] == roms_fwd.nc ! Forward trajectory<br /> [[Variables#ADS|ADSNAME]] == roms_ads.nc ! Adjoint sensitivity functionals</div>

* Input adjoint forcing NetCDF filenames for computing observations impacts during the analysis-forecast cycle. If the forecast error metric is defined in state-space, then [[Variables#FOIA|FOInameA]] and [[Variables#FOIB|FOInameB]] should be regular adjoint forcing files just like [[Variables#ADS|ADSNAME]]. If the forecast error metric is defined in observation space ([[Options#OBS_SPACE|OBS_SPACE]] is activated) then the forecast is initialized [[Variables#OIFA|OIFnameA]] and [[Variables#OIFB|OIFnameB]] (specified in [[s4dvar.in]] input script) will have the structure of a 4D-Var observation file.<div class="box"> [[Variables#FOIA|FOInameA]] == roms_foi_a.nc<br /> [[Variables#FOIB|FOInameB]] == roms_foi_b.nc</div>

* Input NetCDF filenames for the forecasts initialized from the analysis of the current 4D-Var cycle ([[Variables#FCTA|FCTnameA]]) and initialized from the analysis of the previous 4D-Var cycle ([[Variables#FCTB|FCTnameB]]).<div class="box"> [[Variables#FCTA|FCTnameA]] == roms_fct_a.nc<br /> [[Variables#FCTB|FCTnameB]] == roms_fct_b.nc</div>

* Nesting grids connectivity data: contact points information. This NetCDF file is special and complex. It is currently generated using the script '''matlab/grid/contact.m''' from the Matlab repository.<div class="box"> [[Variables#NGCname|NGCNAME]] = roms_ngc.nc</div>

* Input lateral boundary conditions and climatology file names. The '''user''' has the option to split input data time records into several NetCDF files (see the [[Input_Parameter_Files#Important_Notes_on_File_Syntax|File Syntax Notes]]). If so, use a single line per entry with a vertical bar (<span class="red">|</span>) symbol after each entry, except the last one.<div class="box"> [[Variables#BRY|BRYNAME]] == roms_bry.nc ! Open boundary conditions<br /> [[Variables#CLM|CLMNAME]] == roms_clm.nc ! Climatology</div>

* Input climatology nudging coefficients file name.<div class="box"> [[Variables#NUD|NUDNAME]] == roms_nud.nc</div>

* Input Sources/Sinks forcing (like river runoff) file name. This file is separated from the regular forcing files to allow manipulations over nested grids. A particular nesting grid may or may not have Sources/Sinks forcing.<div class="box"> [[Variables#SSF|SSFNAME]] == roms_rivers.nc</div>

* Input tidal forcing file name.<div class="box"> [[Variables#TIDE|TIDENAME]] == roms_tides.nc</div>

* Input forcing NetCDF file name(s). The '''user''' has the option to enter several files names for each nested grid. For example, the '''user''' may have a different files for wind products, heat fluxes, tides, etc. The model will scan the file list and will read the needed data from the first file in the list containing the forcing field. Therefore, the order of the file names is very important. If multiple forcing files per grid, enter first all the file names for grid 1, then grid 2, and so on. It is also possible to split input data time records into several NetCDF files (see the [[Input_Parameter_Files#Important_Notes_on_File_Syntax|File Syntax Notes]]). Use a single line per entry with a continuation ( <span class="red">\</span> ) or vertical bar ( <span class="red">|</span> ) symbol after each entry, except the last one.<div class="box"> [[Variables#nFfiles|NFFILES]] == 1 ! number of unique forcing files<br /><br /> [[Variables#FRC|FRCNAME]] == roms_frc.nc ! forcing file 1, grid 1</div>

== Output NetCDF Files ==

{{note}}'''NOTE:''' Starting with '''revision 460''' file names can be up to 256 characters long. Previously only 80 characters were allowed.

* Output NetCDF file names, [1:[[Ngrids]]] files are expected.<div class="box"> [[Variables#DAI|DAINAME]] == roms_dai.nc ! Data assimilation next cycle initial conditions or restart file<br /> [[Variables#GST|GSTNAME]] == roms_gst.nc ! GST analysis restart<br /> [[Variables#RST|RSTNAME]] == roms_rst.nc ! Restart<br /> [[Variables#HIS|HISNAME]] == roms_his.nc ! History<br /> [[Variables#QCK|QCKNAME]] == roms_qck.nc ! Quicksave<br /> [[Variables#TLM|TLMNAME]] == roms_tlm.nc ! TLM history<br /> [[Variables#TLF|TLFNAME]] == roms_tlf.nc ! Impulse TLM forcing<br /> [[Variables#ADM|ADJNAME]] == roms_adj.nc ! ADM history<br /> [[Variables#AVG|AVGNAME]] == roms_avg.nc ! Averages<br /> [[Variables#HAR|HARNAME]] == roms_har.nc ! least-squares detiding harmonics<br /> [[Variables#DIA|DIANAME]] == roms_dia.nc ! Diagnostics<br /> [[Variables#STA|STANAME]] == roms_sta.nc ! Stations<br /> [[Variables#FLT|FLTNAME]] == roms_flt.nc ! Floats</div>

== Additional Input Scripts ==

{{note}}'''NOTE:''' Starting with '''revision 460''' file names can be up to 256 characters long. Previously only 80 characters were allowed.

* Input ASCII parameter filenames.<div class="box"> [[Variables#aparnam|APARNAM]] = ROMS/External/[[s4dvar.in]]<br /> [[Variables#sposnam|SPOSNAM]] = ROMS/External/[[stations.in]]<br /> [[Variables#fposnam|FPOSNAM]] = ROMS/External/[[floats.in]]<br /> [[Variables#bparnam|BPARNAM]] = ROMS/External/[[biology.in]]<br /> [[Variables#sparnam|SPARNAM]] = ROMS/External/[[sediment.in]]<br /> [[Variables#USRname|USRNAME]] = ROMS/External/MyFile.dat</div>

External Libraries

2025-05-05T18:32:28Z

Robertson:

<div class="title">External Libraries</div>

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

==ARPACK (ARnoldi PACKage)==
Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method ('''IRAM''') for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the '''BLAS''' and '''LAPACK''' libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

To obtain '''ARPACK''', execute the following:
<div class="box">> git clone https://github.com/myroms/roms_libs.git<br />> cd roms_libs/ARPACK</div>
To compile, customize '''ARmake.inc''' for the desired compiler and its flags for the serial ('''FC''' and '''FFLAGS''') and parallel ('''PFC''' and '''PFFLAGS''') versions of the library. Once configured type:<div class="box">> make lib plib</div> to build '''libarpack.a''' and '''libparpack.a'''. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the '''[[my_build_paths]]''' script.

{{note}} '''Note:''' Prior to SVN revision 1080 (July 23, 2021), '''ARPACK''' was distributed in the ROMS '''Lib/ARPACK''' subdirectory. For more information about why this was changed see TRAC ticket [https://www.myroms.org/projects/src/ticket/891 #891]

==ESMF (Earth System Modeling Framework)==
High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model ('''ESM''') components. It includes the '''NUOPC''' (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The '''NUOPC''' ''cap'' file is a Fortran module layer that sits on top of each '''ESM''' component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the '''NUOPC''' layer includes the native CoupleSets for '''ESM''' applications with nested grids. Various types of connections are possible, like '''nest-to-nest''', '''fine-to-coarse''', or '''coarse-to-fine'''. The user decides which nested grids to connect.

*Download from github:<div class="box">> git clone https://github.com/esmf-org/esmf.git esmf<br />> cd esmf<br />> git checkout tags/ESMF_8_1_0</div>

*Configure with environment variables:<div class="box">> export ESMF_DIR=/projects/dmcs_1/src/esmf<br />> export ESMF_COMM=mvapich2<br />> export ESMF_COMPILER=intel<br />> export ESMF_BOPT=O<br />> export ESMF_OPTLEVEL=3<br />> export ESMF_ABI=64<br />> export ESMF_INSTALL_PREFIX=/ESMF/install/dir ESMF_NETCDF="/path/to/netcdf/bin/nc-config"<br />> export ESMF_NETCDF_LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lhdf5_hl -lhdf5 -lm -lz -lcurl"<br />> export ESMF_NETCDF_LIBPATH="/path/to/netcdf/lib /path/to/hdf5/lib"<br />> export ESMF_SHARED_LIB_BUILD=OFF<br />> export ESMF_MPIRUN=/path/to/cluster/submission/script</div>{{note}} Note that ''ESMF_MPIRUN'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-np''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -np N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-np" ]<br />then<br /> echo "Usage: mpirun -np # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install:<div class="box">> make -j 10<br />> make all_tests<br />> make install</div>

==MCT (Model Coupling Toolkit)==

Open-source library distributed as a set of Fortran-90 modules for constructing a coupled model system from Earth System Model (ESM) components ([http://www-unix.mcs.anl.gov/mct Larson et al., 2004]). Each component model has its own grid and runs on its own set of processors. The MCT library provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred.

The MCT library is used to couple ROMS atmosphere models (like WRF) and wave models (like SWAN and WW3). For example, SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

MCT is available via github:

<div class="box">> git clone https://github.com/MCSclimate/MCT<br />> cd MCT<br />> ./configure<br />> make</div>

==NetCDF (Network Common Data Form)==

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides '''API''' in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses '''NetCDF''' for all its input and output data management. Its '''NetCDF3''' or '''NetCDF4''' type files can be processed using the standard library developed and distributed [http://doi.org/10.5065/D6H70CW6 UCAR/Unidata], the Parallel-IO ('''PIO''') developed at ANL/NCAR ([https://doi.org/10.1177/1094342011428143 Dennis ''et al''., 2012]; [https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE Hartnett and Edwards, 2021]). We have migrated to the Fortran-90 '''API''' interface, which consists of a library and one or two module files. See the [[makefile]] and [[build_roms| build script]] for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

===Building NetCDF-4/HDF5===
NetCDF-4 uses HDF5 for it's underlying data format so you first need to build HDF5. We recommend 1.10.6 because 1.10.7 causes errors in the NetCDF-C test suite. Here we will describe how to build parallel versions for use on an HPC cluster. We will note what changes are necessary if you are building for a system that doesn't have parallel I/O.

====Building HDF5====
* First, download '''HDF5''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.6/src/hdf5-1.10.6.tar.gz</div>or<div class="box">> git clone https://github.com/HDFGroup/hdf5<br />> cd hdf5<br />> git checkout tags/hdf5-1_10_6<br />> autoconf -i</div>

* Configure '''HDF5''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort \<br /> RUNPARALLEL="srun --nodes=1 --ntasks=6 --cpus-per-task=1 --time=00:13:00 --export=ALL " \<br /> --prefix=/path/to/parallel/hdf5/1.10.6 --enable-static-exec --enable-shared=no \<br /> --enable-parallel --disable-silent-rules --enable-fortran</div>{{note}} '''Note:''' ''RUNPARALLEL'' is only needed if your system uses a scheduler to run MPI jobs. The example here is for a SLURM system but you will need to adjust this for your cluster.<br /><br />{{note}} '''Note:''' To compile '''serial''', remove ''RUNPARALLEL=...'' and ''--enable-parallel'' and change the CC, CXX, and FC to the corresponding serial compilers (e.g. gcc, g++, and gfortran).<br /><br />{{note}} '''Note:''' If you would like '''shared libraries''' to be built, also remove ''--enable-shared=no''.<br /><br />

* Build, test, and install '''HDF5''':<div class="box">> make -j 5<br />> make all_tests<br />> make install</div>

====Building NetCDF-C====
The C and Fortran APIs for NetCDF are in separate packages and the C version must be built first.

* Download '''NetCDF-C''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-c-4.7.4.tar.gz</div>or<div class="box">> git clone https://github.com/Unidata/netcdf-c<br />> cd netcdf-c<br />> git checkout tags/v4.7.4<br />> autoconf -i</div>

* Configure '''NetCDF-C''':<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>{{note}} '''Note:''' To compile '''serial''', remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

====Building NetCDF-Fortran====
Now we can build the Fortran API.

* Download '''NetCDF-Fortran''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.5.3.tar.gz</div>or<div class="box">git clone https://github.com/Unidata/netcdf-fortran<br />> cd netcdf-fortan<br />> git checkout tags/v4.5.3<br />> autoconf -i</div>

* There is a bug/typo in '''configure''' that makes ''nf-config'' report the wrong linking information to ROMS. To fix the error, edit the '''configure''' file (and '''configure.ac''' if you plan to run ''autoconf -i'' again) and change the line:<div class="box">NC_FLIBS="-lnetcdff $NC_FLIBS"</div>to<div class="box">NC_FLIBS="-lnetcdff $NC_LIBS"</div> more details can be found [https://github.com/Unidata/netcdf-fortran/issues/270 here].<br /><br />

* Configure '''NetCDF-Fortran'''. Note that we set ''prefix'' to the same as '''NetCDF-C''' so the C and Fortran libraries live together:<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 FFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include -I/path/to/parallel/netcdf/4.7.4/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib -L/path/to/parallel/netcdf/4.7.4/lib" \<br /> LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' should point to the submision script you created while building '''NetCDF-C''' and is only needed for running test on clusters that use scheduling systems like SLURM.<br /><br />{{note}} '''Note:''' To compile serial remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* If you are running on clusters that use scheduling systems like SLURM, you will need to edit 4 testing scripts replacing all occurrences of '''mpiexec''' with the full path to the submision script you created for '''NetCDF-C''':<div class="box">> vi examples/F90/run_f90_par_examples.sh<br />> vi nf03_test4/run_f90_par_test.sh<br />> vi nf_test4/run_f77_par_test_03.sh<br />> vi nf_test4/run_f77_par_test.sh</div>

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

==PIO (Parallel I/O)==

Open-source parallel library for reading and writing distributed arrays to several scientific data formats like '''pNetCDF''', '''NetCDF3''', and '''NetCDF4/HDF5'''. '''PIO''' provides the flexibility to regulate the number of I/O tasks through data rearrangement between computational and I/O processes to improve performance and memory usage. It allows both '''synchronous''' and '''asynchronous''' I/O. In '''synchronous mode''', a subset of processes performs both I/O and computations (MPI intra-communications). Alternatively, in '''asynchronous mode''', the I/O is carried out by a set of disjointed and dedicated processes (MPI inter-communications).

The '''PIO''' library is intended for ROMS Message Passing Interface (MPI) applications running on a large number of processes in an HPC computer with a '''Parallel File System''' (Lustre, GPFS, and so on). It uses the '''MPI-IO''' interface to facilitate the partitioning of data across computational or dedicated I/O processes. [https://github.com/NCAR/ParallelIO PIO] will work with ROMS [https://www.myroms.org/projects/src/ticket/884 svn revision 1064] (May 10, 2021) or newer. We recommend you read the manuals for all the build options, but below we show example build processes for '''PIO'''.

'''PIO''' can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for '''PIO''', PnetCDF is optional. {{note}}However, the ROMS build system assumes that '''PIO''' is built with PnetCDF support and will need customization if '''PIO''' is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

'''PIO''' recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the '''PIO''' library but ROMS Parallel I/O is not currently implemented this way. In order to build '''PIO''' with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The '''PIO''' library can be built with the classic Autotools or Cmake. Both are shown below.

===Building PIO With Autotools===

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_5_4<br />> autoconf -i</div>

*If you downloaded the gzipped tarball, you should run ''make clean'' because some Fortran .mod files are included and need to be removed:<div class="box">> make clean</div>

*Configure '''PIO''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/netcdf/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \<br /> LDFLAGS="-L/path/to/netcdf/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \<br /> LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz" \<br /> --with-mpiexec=/path/to/cluster/submission/script \<br /> --prefix=/pio/install/path --enable-fortran --disable-timing \<br /> --disable-silent-rules --disable-shared</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install '''PIO''':<div class="box">> make -j 5<br />> make check<br />> make install</div>{{warning}} '''Warning:''' If you are using the Intel compilers, you will probably need to remove "'''test_darray_async_many'''" from line 50 of '''test/cunit/run_tests.sh''' for ''make check'' to complete without errors. Details [https://github.com/NCAR/ParallelIO/issues/1867 here].

===Building PIO With Cmake===

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_5_4</div>

*Configure '''PIO''':<div class="box">> cd ..<br />> mkdir build_pio<br />> cd build_pio<br />> CC=mpicc CXX=mpicxx FC=mpifort cmake \<br /> -DNetCDF_PATH=/path/to/netcdf \<br /> -DPnetCDF_PATH=/path/to/pnetcdf \<br /> -DMPIEXEC=/path/submission/scipt \<br /> -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \<br /> -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \<br /> -DCMAKE_INSTALL_PREFIX=/pio/install/path ../ParallelIO</div>{{note}} Note that ''-DMPIEXEC'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Compile, test, and install:<div class="box">> make -j 6<br />> make -j 6 tests<br />> ctests<br />> make install</div>

IO

2025-05-05T18:18:20Z

Robertson:

<div class="title">ROMS I/O</div>

ROMS uses NetCDF for all its input and output data management. The NetCDF files can be processed using the standard library developed by [http://doi.org/10.5065/D6H70CW6 UNIDATA], the Parallel-IO ('''PIO''') library developed at NCAR (Hartnett and Edwards, 2021; [https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE unpublished paper]). Furthermore, another parallel I/O strategy has been available in ROMS for several years with the NetCDF4/HDF5 libraries by activating the [[Options#PARALLEL_IO|PARALLEL_IO]] and [[Options#HDF5|HDF5]] CPP options.

Generally, writing is usually a more frequent and more complicated operation than reading. There are four strategies for writing (Mendez ''et al.'', 2019; [https://www.researchgate.net/publication/332190037_Best_Practice_Guide_-_Parallel_IO Preprint]):

#<span class="blue">Single file, single writer:</span> '''Serial I/O''' in non-parallel or parallel applications. It is the default strategy in ROMS using the '''NetCDF3''' or '''NetCDF4''' libraries.
#<span class="blue">Single file, multiple writers:</span> '''Parallel I/O''' with each partition tile writing its data to a single file. In ROMS, this capability is achieved by activating [[Options#PARALLEL_IO|PARALLEL_IO]] and [[Options#HDF5|HDF5]]. It is only possible with the '''NetCDF4/HDF5''' libraries.
#<span class="blue">Single file, collective writers:</span> '''Parallel I/O''' with either one or a subset of processes performing I/O operations. The I/O operations can be '''synchronous''' or '''asynchronous'''. In ROMS, this capability uses the '''PIO''' library and is available when [[Options#PIO_LIB|PIO_LIB]] is activated.
#<span class="blue">Multiple files, multiple writers:</span> '''Parallel I/O''' in which each distributed-memory or shared-memory tile decomposition writes its data into the partition file. Still, post-processing is required to pack the data into a single file. Currently, this capability is not available in ROMS but can be easily implemented within its current I/O infrastructure. However, this strategy is cumbersome and undesired in variational data assimilation (4D-Var) algorithms that require reading forwards and backward, in time, the state trajectories. As a consequence, reading becomes the bottleneck.

__TOC__

==Serial I/O==
Serial I/O is the standard option that has been in ROMS since the beginning. In this setup, all input and output data flows through the master parallel process. All data for output is collected from all processes by the master process and written to disk. Likewise, all data for input is read by the master process and distributed to the rest of the processes. When using serial I/O, files can be written in NetCDF classic/64-bit offset (NetCDF-3) or NetCDF-4/HDF5 ([[Options#HDF5|HDF5]] CPP option) formats. File compression is available in the NetCDF-4/HDF5 library.

==Parallel I/O with NetCDF-4==
Parallel I/O using parallel HDF5 and NetCDF-4 has been available in ROMS for many years. This I/O option requires parallel enabled HDF5 and NetCDF-4 and is activated by defining [[Options#PARALLEL_IO|PARALLEL_IO]] and [[Options#HDF5|HDF5]] CPP options. Each parallel tile reads and writes the decomposed data. This approach does not scale well because it requires every process to participate in reading and writing, which quickly overloads the file system with requests as the number of tiles (NtileI x NtileJ) increases.

==Parallel I/O with PIO==

The '''PIO''' library is primarily intended for ROMS '''distributed-memory (MPI) applications''' running on a large number of processes in an HPC system with a Parallel File System (like Lustre, GPFS, and so on) for high-performance I/O. The '''PIO''' library uses the '''MPI-IO''' interface to facilitate the partitioning of the data across computational or dedicated I/O processes. For example, in an HPC cluster environment with a '''Parallel File System''', the user can instruct '''PIO''' to designate which processes per node to perform I/O. This is a much more reasonable approach for larger applications running on hundreds of processors. To use this Parallel I/O strategy, the '''PIO''' library must be linked to ROMS at compile time by defining the [[Options#PIO_LIB|PIO_LIB]] CPP option. <span class="blue">It is only available in distributed-memory applications since it uses '''MPI-IO'''</span>.

There are two modes of parallel I/O in '''PIO''':

<ol style="list-style-type: upper-alpha">
<li>'''Synchronous:''' MPI intra-communications mode. A subset or all processes does I/O and also computations. The user specifies the total number of I/O tasks and how they are distributed across the HPC nodes as a function of the ROMS MPI-communicator object, '''OCN_COMM_WORLD'''. It is often desirable to shift the first I/O task away from the first computation task since it has higher memory requirements than other processes. If the MPI processes are spread over several computer nodes, it is highly recommended to spread all I/O tasks over all nodes. Avoid all I/O processes occupying the same node.</li>
<li>'''Asynchronous:''' MPI inter-communications mode. The I/O tasks are a disjointed set of dedicated I/O processes and do not perform computations. It is possible to have groups of computational units running separate models (coupling) where all the I/O data are sent to dedicated processes. In ROMS, the asynchronous mode is possible by activating [[Options#ASYNCHRONOUS_PIO|ASYNCHRONOUS_PIO]]. '''<span class="red">However</span>, this capability is still under development and not recommended for use at this time.'''</li>
</ol>

The Parallel I/O configuration options are set in [[roms.in]]:
{{#lst:roms.in|PIO}}

==Output Multi-Files==

Sometimes, it is advantageous to time-split ROMS output data (<span class="twilightBlue">averages</span>, <span class="twilightBlue">diagnostic</span>, <span class="twilightBlue">history</span>, and <span class="twilightBlue">quicksave</span>) into multiple NetCDF files to avoid creating huge files on disk for storage in applications with large grids. Smaller files are easy to handle and can be concatenated in OpenDAP catalogs.

For example, an application for a particular region with a substantial grid size needs to be run for one year. The ROMS timestep is [[Variables#DT|DT]] = 300 seconds, as shown below. Then, one could split the <span class="twilightBlue">history</span> and <span class="twilightBlue">quicksave</span> output data into daily NetCDF files with single or multiple time records every '''3''', '''6''', or '''24''' hours. In this case, ROMS will generate a sequence of files with suffixes <span class="violet">_0001.nc</span> to <span class="violet">_0365.nc</span> filenames. Therefore, we need the following parameters in [[roms.in|roms.in]]:

* Timestepping parameters.<div class="box"> [[Variables#ntimes|NTIMES]] = 105120 ! Number of timesteps <span class="mediumOrchid">(288 steps per day; 365 days simulation)</span><br /> [[Variables#dt|DT]] == 300.0d0 ! Timestep size (seconds)<br /> [[Variables#ndtfast|NDTFAST]] == 30 ! Number of barotropic steps</div>

* Flags controlling the frequency of output.<div class="box"> [[Variables#nrrec|NRREC]] = 0 ! Model restart flag<br /> [[Variables#LcycleRST|LcycleRST]] == T ! Switch to recycle restart time records <span class="mediumOrchid">(file with only 2 cycling time records)</span><br /> [[Variables#nRST|NRST]] == 288 ! Number of timesteps between writing restart records <span class="mediumOrchid">(daily)</span><br /> [[Variables#nSTA|NSTA]] == 1 ! Number of timesteps between stations records<br /> [[Variables#nFLT|NFLT]] == 1 ! Number of timesteps between floats records<br /> [[Variables#ninfo|NINFO]] == 1 ! Number of timesteps between printing information diagnostics</div>

* Output history, average, and diagnostic file parameters.<div class="box"> [[Variables#ldefout|LDEFOUT]] == T ! File creation/append switch<br /> [[Variables#nHIS|NHIS]] == 72 ! Number of timesteps between writing history records <span class="mediumOrchid">(every 6 hours)</span><br /> [[Variables#ndefHIS|NDEFHIS]] == 288 ! Number of timesteps between the creation of new history files <span class="mediumOrchid">(daily files)</span><br /> [[Variables#nQCK|NQCK]] == 36 ! Number of timesteps between writing quicksave records <span class="mediumOrchid">(every 3 hours)</span><br /> [[Variables#ndefQCK|NDEFQCK]] == 288 ! Number of timesteps between the creation of new quicksave file <span class="mediumOrchid">(daily)</span><span class="mediumOrchid">(daily, single record files)</span><br /> [[Variables#ntsAVG|NTSAVG]] == 1 ! Starting averages timestep<br /> [[Variables#nAVG|NAVG]] == 288 ! Number of timesteps between writing averages records <span class="mediumOrchid">(daily averages)</span><br /> [[Variables#ndefAVG|NDEFAVG]] == 288 ! Number of timesteps between the creation of new averages file <span class="mediumOrchid">(daily, single record files)</span><br /> [[Variables#ntsDIA|NTSDIA]] == 1 ! Starting diagnostics timestep<br /> [[Variables#nDIA|NDIA]] == 288 ! Number of timesteps between writing diagnostics records <span class="mediumOrchid">(daily averages)</span><br /> [[Variables#ndefDIA|NDEFDIA]] == 288 ! Number of timesteps between the creation of new diagnostics file <span class="mediumOrchid">(daily, single record files)</span></div>

* Timestamp assigned for model initialization, reference time origin for tidal forcing, and model reference time for output NetCDF units attribute.<div class="box"> [[Variables#dstart|DSTART]] = 365.0d0 ! days <span class="mediumOrchid">(ROMS is initialized on 2007-01-01 00:00:00Z)</span><br /> [[Variables#tide_start|TIDE_START]] = 0.0d0 ! days <span class="mediumOrchid">(zero phase date is set when preparing the tidal data for 2006-01-01)</span><br /> [[Variables#time_ref|TIME_REF]] = 20060101.0d0 ! yyyymmdd.dd <span class="mediumOrchid">(Very important: ROMS time is seconds since 2006-01-01 00:00:00Z)</span></div>

Notice that all the parameters are an exact integer multiple from each other:
<div class="box"> MOD([[Variables#ntimes|NTIMES]], [[Variables#ndefHIS|NDEFHIS]]) = 0 <br /> MOD([[Variables#ntimes|NTIMES]], [[Variables#ndefHIS|NDEFQCK]]) = 0 <br /> MOD([[Variables#ntimes|NTIMES]], [[Variables#ndefAVG|NDEFAVG]]) = 0 <br /> MOD([[Variables#ntimes|NTIMES]], [[Variables#ndefDIA|NDEFDIA]]) = 0 <br /><br /> MOD([[Variables#ntimes|NTIMES]], [[Variables#nRST|NRST]]) = 0 <br /><br /> MOD([[Variables#ndefHIS|NDEFHIS]], [[Variables#nHIS|NHIS]]) = 0 <br /> MOD([[Variables#ndefQCK|NDEFQCK]], [[Variables#nQCK|NQCK]]) = 0 <br /> MOD([[Variables#ndefAVH|NDEFAVG]], [[Variables#nAVG|NAVG]]) = 0 <br /> MOD([[Variables#ndefDIA|NDEFDIA]], [[Variables#nHIS|NDIA]]) = 0 <br /></div>

where the Fortran intrinsic function '''MOD(<span class="red">X</span>,<span class="red">Y</span>)''' computes the remainder of the division of <span class="red">X</span> by <span class="red">Y</span>, and has to be always '''zero''' for ROMS multi-file option to work. Notice that the first files in the <span class="twilightBlue">history</span> data series (<span class="violet">roms_his_0001.nc</span>) will contain '''5'''-time records because of the initial conditions, and the rest of the files will have '''4'''-time records. Similarly, the first file in the <span class="twilightBlue">quicksave</span> series (<span class="violet">roms_qck_0001.nc</span>) will contain '''9'''-time records, and the rest will have '''8'''-time records. The time-averaged data in the <span class="twilightBlue">averages</span> and <span class="twilightBlue">diagnostic</span> files are single records files representing daily averaged fields. The above analysis also holds when converted to time in seconds since every parameter is multiplied by the [[Variables#dt|DT]] timestep.

<span class="red">WARNING:</span> If running your application in a supercomputer with limited-time job queues, long simulations will require a restart. You '''cannot''' change the values of [[Variables#ntimes|NTIMES]] or [[Variables#dstart|DSTART]] because the multi-file will fail. If [[Variables#dstart|DSTART]] is changed, it will reset the timestep internal counter <span class="red">iic(ng)</span>, and we are in deep trouble! Notice that the value [[Variables#nRST|NRST]] is crucial for the restart of ROMS with multi-file. In addition, we need the following mathematical equalities:
<div class="box"> MOD([[Variables#nRST|NRST]], [[Variables#nHIS|NHIS]]) = 0<br /> MOD([[Variables#nRST|NRST]], [[Variables#nQCK|NQCK]]) = 0<br /> MOD([[Variables#nRST|NRST]], [[Variables#nAVG|NAVG]]) = 0<br /> MOD([[Variables#nRST|NRST]], [[Variables#nDIA|NDIA]]) = 0<br /></div>
Also, it will be trivial to restart if [[Variables#nAVG|NAVG]] = [[Variables#nDIA|NDIA]] = [[Variables#nRST|NRST]] because of the accumulation sums when computing time-averaging fields. If balancing terms and budgets from output data, you will need [[Variables#nAVG|NAVG]] = [[Variables#nDIA|NDIA]] too.

my build paths

2025-05-05T18:14:15Z

Robertson:

<div class="title">Library and Executable Paths - <span class="red">my_build_paths.csh, my_build_paths.sh</span></div>

{{note}}'''Note:''' in SVN revision 933 (January 26, 2019) the custom libraries section was removed from the ROMS [[build_roms|build script]] and placed in <span class="red">Compilers/my_build_paths.csh</span> and <span class="red">Compilers/my_build_paths.sh</span> to both streamline the ROMS build script and make setting up new applications less tedious. More information can be found in ROMS [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. Prior to ROMS release 933, the paths mentioned below were included in the ROMS build script.

Rather than editing files released with ROMS and possibly creating conflicts during updates, we suggest that you copy the appropriate '''my_build_paths''' file to another folder (we recommend <span class="red">~/Compilers/ROMS</span>) and set [[MY_PATHS]] to the full path to your '''my_build_paths''' file (i.e. ~/Compilers/ROMS/my_build_paths.csh).

The path of the libraries and MPI implementations required by ROMS can be set using environment variables which take precedence over the values specified in the makefile macro definitions file ('''Compilers/*.mk'''). To activate custom paths set in <span class="red">my_build_paths</span>, set [[USE_MY_LIBS]] to '''yes''' and make sure [[MY_PATHS]] is properly set in the ROMS [[build_roms|build script]].

==Compiler and MPI Implementation==

There is a section (the one from <span class="red">my_build_paths.csh</span> is shown below) in the [[MY_PATHS]] file to make it easier to switch between different compilers and '''MPI''' implementations. This will only apply to '''MPI''' implementations that use the <span class="red">mpif90</span> style wrapper script ('''MPICH''', '''MPICH2''', '''OpenMPI''', etc.).

:<div class="box">if ($?USE_MPIF90) then<br /> switch ($FORT)<br /><br /> case "ifort"<br /> if ($which_MPI == "mpich" ) then<br /> setenv MPI_ROOT /opt/intelsoft/mpich<br /> else if ($which_MPI == "mpich2" ) then<br /> setenv MPI_ROOT /opt/intelsoft/mpich2<br /> else if ($which_MPI == "openmpi" ) then<br /> setenv MPI_ROOT /opt/intelsoft/openmpi<br /> else if ($which_MPI == "mvapich2" ) then<br /> setenv MPI_ROOT /opt/intelsoft/mvapich2<br /> endif<br /> setenv PATH ${MPI_ROOT}/bin:$PATH<br /> setenv MPI_INCDIR ${MPI_ROOT}/include<br /> breaksw<br /><br /> case "pgi"<br /> if ($which_MPI == "mpich" ) then<br /> setenv MPI_ROOT /opt/pgisoft/mpich<br /> else if ($which_MPI == "mpich2" ) then<br /> setenv MPI_ROOT /opt/pgisoft/mpich2<br /> else if ($which_MPI == "openmpi" ) then<br /> setenv MPI_ROOT /opt/pgisoft/openmpi<br /> else if ($which_MPI == "mvapich2" ) then<br /> setenv MPI_ROOT /opt/pgisoft/mvapich2<br /> endif<br /> setenv PATH ${MPI_ROOT}/bin:$PATH<br /> setenv MPI_INCDIR ${MPI_ROOT}/include<br /> breaksw<br /><br /> case "gfortran"<br /> if ($which_MPI == "mpich2" ) then<br /> setenv MPI_ROOT /opt/gfortransoft/mpich2<br /> else if ($which_MPI == "openmpi" ) then<br /> setenv MPI_ROOT /opt/gfortransoft/openmpi<br /> else if ($which_MPI == "mvapich2" ) then<br /> setenv MPI_ROOT /opt/gfortransoft/mvapich2<br /> endif<br /> setenv PATH ${MPI_ROOT}/bin:$PATH<br /> setenv MPI_INCDIR ${MPI_ROOT}/include<br /> breaksw<br /><br /> endsw<br />endif</div>

{{note}}Keep in mind that you need to adjust the paths to your '''MPI''' implementations for your particular system. You must also set '''USE_MPIF90''' to on.

==External Library Paths==

For information about required and optional external libraries visit the [[External Libraries]] page.

For most applications, only the location of the '''NetCDF''' library ('''NETCDF_LIBDIR''') and include directory ('''NETCDF_INCDIR''') are needed. Notice that when the [[#USE_NETCDF4|USE_NETCDF4]] macro is activated, we need the serial or parallel version of the '''NetCDF-4/HDF5''' library. The configuration script '''NF_CONFIG''' (available since NetCDF 4.2) is used to set up all the required libraries according to the installed options ('''OPeNDAP''', '''NetCDF4/HDF5''' file format). The parallel library uses the '''MPI-I/O''' layer (available in most modern MPI implementation) requiring compiling with the selected '''MPI''' library.

In ROMS distributed-memory applications, you may use either the serial or parallel version of the '''NetCDF-4/HDF5''' library. The parallel version is required when parallel I/O is activated (ROMS cpp option [[Options#PARALLEL_IO|PARALLEL_IO]] and [[Options#HDF5|HDF5]]). For more I/O options see the [[IO|ROMS I/O page]].

However, in serial or shared-memory ROMS applications, we need to use the serial version of the '''NetCDF-4/HDF5''' to avoid conflicts with the compiler. We cannot activate '''MPI''' constructs in serial or shared-memory ROMS code. Hybrid parallelism is not possible.

Below is an excerpt from the custom library section.

:<div class="box">setenv MPI_SOFT ""<br /><br />switch ($FORT)<br /><br /># Intel Compiler:<br /><br /> case "ifort"<br /> setenv ESMF_COMPILER intelgcc<br /> if ($?USE_DEBUG) then<br /> setenv ESMF_BOPT g<br /> else<br /> setenv ESMF_BOPT O<br /> endif<br /><br /> setenv ESMF_ABI 64<br /> setenv ESMF_COMM ${which_MPI}<br /> setenv ESMF_SITE default<br /><br /> setenv ARPACK_LIBDIR /opt/intelsoft/serial/ARPACK<br /><br /> if ($?USE_MPI) then<br /> if ($which_MPI == "mpich" ) then<br /> setenv MPI_SOFT /opt/intelsoft/mpich<br /> else if ($which_MPI == "mpich2" ) then<br /> setenv MPI_SOFT /opt/intelsoft/mpich2<br /> else if ($which_MPI == "openmpi" ) then<br /> setenv MPI_SOFT /opt/intelsoft/openmpi<br /> else if ($which_MPI == "mvapich2" ) then<br /> setenv MPI_SOFT /opt/intelsoft/mvapich2<br /> endif<br /><br /> setenv MCT_INCDIR ${MPI_SOFT}/mct/include<br /> setenv MCT_LIBDIR ${MPI_SOFT}/mct/lib<br /> setenv PARPACK_LIBDIR ${MPI_SOFT}/PARPACK<br /> endif<br /><br /> if ($?USE_NETCDF4) then<br /> if ($?USE_PARALLEL_IO && $?USE_MPI) then<br /> setenv ESMF_DIR ${MPI_SOFT}/esmf_nc4<br /> setenv NETCDF ${MPI_SOFT}/netcdf4<br /> setenv NF_CONFIG ${NETCDF}/bin/nf-config<br /> setenv NETCDF_INCDIR ${NETCDF}/include<br /> setenv NETCDF4 1<br /> else<br /> setenv ESMF_DIR ${MPI_SOFT}/esmf_nc4<br /> setenv NETCDF /opt/intelsoft/serial/netcdf4<br /> setenv NF_CONFIG ${NETCDF}/bin/nf-config<br /> setenv NETCDF_INCDIR ${NETCDF}/include<br /> setenv NETCDF4 1<br /> endif<br /> else<br /> setenv ESMF_DIR ${MPI_SOFT}/esmf_nc3<br /> setenv NETCDF /opt/intelsoft/serial/netcdf3<br /> setenv NETCDF_INCDIR ${NETCDF}/include<br /> setenv NETCDF_LIBDIR ${NETCDF}/lib<br /> endif<br /><br /> if ($?USE_PIO) then<br /> setenv PIO ${MPI_SOFT}/pio<br /> setenv PIO_LIBDIR ${PIO}/lib<br /> setenv PIO_INCDIR ${PIO}/include<br /> setenv PNETCDF ${MPI_SOFT}/pnetcdf<br /> setenv PNETCDF_LIBDIR ${PNETCDF}/lib<br /> setenv PNETCDF_INCDIR ${PNETCDF}/include<br /> endif<br /><br /> if ($?USE_HDF5) then<br /> if ($?USE_PARALLEL_IO && $?USE_MPI) then<br /> setenv HDF5 ${MPI_SOFT}/hdf5<br /> setenv HDF5_LIBDIR ${HDF5}/lib<br /> setenv HDF5_INCDIR ${HDF5}/include<br /> else<br /> setenv HDF5 /opt/intelsoft/serial/hdf5<br /> setenv HDF5_LIBDIR ${HDF5}/lib<br /> setenv HDF5_INCDIR ${HDF5}/include<br /> endif<br /> endif<br /><br /> . . .<br /><br /> breaksw</div>

Git

2025-03-25T17:19:12Z

Robertson: /* Configuring GIT */

<div class="title">Git</div>
'''ROMS''' source code is now exclusively distributed using [https://git-scm.com Git]. The '''GitHub''' repository includes the full history since 2002 of changes to the '''ROMS''' source code. There are command line and GUI Git clients available for nearly every operating system and a list of popular clients can be found [https://git-scm.com/downloads/guis here]. This page will help you get started with downloading '''ROMS''' with Git.

==Git Overview==
Git is a free and open-source distributed version control system designed to handle everything from small to massive projects quickly and efficiently. All the ROMS files are stored in a Git repository on <span class="twilightBlue">github.com/myroms</span>.

This Git repository is the official version of the code, which only developers are allowed to change. Users should download the ROMS code to their local machines using a '''git client'''. Don't attempt to use a regular web browser to browse or download files from the Git repository—there are much better tools for interacting with the code repository.

We strongly recommend users check out the current <span class="red">develop</span> branch, which has the most recent updates and bug fixes. The ''tags'' are mainly kept as a historical record of stable releases after major code upgrades.

If you are making changes yourself, keep them in a separate branch, leaving the <span class="red">develop</span> branch to track changes from the source. Please check the [https://github.com/myroms/roms/wiki/Useful-GIT-Commands ROMS GitHub Wiki] for useful Git commands.

Below is a general description of how Git works. For more detailed information, please review the [https://git-scm.com/book/en/v2 Pro Git] book. We have not tried GUI clients but may add brief how-tos for the most popular GUIs later.

==Configuring GIT==

Before downloading any of '''ROMS''' repositories, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading some '''''roms_test''''' input and observation NetCDF files. Otherwise, the '''Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box">more ~/.gitconfig<br /><br /> [User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br /> [credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br /> [filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you clone the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a git repository, '''git client''' software must be installed on your local machine. Most Linux distributions come with git, so shell commands may be used without installing additional software. The general form of git commands is:

<span class="red">git</span> action <repository>

To check-out the files from the ROMS repository '''develop''' (latest version), enter (notice https instead of http):

> <span class="red">git clone <nowiki>https://github.com/myroms/roms.git</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If '''MyDir''' is omitted the code will be downloaded to a directory named '''roms'''. You only clone once because Git will keep track of the source, destination and a bunch of other information. For more detail on command line use and syntax, see the [https://git-scm.com/book/en/v2 Pro Git] book.

The idealized and realistic ROMS Test Cases and the Matlab processing software can be downloaded from:

> <span class="red">git clone <nowiki>https://github.com/myroms/roms_test.git </nowiki></span>
> <span class="red">git clone <nowiki>https://github.com/myroms/roms_matlab.git</nowiki></span>

We highly recommend that Users clone all the ROMS repositories from the same root directory in their computer and define the '''ROMS_ROOT_DIR''' variable in their computer shell login environment, specifying where the User cloned/downloaded the ROMS source code, Test Cases, and Matlab processing software. That is, use the value of pwd from the computer directory where you executed git clone https://github.com/myroms/roms.git

For bash shells:
export ROMS_ROOT_DIR=MyDownlodLocationDirectory

For csh/tcsh shells:
setenv ROMS_ROOT_DIR MyDownlodLocationDirectory

The build scripts will use this environmental variable when compiling any of the ROMS Test Cases without the need to customize the location of the ROMS source code. Also, it is used for loading the path of Matlab scripts in the startup.m configuration file.

===Git Hash Revision Information===
{{note}}'''Note:''' While this is not a required step, we highly recommend it to make bug reporting and tracking easier.

In order to get Git hash revision information in the header for each file, we have created a Git filter that will insert the git hash of the last time a file was modified, similar to the Subversion '''''$Id$''''' keyword. In order for the filter to work, it has to be explicitly enabled by the user. After the git clone command from above completes change into <span class="limeGreen">MyDir</span> and execute the following:

> <span class="red">git config filter.id.smudge ".git_filters/id.smudge %f"</span>
> <span class="red">git config filter.id.clean ".git_filters/id.clean %f"</span>

In order to get the Git Id hashes to fill in, you will need to force the smudge filter to run on the source code you have already downloaded. This takes several minutes but only needs to be done once.

For '''bash''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "$(git rev-parse --show-toplevel)"</span>

For '''csh/tcsh''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "`git rev-parse --show-toplevel`"</span>

If you look at the top of any ROMS source file, you will see the Git commit hash for the last time that file was modified. For example, the first few lines of <span class="limeGreen">ROMS/Modules/mod_ncparam.F</span> look like this:<div class="box">#include "cppdefs.h"<br /> MODULE mod_ncparam<br />!<br />!git $Id: caf970e3 2024-08-09 17:15:26 UTC arango $<br />!================================================== Hernan G. Arango ===<br />! Copyright (c) 2002-2024 The ROMS/TOMS Group !<br />! Licensed under a MIT/X style license !<br />! See License_ROMS.md !<br />!=======================================================================</div>

You can then use the 8 character (hexidecimal) Git hash to show more information about that code update:<div class="box">> <span class="red">git show --name-only caf970e3</span></div>

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

> <span class="red">git pull</span>

Git will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

==Managing Your Own Modifications==
This assumes that you have a fresh clone of the myroms repository on the '''develop''' branch. You want to keep '''develop''' as a pure copy of the source version and keep your own changes in say the '''arctic''' branch. Start by creating a branch and switching to it:

> <span class="red">git branch arctic<br /></span>> <span class="red">git checkout arctic</span>

Now you can make whatever modifications you like (and test them out). To see what changed, you can use <span class="red">git status</span> and <span class="red">git diff</span>. To save your changes, do a:

> <span class="red">git commit -a</span>

Though if you add new files you will have to <span class="red">git add</span> them first.

===Getting the Updates===
It is easy to fetch and merge the updates. Start by making sure your directory has been cleanly checked in with <span class="red">git status</span>. Then you can update your '''develop''' branch:

> <span class="red">git checkout develop</span><br />> <span class="red">git pull</span>

Then bring the changes into your '''arctic''' branch:

> <span class="red">git checkout arctic</span><br />> <span class="red">git merge develop</span>

This will bring in everything that changed since your last <span class="red">git pull</span>, so you might find it easier to keep on top of things by doing this often, not putting it off for years. You can also bring in changes one at a time with <span class="red">git cherry-pick</span>. Again, check the Pro Git book for much more information about all of these operations.

Note that this will save your '''arctic''' branch locally, under the .git directory. You can back this up as you would any other important files you have.

==Useful Git Commands==
{| border="1" cellspacing="0" cellpadding="5" align="center"
|'''Command'''
|'''Description'''
|-
| '''git add''' <_FileName_>
| Add a file to the repository staging area
|-
| '''git branch'''
| List all local branches (the asterisk denotes the current branch)
|-
| '''git branch -a'''
| List all local and remote branches
|-
| '''git branch -d''' <_BranchName_>
| Delete a local branch
|-
| '''git branch -D''' <_BranchName_>
| Delete a local branch forcefully
|-
| '''git branch -m''' <_OldName_> <_NewName_>
| Rename a local branch
|-
| '''git checkout -b''' <_BranchName_>
| Create a new local branch and switch to it
|-
| '''git checkout''' <_BranchName_>
| Switch to an existing local or remote branch
|-
| '''git clone''' <_RepositoryURL_>
| Clone a public repository
|-
| '''git commit -am''' "<_Message_>"
| Commit changes to all files
|-
| '''git commit -am''' "<_Message_>" '''-m''' "<_MessageURL_>"
| Commit changes to all files with two message lines
|-
| '''git commit --amend -m''' "<_NewMessage_>"
| Amend previous commit message
|-
| '''git diff'''
| Show all changes between HEAD and working branch
|-
| '''git diff''' <_FileName_>
| Show changes for a specific file between HEAD and working branch
|-
| '''git difftool -d''' <_BranchName1_> <_BranchName2_>
| Compare the difference between two branches with '''`KDIFF3`'''
|-
| '''git fetch'''
| Retrieve new work done by other people
|-
| '''git log'''
| View repository changes
|-
| '''git log --oneline'''
| Show the list of commits in one-line format
|-
| '''git log --summary'''
| View repository detailed changes
|-
| '''git merge --no-ff --no-commit''' <_BranchName_>
| Merge a branch into the active branch
|-
| '''git pull'''
| Update local repository or branch to the newest origin commit
|-
| '''git push origin --delete''' <_BranchName_>
| Delete a remote branch
|-
| '''git push -u origin''' <_BranchName_>
| Push changes to the remote repository (and remember the branch)
|-
| '''git reset --hard HEAD~1'''
| Undo last commit
|-
| '''git revert''' <_CommitID_>
| Revert commit changes
|-
| '''git status'''
| Check/display changes to the repository or particular branch
|-
| '''git tag'''
| List all tags
|}

==Complicated Commands==

If a branch is renamed in a repository on GitHub, the local clone on a computer needs to updated:<div class="box">git branch -m OldName NewName<br />git fetch origin<br />git branch -u origin/NewName NewName<br />git remote set-head origin -a</div>

How to merge changes in the '''feature/name1''' branch into the '''feature/name2''' branch:<div class="box">git checkout feature/name1<br />git pull<br /><br />git checkout feature/name2<br />git pull<br />git merge --no-ff --no-commit feature/name1<br />git commit -am "Merging feature/name1 into feature/name2" -m "MessageURL"<br />git difftool -d feature/name1 feature/name2<br />git push -u origin feature/name2</div>

Workshops

2025-01-30T17:23:26Z

Robertson: Created page with "{| border="1" cellspacing="0" cellpadding="5" align="center" ! Date ! Location ! Description ! Attendees ! Group Picture |- | 08/20/2001 - 08/22/2001 | Boulder, CO | The second joint biannual workshop for users of Terrain-Following Coordinates Ocean Models | 70 | [https://www.myroms.org/images/ROMS_2001_1.gif Picture] |- | 08/04/2003 - 08/06/2003 | Seattle, WA | Final joint POM/SPEM/SCRUM/ROMS workshop | 47 | [https://www.myroms.org/Workshops/Workshop2003/Pictures/Day3/..."

{| border="1" cellspacing="0" cellpadding="5" align="center"
! Date
! Location
! Description
! Attendees
! Group Picture
|-
| 08/20/2001 - 08/22/2001
| Boulder, CO
| The second joint biannual workshop for users of Terrain-Following Coordinates Ocean Models
| 70
| [https://www.myroms.org/images/ROMS_2001_1.gif Picture]
|-
| 08/04/2003 - 08/06/2003
| Seattle, WA
| Final joint POM/SPEM/SCRUM/ROMS workshop
| 47
| [https://www.myroms.org/Workshops/Workshop2003/Pictures/Day3/group04.JPG Picture]
|-
| 10/18/2004 - 10/20/2004
| Venice, Italy
| First European ROMS meeting
| 56
| N/A
|-
| 10/24/2005 - 10/26/2005
| La Jolla, CA
| Workshop focused on adjoint modeling and applications
| 64
| [http://marine.rutgers.edu/po/Workshops/ROMS2005/Pics/group.jpg Picture]
|-
| 11/06/2006 - 11/08/2006
| Alcalá de Henares, Spain
| Second European ROMS meeting at historic University of Alcalá
| 57
| [https://www.myroms.org/Workshops/ROMS2006/Group_Picture.jpg Picture]
|-
| 10/01/2007 - 10/03/2007
| Los Angeles, CA
| Held at UCLA followed by the Community Sediment Transport Modeling (CSTM) meeting
| 77
| [https://www.myroms.org/Workshops/ROMS2007/pics/Group.jpg Picture]
|-
| 10/04/2007 - 10/05/2007
| Los Angeles, CA
| CSTM meeting with status updates and breakout groups
| 37
| N/A
|-
| 10/06/2008 - 10/08/2008
| Grenoble, France
| Third European ROMS meeting in the French Rhone-Alps
| 70
| [https://www.myroms.org/Workshops/ROMS2008/pics/Group.jpg Picture]
|-
| 03/31/2009 - 04/02/2009
| Sydney, Australia
| First Asia-Pacific ROMS meeting
| 64
| [https://www.myroms.org/Workshops/ROMS2009a/pics/Group.jpg Picture]
|-
| 04/05/2010 - 04/08/2010
| Honolulu, HI
| Third Domestic ROMS meeting
| 51
| [https://www.myroms.org/Workshops/ROMS2010/pics/group.jpg Picture]
|-
| 07/12/2010 - 07/16/2010
| Santa Cruz, CA
| Workshop on the ROMS 4D-Var Data Assimilation Systems for advanced ROMS users
| 40
| [https://www.myroms.org/Workshops/4DVAR2010/Pics/group.jpg Picture]
|-
| 10/22/2012 - 10/25/2012
| Rio de Janeiro, Brazil
| First South American ROMS workshop
| 104
| [https://www.myroms.org/Workshops/ROMS2012/pics/group.jpg Picture]
|-
| 05/26/2014 - 05/29/2014
| Rovinj, Croatia
| Fourth European ROMS workshop
| 66
| [https://www.myroms.org/Workshops/ROMS2014/pics/group.jpg Picture]
|-
| 10/17/2016 - 10/20/2016
| Hobart, Tasmania
| Second Asia-Pacific ROMS meeting
| 66
| [https://www.myroms.org/Workshops/ROMS2016/pics/group.jpg Picture]
|-
| 07/29/2019 - 08/02/2019
| College Park, MD
| ROMS 4D-Var training workshop
| 29
| N/A
|}

Subversion

2025-01-17T16:43:40Z

Robertson:

<div class="title">Subversion (Deprecated)</div>
{{warning}} <span class="red">'''Warning:'''</span> The '''ROMS''' source code is no longer distributed using [http://subversion.tigris.org Subversion](SVN). '''ROMS''' is exclusively distributed from its '''GitHub''' repository at [https://github.com/myroms https://github.com/myroms]. Updating the '''SVN''' repositories at [https://www.myroms.org https://www.myroms.org] has been discontinued but kept for historical reasons. The '''GitHub''' repository contains the full history of the '''ROMS''' source code since 2002.

There are SVN clients available for nearly every operating system and a list of popular clients can be found [http://subversion.tigris.org/links.html#clients here]. If you wish to use Subversion to maintain your own copy of ROMS you will also need an [http://subversion.tigris.org/project_packages.html svn server]. This page will help you get started with downloading ROMS with Subversion. There is also a way to download ROMS using git, described [[ROMS_git | elsewhere]].

==SVN Overview==
Subversion is a tool for managing software development that keeps track of who modified what and allows the return to a previous version if changes don't work as expected. All the ROMS/TOMS files are stored in a SVN repository on '''www.myroms.org''' with access controlled by requiring authentication with the same ROMS Username/Password combination assigned to registered users of the ROMS Forum.

This SVN repository is the official version of the code which only the developers are allowed to change. Users should download the ROMS code to their local machines using an '''svn client'''. Don't attempt to use a regular web browser to browse or download files from the SVN repository - there are much better tools for interacting with the code repository. See the notes below under the heading '''View the Repository'''.

We strongly recommend users always check out the current ''trunk'' version since this has the most recent updates and bug fixes. The ''tags'' version is kept largely as an historical record of stable releases at the conclusion of major code upgrades.

Below is a general description of how subversion works. Please look at the [http://svnbook.red-bean.com/ svn book] for more detailed information. Brief instructions for using two GUI clients (one for PC and one for Linux) we've tried are included.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you checkout the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a Subversion repository, '''svn client''' software must be installed on your local machine. If you are compiling subversion on your own be sure to build it with '''SSL support''' or you will not be able to download the ROMS source code. Most Linux distributions come with subversion (the command name is '''svn'''), so shell commands may be used without installing additional software. If your username on your local computer is not the same as your ROMS username you will need to pass the '''--username <username>''' option to '''svn'''; an example is given below. The general form of subversion commands is:

<span class="red">svn</span> action from to {optional_qualifiers}

To check-out the files from the ROMS repository '''trunk''' (latest version), enter (notice https instead of http):

<span class="red">svn checkout <nowiki>https://www.myroms.org/svn/src/trunk</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If your '''username''' on your local computer is not the same as your ROMS username you will need to pass the '''--username''' option to '''svn''':

<span class="red">svn checkout --username joe_roms <nowiki>https://www.myroms.org/svn/src/trunk</nowiki> MyDir</span>

===Files for test cases and other repositories===
There is an additional repository that provides idealized [[Test Cases]] and realistic applications to introduce you to ROMS many capabilities. You may use these examples as a guideline in setting up your own application. Currently, this repository requires '''~388MB''' of disk space. This is because the realistic applications require several input NetCDF files. To checkout this repository, enter:

<span class="red">svn checkout <nowiki>https://www.myroms.org/svn/src/test</nowiki> MyTest</span>

Similarly, to check-out the files from the ROMS repository '''matlab''' (official scripts), enter:

<span class="red">svn checkout <nowiki>https://www.myroms.org/svn/src/matlab</nowiki> MyDir</span>

To check-out the files from the ROMS repository '''plot''' (NCAR's library-based plotting package), enter:

<span class="red">svn checkout <nowiki>https://www.myroms.org/svn/src/plot</nowiki> MyDir</span>

To check-out the files from the ROMS repository '''branches''', '''plot''', '''matlab''', '''tags''', '''test''', and '''trunk''', (this is not recommended because it takes a long time to complete) enter:

<span class="red">svn checkout <nowiki>https://www.myroms.org/svn/src</nowiki> MyDir</span>

You only check out once, after that, a hidden directory called '''.svn''' exists to keep track of the source, destination and a bunch of other information. Your username and password will also be saved. For more detail on command line use and syntax, see the [http://svnbook.red-bean.com/ svn book].

Several GUI front-ends to subversion exist, allowing the user to have visual prompts to help them manage their files.

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

<span class="red">svn update</span>

Subversion will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

===Code changes===

As you use ROMS, you may find yourself adding new files and new chunks of code to existing files. Unless you are a developer with write access to the repository at www.myroms.org, you have no easy way to save your changes within the svn framework, since any one directory can only point to one repository. To keep getting updates from the trunk, you must keep using the svn server at myroms.org. At the very least, saving a tarball before fetching major updates is a prudent step. A more organized approach to revision control for your own files can be achieved by using [http://git-scm.com/ git].

===Conflicts===

Sometimes when you make changes to your copy of the ROMS code, those changes will conflict with changes made to the repository (the code that resides at <span class="forestGreen"><nowiki>https://www.myroms.org/svn/src/trunk</nowiki></span>). This means that the changes from the server overlapped with your own, and now you have to manually choose between them.

Whenever a conflict occurs, three things typically occur to assist you in resolving that conflict:
:*Subversion halts during the update, offering you several choices, and remembers that the file is in a state of conflict if you don't clear it right then.
:*If Subversion considers the file to be mergeable, it places conflict markers (special strings of text which delimit the “sides” of the conflict, usually "<" and ">" characters) into the file to visibly demonstrate the overlapping areas.
:*For every conflicted file, Subversion places three extra unversioned (not under Subversion control) files in your working copy:
::*'''filename.mine:''' This is your file as it existed in your working copy (local copy) before you updated your working copy. This file has only your latest changes in it. (If Subversion considers the file to be unmergeable, then the .mine file isn't created, since it would be identical to the working file.)
::*'''filename.rOLDREV:''' This is the file that was the '''BASE''' revision before you updated your working copy. That is, the file that you checked out before you made your latest edits.
::*'''filename.rNEWREV:''' This is the file that your Subversion client just received from the server when you updated your working copy. This file corresponds to the HEAD (latest) revision of the repository.

For example, let's say you checked out revision 280 and made some changes to <span class="forestGreen">User/Functionals/ana_hmixcoef.h</span>. Now you want to update your ROMS source code to take advantage of a new algorithm but when you run <span class="red">svn update</span> your <span class="forestGreen">ana_hmixcoef.h</span> is now in conflict with the new version in the repository.

:<div class="box">> <span class="red">svn update</span><br>U Version<br>U ROMS/Modules/mod_mixing.F<br>U ROMS/Functionals/ana_hmixcoef.h<br>C User/Functionals/ana_hmixcoef.h<br>Updated to revision 291.<br><br>></div>

The above is with an older version of svn. The latest, greatest does this:

:<div class="box">Conflict discovered in 'ROMS/Utility/inp_par.F'.<br>Select: (p) postpone, (df) diff-full, (e) edit,<br> (mc) mine-conflict, (tc) theirs-conflict,<br> (s) show all options:</div>

Selecting (p) will behave as the old version.

If you get a conflict, you need to do one of three things:
:*Merge the conflicted text “by hand” by examining and editing the conflict markers within the file.
:*Copy one of the temporary files on top of your working file.
:*Run <span class="red">svn revert <filename></span> to throw away all of your local changes.

Once you've resolved the conflict, you need to let Subversion know by running <span class="red">svn resolved</span>. This removes the three temporary files and Subversion no longer considers the file to be in a state of conflict. More on this below.

====Merging Conflicts by Hand====

{{note}}'''Note:''' We recommend that you make a backup of the '''<filename>.mine''' file before you begin to be certain you don't accidentally lose your modifications.

To merge your changes with those from the latest revision in the repository, open <span class="forestGreen">User/Functionals/ana_hmixcoef.h</span> in your favorite editor and look for a string of "<" characters. You should see something like this:

:<div class="box"><<<<<<< .mine<br>#ifndef DISTRIBUTE<br> IF (Lanafile.and.(tile.eq.0)) THEN<br>#else<br> IF (Lanafile) THEN<br>#endif<br>=======<br>#ifdef DISTRIBUTE<br> IF (Lanafile) THEN<br>#else<br> IF (Lanafile.and.(tile.eq.0)) THEN<br>#endif<br>>>>>>>> .r291</div>

After comparing the two code blocks (separated by the "=" symbol), you decide that you prefer the logic from the repository so you remove the conflict markers and your code so the section now looks like this:

:<div class="box">#ifdef DISTRIBUTE<br> IF (Lanafile) THEN<br>#else<br> IF (Lanafile.and.(tile.eq.0)) THEN<br>#endif</div>

Now you can save the file and let Subversion know that you have resolved the conflict:

:<div class="box">> <span class="red">svn resolved User/Functionals/ana_hmixcoef.h</span><br>Resolved conflicted state of 'User/Functionals/ana_hmixcoef.h'</div>

====Copying a File Onto Your Working File====

{{note}}'''Note:''' We recommend that you make a backup of the '''<filename>.mine''' file before you begin to be certain you don't accidentally lose your modifications.

If you get a conflict and decide that you want to throw out your changes, you can merely copy one of the temporary files created by Subversion over the file in your working copy. Let's say you want to use the new revision from the repository:

:<div class="box">> <span class="red">cd User/Functionals</span><br>> <span class="red">ls ana_hmixcoef.h*</span><br>ana_hmixcoef.h ana_hmixcoef.h.mine ana_hmixcoef.h.r280 ana_hmixcoef.h.r291<br><br>> <span class="red">cp ana_hmixcoef.h.r291 ana_hmixcoef.h</span><br>> <span class="red">svn resolved ana_hmixcoef.h</span><br>Resolved conflicted state of 'ana_hmixcoef.h'</div>

====Punting: Using <span class="red">svn revert</span>====

{{note}}'''Note:''' We recommend that you make a backup of the '''<filename>.mine''' file before you begin to be certain you don't accidentally lose your modifications.

If you get a conflict, and upon examination decide that you want to throw out your changes and start your edits again, just revert your changes:

:<div class="box">> <span class="red">cd User/Functionals</span><br>> <span class="red">svn revert ana_hmixcoef.h</span><br>Reverted 'ana_hmixcoef.h'</div>

Note that when you revert a conflicted file, you don't have to run <span class="red">svn resolved</span>.

==Using GUI SVN Clients==
{{warning}}'''WARNING:''' It is strongly suggested that you checkout the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

Most of the GUI '''svn clients''' are very similar to each other. In this example we will use eSVN, although several users have reported better luck with the free "Foundation Version" of [http://www.syntevo.com/smartsvn/index.jsp/ SmartSVN].

===View the Repository===
The easiest way to view a repository is to simply look at it with [http://trac.edgewall.org/ Trac] in a web browser. Enter the URL:

https://www.myroms.org/projects/src/browser

You will be prompted for your ROMS username and password. This is also a good way to see individual files in the repository without actually downloading them.

To view the repository with eSVN, Click on the file menu and choose "Browse Repository" and enter repository URL:

[[Image:esvn_browser.jpg]]

Click "Browse" and you should be able to look at the '''branches''', '''tags''', and '''trunk'''. You can navigate the repository with a web browser as described above, but cannot do any of the version control stuff that follows.

===Checkout a Copy of the Trunk===

This process brings a copy of the main code from the repository to your computer and places it under svn control. "Local Path" tells eSVN where you would like your copy of the source code to be located.

[[Image:esvn_checkout.jpg]]

You should see a bunch of text stream by like:

svn checkout <nowiki>https://www.myroms.org/svn/src/trunk</nowiki> /home/ocean/src --username ocean
--password ****** --non-interactive
A /home/ocean/src/ROMS
A /home/ocean/src/ROMS/License_ROMS.txt
A /home/ocean/src/ROMS/Tangent
A /home/ocean/src/ROMS/Tangent/tl_rho_eos.F
A /home/ocean/src/ROMS/Tangent/tl_uv3dmix.F
A /home/ocean/src/ROMS/Tangent/tl_def_his.F
...
A /home/ocean/src/Compilers/UNICOS-mk-f90.mk
A /home/ocean/src/Compilers/CYGWIN-g95.mk
A /home/ocean/src/Compilers/OSF1-f90.mk
A /home/ocean/src/Compilers/Darwin-ifort.mk
A /home/ocean/src/Compilers/AIX-xlf.mk
A /home/ocean/src/Compilers/SunOS-f95.mk
A /home/ocean/src/makefile
Checked out revision 11.
successfully (0)

Source Code

2025-01-17T16:43:09Z

Robertson:

<div class="title">Regional Ocean Modeling System (ROMS)</div>

This official community version of '''ROMS''' is developed and maintained at Rutgers, The State University of New Jersey, New Brunswick, New Jersey, USA. Currently, '''ROMS''' source code is distributed using both [https://git-scm.com Git] and [https://subversion.apache.org Subversion]('''SVN'''). However, starting <span class="red">January 1, 2025</span>, '''ROMS''' will be exclusively distributed from its '''GitHub''' repository at [https://github.com/myroms https://github.com/myroms]. Updating the '''SVN''' repositories at [https://www.myroms.org https://www.myroms.org] will be discontinued but kept for historical reasons.

The '''ROMS''' framework is intended for users interested in ocean modeling. It requires an extensive background in ocean dynamics, numerical modeling, and computers to configure, run, and analyze the results to ensure you get the correct solution for your application. Therefore, we highly recommend users register at [https://www.myroms.org https://www.myroms.org] and set up a <span class="red">username</span> and <span class="red">password</span> to access the '''ROMS''' forum, email notifications for bugs/updates, technical support from the community, trac code maintenance history, tutorials, workshops, and publications. The User's '''ROMS''' forum has over 24,000 posts with helpful information. Technical support is limited to registered users. We do not provide user technical support, usage, or answers in '''GitHub'''.

The <span class="red">ROMS GitHub</span> repository also has its own wiki at [https://github.com/myroms/roms/wiki https://github.com/myroms/roms/wiki] with useful information and instructions. This '''GitHub''' version becomes the official '''Git''' repository for downloading, updating, improving, and correcting defects/bugs in the '''ROMS''' source code. Also, it is the version used in the <span class="red">ROMS-JEDI</span> interface hosted at [https://github.com/JCSDA-internal https://github.com/JCSDA-internal], which is currently private.

{{note}}<span class="mediumOrchid">Note:</span> '''GitHub''' nowadays requires two-factor authentication!

<center>[[Image:ROMS_circle_logo.png]]   [[Image:ROMS-JEDI_circle_logo.png]]</center>

==<span class="red">WARNING:</span> <span class="blue">The updating of ROMS svn repositories has been discontinued as of</span> <span class="red">January 1, 2025</span>==

Git

2025-01-17T16:34:12Z

Robertson:

<div class="title">Git</div>
'''ROMS''' source code is now exclusively distributed using [https://git-scm.com Git]. The '''GitHub''' repository includes the full history since 2002 of changes to the '''ROMS''' source code. There are command line and GUI Git clients available for nearly every operating system and a list of popular clients can be found [https://git-scm.com/downloads/guis here]. This page will help you get started with downloading '''ROMS''' with Git.

==Git Overview==
Git is a free and open-source distributed version control system designed to handle everything from small to massive projects quickly and efficiently. All the ROMS files are stored in a Git repository on <span class="twilightBlue">github.com/myroms</span>.

This Git repository is the official version of the code, which only developers are allowed to change. Users should download the ROMS code to their local machines using a '''git client'''. Don't attempt to use a regular web browser to browse or download files from the Git repository—there are much better tools for interacting with the code repository.

We strongly recommend users check out the current <span class="red">develop</span> branch, which has the most recent updates and bug fixes. The ''tags'' are mainly kept as a historical record of stable releases after major code upgrades.

If you are making changes yourself, keep them in a separate branch, leaving the <span class="red">develop</span> branch to track changes from the source. Please check the [https://github.com/myroms/roms/wiki/Useful-GIT-Commands ROMS GitHub Wiki] for useful Git commands.

Below is a general description of how Git works. For more detailed information, please review the [https://git-scm.com/book/en/v2 Pro Git] book. We have not tried GUI clients but may add brief how-tos for the most popular GUIs later.

==Configuring GIT==

Before downloading any of '''ROMS''' repositories, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading some '''''roms_test''''' input and observation NetCDF files. Otherwise, the '''Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box">more ~/.gitconf<br /><br /> [User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br /> [credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br /> [filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you clone the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a git repository, '''git client''' software must be installed on your local machine. Most Linux distributions come with git, so shell commands may be used without installing additional software. The general form of git commands is:

<span class="red">git</span> action <repository>

To check-out the files from the ROMS repository '''develop''' (latest version), enter (notice https instead of http):

> <span class="red">git clone <nowiki>https://github.com/myroms/roms.git</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If '''MyDir''' is omitted the code will be downloaded to a directory named '''roms'''. You only clone once because Git will keep track of the source, destination and a bunch of other information. For more detail on command line use and syntax, see the [https://git-scm.com/book/en/v2 Pro Git] book.

The idealized and realistic ROMS Test Cases and the Matlab processing software can be downloaded from:

> <span class="red">git clone <nowiki>https://github.com/myroms/roms_test.git </nowiki></span>
> <span class="red">git clone <nowiki>https://github.com/myroms/roms_matlab.git</nowiki></span>

We highly recommend that Users clone all the ROMS repositories from the same root directory in their computer and define the '''ROMS_ROOT_DIR''' variable in their computer shell login environment, specifying where the User cloned/downloaded the ROMS source code, Test Cases, and Matlab processing software. That is, use the value of pwd from the computer directory where you executed git clone https://github.com/myroms/roms.git

For bash shells:
export ROMS_ROOT_DIR=MyDownlodLocationDirectory

For csh/tcsh shells:
setenv ROMS_ROOT_DIR MyDownlodLocationDirectory

The build scripts will use this environmental variable when compiling any of the ROMS Test Cases without the need to customize the location of the ROMS source code. Also, it is used for loading the path of Matlab scripts in the startup.m configuration file.

===Git Hash Revision Information===
{{note}}'''Note:''' While this is not a required step, we highly recommend it to make bug reporting and tracking easier.

In order to get Git hash revision information in the header for each file, we have created a Git filter that will insert the git hash of the last time a file was modified, similar to the Subversion '''''$Id$''''' keyword. In order for the filter to work, it has to be explicitly enabled by the user. After the git clone command from above completes change into <span class="limeGreen">MyDir</span> and execute the following:

> <span class="red">git config filter.id.smudge ".git_filters/id.smudge %f"</span>
> <span class="red">git config filter.id.clean ".git_filters/id.clean %f"</span>

In order to get the Git Id hashes to fill in, you will need to force the smudge filter to run on the source code you have already downloaded. This takes several minutes but only needs to be done once.

For '''bash''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "$(git rev-parse --show-toplevel)"</span>

For '''csh/tcsh''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "`git rev-parse --show-toplevel`"</span>

If you look at the top of any ROMS source file, you will see the Git commit hash for the last time that file was modified. For example, the first few lines of <span class="limeGreen">ROMS/Modules/mod_ncparam.F</span> look like this:<div class="box">#include "cppdefs.h"<br /> MODULE mod_ncparam<br />!<br />!git $Id: caf970e3 2024-08-09 17:15:26 UTC arango $<br />!================================================== Hernan G. Arango ===<br />! Copyright (c) 2002-2024 The ROMS/TOMS Group !<br />! Licensed under a MIT/X style license !<br />! See License_ROMS.md !<br />!=======================================================================</div>

You can then use the 8 character (hexidecimal) Git hash to show more information about that code update:<div class="box">> <span class="red">git show --name-only caf970e3</span></div>

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

> <span class="red">git pull</span>

Git will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

==Managing Your Own Modifications==
This assumes that you have a fresh clone of the myroms repository on the '''develop''' branch. You want to keep '''develop''' as a pure copy of the source version and keep your own changes in say the '''arctic''' branch. Start by creating a branch and switching to it:

> <span class="red">git branch arctic<br /></span>> <span class="red">git checkout arctic</span>

Now you can make whatever modifications you like (and test them out). To see what changed, you can use <span class="red">git status</span> and <span class="red">git diff</span>. To save your changes, do a:

> <span class="red">git commit -a</span>

Though if you add new files you will have to <span class="red">git add</span> them first.

===Getting the Updates===
It is easy to fetch and merge the updates. Start by making sure your directory has been cleanly checked in with <span class="red">git status</span>. Then you can update your '''develop''' branch:

> <span class="red">git checkout develop</span><br />> <span class="red">git pull</span>

Then bring the changes into your '''arctic''' branch:

> <span class="red">git checkout arctic</span><br />> <span class="red">git merge develop</span>

This will bring in everything that changed since your last <span class="red">git pull</span>, so you might find it easier to keep on top of things by doing this often, not putting it off for years. You can also bring in changes one at a time with <span class="red">git cherry-pick</span>. Again, check the Pro Git book for much more information about all of these operations.

Note that this will save your '''arctic''' branch locally, under the .git directory. You can back this up as you would any other important files you have.

==Useful Git Commands==
{| border="1" cellspacing="0" cellpadding="5" align="center"
|'''Command'''
|'''Description'''
|-
| '''git add''' <_FileName_>
| Add a file to the repository staging area
|-
| '''git branch'''
| List all local branches (the asterisk denotes the current branch)
|-
| '''git branch -a'''
| List all local and remote branches
|-
| '''git branch -d''' <_BranchName_>
| Delete a local branch
|-
| '''git branch -D''' <_BranchName_>
| Delete a local branch forcefully
|-
| '''git branch -m''' <_OldName_> <_NewName_>
| Rename a local branch
|-
| '''git checkout -b''' <_BranchName_>
| Create a new local branch and switch to it
|-
| '''git checkout''' <_BranchName_>
| Switch to an existing local or remote branch
|-
| '''git clone''' <_RepositoryURL_>
| Clone a public repository
|-
| '''git commit -am''' "<_Message_>"
| Commit changes to all files
|-
| '''git commit -am''' "<_Message_>" '''-m''' "<_MessageURL_>"
| Commit changes to all files with two message lines
|-
| '''git commit --amend -m''' "<_NewMessage_>"
| Amend previous commit message
|-
| '''git diff'''
| Show all changes between HEAD and working branch
|-
| '''git diff''' <_FileName_>
| Show changes for a specific file between HEAD and working branch
|-
| '''git difftool -d''' <_BranchName1_> <_BranchName2_>
| Compare the difference between two branches with '''`KDIFF3`'''
|-
| '''git fetch'''
| Retrieve new work done by other people
|-
| '''git log'''
| View repository changes
|-
| '''git log --oneline'''
| Show the list of commits in one-line format
|-
| '''git log --summary'''
| View repository detailed changes
|-
| '''git merge --no-ff --no-commit''' <_BranchName_>
| Merge a branch into the active branch
|-
| '''git pull'''
| Update local repository or branch to the newest origin commit
|-
| '''git push origin --delete''' <_BranchName_>
| Delete a remote branch
|-
| '''git push -u origin''' <_BranchName_>
| Push changes to the remote repository (and remember the branch)
|-
| '''git reset --hard HEAD~1'''
| Undo last commit
|-
| '''git revert''' <_CommitID_>
| Revert commit changes
|-
| '''git status'''
| Check/display changes to the repository or particular branch
|-
| '''git tag'''
| List all tags
|}

==Complicated Commands==

If a branch is renamed in a repository on GitHub, the local clone on a computer needs to updated:<div class="box">git branch -m OldName NewName<br />git fetch origin<br />git branch -u origin/NewName NewName<br />git remote set-head origin -a</div>

How to merge changes in the '''feature/name1''' branch into the '''feature/name2''' branch:<div class="box">git checkout feature/name1<br />git pull<br /><br />git checkout feature/name2<br />git pull<br />git merge --no-ff --no-commit feature/name1<br />git commit -am "Merging feature/name1 into feature/name2" -m "MessageURL"<br />git difftool -d feature/name1 feature/name2<br />git push -u origin feature/name2</div>

Documentation Portal

2025-01-16T19:22:47Z

Robertson:

<div class="title">Documentation Portal</div>

{|class="main" width="99%"
|style="vertical-align:top;width:33%" | 
'''Technical Documentation'''
* [[Regional_Ocean_Modeling_System_(ROMS) | Introduction]]
* [[Equations of Motion]]
* [[Vertical S-coordinate]]
* [[Terrain-Following Coordinate Transformation]]
* [[Curvilinear Coordinates Transformation]]
* [[Fractional Coordinate System (ξ - η space)]]
* [[Time-stepping Schemes Review]]
* [[Numerical Solution Technique]]
* [[Nested Grids]]
* [[Boundary Conditions]]
* [[Horizontal Mixing]]
* [[Vertical Mixing Parameterizations]]
* [[Atmospheric Boundary Layer]]
* [[Bottom Boundary Layer]]
* [[Lagrangian Drifters]]
* [[Balance Term Diagnostics]]
* [[Biogeochemical Models]]
* [[Sediment Model]]
* [[Wave Model]]
* [[Atmospheric Model]]
* [[Model Coupling]]
** [[NUOPC_Cap | ROMS Native ESMF/NUOPC Cap]]
** [[NUOPC_Cap_UFS | ROMS Stand-Alone ESMF/NUOPC Cap]]
* [[Model Diagnostics]]
* [[Sea-Ice Model]]
* [[Variational Data Assimilation]]
* [[ROMS-JEDI_Intro| ROMS-JEDI]]
** [[ROMS-JEDI_Compiling | Compiling]]
** [[ROMS-JEDI_Implementation | Implementation]]
** [[ROMS-JEDI_Observations | Observations]]
** [[ROMS-JEDI_Unit_Test_Cases | Unit Test Cases]]
* [[IO | ROMS I/O]]
* [[Bibliography]]
|style="vertical-align:top;width:33%;border-left:1px solid #aaa;padding-left:1em"| 
'''User Guide'''
* [[Getting Started]]
* [[Tutorials]]
* [[Nesting Examples]]
* [[Frequently Asked Questions]]
* [[Source Code]]
** [[Git]]
** [[Modules]]
** [[Drivers]]
** [[Subroutines]]
** [[Variables]]
** [[Functionals]]
** [[Lateral Boundary Conditions]]
* [[C Preprocessor]]
** [[Options]]
** [[cppdefs.h]]
** [[globaldefs.h]]
* [[Compiling and Linking]]
** [[makefile]]
** [[build_roms]]
** [[my_build_paths]]
** [[make]]
** [[gmake]]
** [[MakeDepend]]
** [[External Libraries]]
* [[Parallelization]]
* [[Benchmarking]]
* [[Matlab Scripts]]
* [[Visualization]]
|style="vertical-align:top;width:33%;border-left:1px solid #aaa;padding-left:1em"| 
'''Examples and Applications'''
* [[Test Cases]]
* [[Model Coupling ESMF | Coupling Examples]]
* [[Lake_Jersey_Refinement_Example_Introduction | Nesting Examples]]
* [[Application Set-up]]
* [[Metadata Design]]
** [[YAML_Parser | YAML Parser]]
** [[coupling_esmf_atm.yaml]]
** [[roms_cmeps.yaml]]
** [[varinfo.yaml]]
* [[Grid Generation]]
* [[Initialization]]
* [[Forcing]]
** [[Atmospheric Forcing]]
** [[Tidal Forcing]]
** [[River Runoff]]
* [[Input Parameter Files]]
** [[roms.in]]
** [[s4dvar.in]]
** [[biology.in]]
** [[sediment.in]]
** [[floats.in]]
** [[stations.in]]
** [[ice.in]]
** [[swan.in]]
* [[Standard Output Files]]
|}
----

<center>[[Image:ROMS_circle_logo.png]]   [[Image:ROMS-JEDI_circle_logo.png]]</center>

Documentation Portal

2025-01-06T19:58:00Z

Robertson:

External Libraries

2025-01-06T19:38:46Z

Robertson:

<div class="title">External Libraries</div>

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

==ARPACK (ARnoldi PACKage)==
Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method ('''IRAM''') for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the '''BLAS''' and '''LAPACK''' libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

To obtain '''ARPACK''', execute the following:
<div class="box">> git clone https://github.com/myroms/roms_libs.git<br />> cd roms_libs/ARPACK</div>
To compile, customize '''ARmake.inc''' for the desired compiler and its flags for the serial ('''FC''' and '''FFLAGS''') and parallel ('''PFC''' and '''PFFLAGS''') versions of the library. Once configured type:<div class="box">> make lib plib</div> to build '''libarpack.a''' and '''libparpack.a'''. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the '''[[my_build_paths]]''' script.

{{note}} '''Note:''' Prior to SVN revision 1080 (July 23, 2021), '''ARPACK''' was distributed in the ROMS '''Lib/ARPACK''' subdirectory. For more information about why this was changed see TRAC ticket [https://www.myroms.org/projects/src/ticket/891 #891]

==ESMF (Earth System Modeling Framework)==
High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model ('''ESM''') components. It includes the '''NUOPC''' (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The '''NUOPC''' ''cap'' file is a Fortran module layer that sits on top of each '''ESM''' component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the '''NUOPC''' layer includes the native CoupleSets for '''ESM''' applications with nested grids. Various types of connections are possible, like '''nest-to-nest''', '''fine-to-coarse''', or '''coarse-to-fine'''. The user decides which nested grids to connect.

*Download from github:<div class="box">> git clone https://github.com/esmf-org/esmf.git esmf<br />> cd esmf<br />> git checkout tags/ESMF_8_1_0</div>

*Configure with environment variables:<div class="box">> export ESMF_DIR=/projects/dmcs_1/src/esmf<br />> export ESMF_COMM=mvapich2<br />> export ESMF_COMPILER=intel<br />> export ESMF_BOPT=O<br />> export ESMF_OPTLEVEL=3<br />> export ESMF_ABI=64<br />> export ESMF_INSTALL_PREFIX=/ESMF/install/dir ESMF_NETCDF="/path/to/netcdf/bin/nc-config"<br />> export ESMF_NETCDF_LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lhdf5_hl -lhdf5 -lm -lz -lcurl"<br />> export ESMF_NETCDF_LIBPATH="/path/to/netcdf/lib /path/to/hdf5/lib"<br />> export ESMF_SHARED_LIB_BUILD=OFF<br />> export ESMF_MPIRUN=/path/to/cluster/submission/script</div>{{note}} Note that ''ESMF_MPIRUN'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-np''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -np N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-np" ]<br />then<br /> echo "Usage: mpirun -np # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install:<div class="box">> make -j 10<br />> make all_tests<br />> make install</div>

==MCT (Model Coupling Toolkit)==

Open-source library distributed as a set of Fortran-90 modules for constructing a coupled model system from Earth System Model (ESM) components ([http://www-unix.mcs.anl.gov/mct Larson et al., 2004]). Each component model has its own grid and runs on its own set of processors. The MCT library provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred.

The MCT library is used to couple ROMS atmosphere models (like WRF) and wave models (like SWAN and WW3). For example, SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

MCT is available via github:

<div class="box">> git clone https://github.com/MCSclimate/MCT<br />> cd MCT<br />> ./configure<br />> make</div>

==NetCDF (Network Common Data Form)==

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides '''API''' in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses '''NetCDF''' for all its input and output data management. Its '''NetCDF3''' or '''NetCDF4''' type files can be processed using the standard library developed and distributed [http://doi.org/10.5065/D6H70CW6 UCAR/Unidata], the Parallel-IO ('''PIO''') developed at ANL/NCAR ([https://doi.org/10.1177/1094342011428143 Dennis ''et al''., 2012]; [https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE Hartnett and Edwards, 2021]), or the Software for Cashing Output and Reads for Parallel I/O ('''SCORPIO''') library available in [https://e3sm.org/scorpio-parallel-io-library E3SM]. We have migrated to the Fortran-90 '''API''' interface, which consists of a library and one or two module files. See the [[makefile]] and [[build_roms| build script]] for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

===Building NetCDF-4/HDF5===
NetCDF-4 uses HDF5 for it's underlying data format so you first need to build HDF5. We recommend 1.10.6 because 1.10.7 causes errors in the NetCDF-C test suite. Here we will describe how to build parallel versions for use on an HPC cluster. We will note what changes are necessary if you are building for a system that doesn't have parallel I/O.

====Building HDF5====
* First, download '''HDF5''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.6/src/hdf5-1.10.6.tar.gz</div>or<div class="box">> git clone https://github.com/HDFGroup/hdf5<br />> cd hdf5<br />> git checkout tags/hdf5-1_10_6<br />> autoconf -i</div>

* Configure '''HDF5''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort \<br /> RUNPARALLEL="srun --nodes=1 --ntasks=6 --cpus-per-task=1 --time=00:13:00 --export=ALL " \<br /> --prefix=/path/to/parallel/hdf5/1.10.6 --enable-static-exec --enable-shared=no \<br /> --enable-parallel --disable-silent-rules --enable-fortran</div>{{note}} '''Note:''' ''RUNPARALLEL'' is only needed if your system uses a scheduler to run MPI jobs. The example here is for a SLURM system but you will need to adjust this for your cluster.<br /><br />{{note}} '''Note:''' To compile '''serial''', remove ''RUNPARALLEL=...'' and ''--enable-parallel'' and change the CC, CXX, and FC to the corresponding serial compilers (e.g. gcc, g++, and gfortran).<br /><br />{{note}} '''Note:''' If you would like '''shared libraries''' to be built, also remove ''--enable-shared=no''.<br /><br />

* Build, test, and install '''HDF5''':<div class="box">> make -j 5<br />> make all_tests<br />> make install</div>

====Building NetCDF-C====
The C and Fortran APIs for NetCDF are in separate packages and the C version must be built first.

* Download '''NetCDF-C''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-c-4.7.4.tar.gz</div>or<div class="box">> git clone https://github.com/Unidata/netcdf-c<br />> cd netcdf-c<br />> git checkout tags/v4.7.4<br />> autoconf -i</div>

* Configure '''NetCDF-C''':<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>{{note}} '''Note:''' To compile '''serial''', remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

====Building NetCDF-Fortran====
Now we can build the Fortran API.

* Download '''NetCDF-Fortran''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.5.3.tar.gz</div>or<div class="box">git clone https://github.com/Unidata/netcdf-fortran<br />> cd netcdf-fortan<br />> git checkout tags/v4.5.3<br />> autoconf -i</div>

* There is a bug/typo in '''configure''' that makes ''nf-config'' report the wrong linking information to ROMS. To fix the error, edit the '''configure''' file (and '''configure.ac''' if you plan to run ''autoconf -i'' again) and change the line:<div class="box">NC_FLIBS="-lnetcdff $NC_FLIBS"</div>to<div class="box">NC_FLIBS="-lnetcdff $NC_LIBS"</div> more details can be found [https://github.com/Unidata/netcdf-fortran/issues/270 here].<br /><br />

* Configure '''NetCDF-Fortran'''. Note that we set ''prefix'' to the same as '''NetCDF-C''' so the C and Fortran libraries live together:<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 FFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include -I/path/to/parallel/netcdf/4.7.4/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib -L/path/to/parallel/netcdf/4.7.4/lib" \<br /> LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' should point to the submision script you created while building '''NetCDF-C''' and is only needed for running test on clusters that use scheduling systems like SLURM.<br /><br />{{note}} '''Note:''' To compile serial remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* If you are running on clusters that use scheduling systems like SLURM, you will need to edit 4 testing scripts replacing all occurrences of '''mpiexec''' with the full path to the submision script you created for '''NetCDF-C''':<div class="box">> vi examples/F90/run_f90_par_examples.sh<br />> vi nf03_test4/run_f90_par_test.sh<br />> vi nf_test4/run_f77_par_test_03.sh<br />> vi nf_test4/run_f77_par_test.sh</div>

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

==PIO (Parallel I/O)==

Open-source parallel library for reading and writing distributed arrays to several scientific data formats like '''pNetCDF''', '''NetCDF3''', and '''NetCDF4/HDF5'''. '''PIO''' provides the flexibility to regulate the number of I/O tasks through data rearrangement between computational and I/O processes to improve performance and memory usage. It allows both '''synchronous''' and '''asynchronous''' I/O. In '''synchronous mode''', a subset of processes performs both I/O and computations (MPI intra-communications). Alternatively, in '''asynchronous mode''', the I/O is carried out by a set of disjointed and dedicated processes (MPI inter-communications).

The '''PIO''' or '''SCORPIO''' library is intended for ROMS Message Passing Interface (MPI) applications running on a large number of processes in an HPC computer with a '''Parallel File System''' (Lustre, GPFS, and so on). It uses the '''MPI-IO''' interface to facilitate the partitioning of data across computational or dedicated I/O processes. Both [https://github.com/NCAR/ParallelIO PIO] and [https://github.com/E3SM-Project/scorpio SCORPIO] will work with ROMS [https://www.myroms.org/projects/src/ticket/884 svn revision 1064] (May 10, 2021) or newer but we recommend '''PIO''' for its better performance. We recommend you read the manuals for all the build options, but below we show an example build process for both '''PIO''' and '''SCORPIO'''.

===Building PIO (autotools)===

'''PIO''' can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for '''PIO''', PnetCDF is optional. {{note}}However, the ROMS build system assumes that '''PIO''' is built with PnetCDF support and will need customization if '''PIO''' is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

'''PIO''' recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the '''PIO''' library but ROMS Parallel I/O is not currently implemented this way. In order to build '''PIO''' with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The '''PIO''' library can be built with the classic Autotools or Cmake. For '''PIO''' we show the Autotools procedure but will show Cmake for '''SCORPIO''' in the next section.

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_5_4<br />> autoconf -i</div>

*If you downloaded the gzipped tarball, you should run ''make clean'' because some Fortran .mod files are included and need to be removed:<div class="box">> make clean</div>

*Configure '''PIO''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/netcdf/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \<br /> LDFLAGS="-L/path/to/netcdf/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \<br /> LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz" \<br /> --with-mpiexec=/path/to/cluster/submission/script \<br /> --prefix=/pio/install/path --enable-fortran --disable-timing \<br /> --disable-silent-rules --disable-shared</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install '''PIO''':<div class="box">> make -j 5<br />> make check<br />> make install</div>{{warning}} '''Warning:''' If you are using the Intel compilers, you will probably need to remove "'''test_darray_async_many'''" from line 50 of '''test/cunit/run_tests.sh''' for ''make check'' to complete without errors. Details [https://github.com/NCAR/ParallelIO/issues/1867 here].

===Building SCORPIO (Cmake)===

{{note}} '''Note:''' '''SORPIO''' recently released version 1.2.0 and 1.2.1 but ROMS has not been tested with these versions. It is likely that the new version will work fine with ROMS but use these new versions at your own risk.

'''SCORPIO''' can use NetCDF (version 4.3.3+), PnetCDF (version 1.6.0+) or ADIOS (version 2.6.0+) for I/O. {{note}} ADIOS is not implemented in ROMS.

Ideally, the NetCDF version should be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. Additionally, HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP. If your NetCDF is built with DAP support, you will need to edit '''cmake/TryNetCDF_DAP.c''', changing line 8 from <div class="box">#if NC_HAS_DAP==1</div> to <div class="box">#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1</div>

*Download the '''SCORPIO''' source code:<div class="box">> git clone https://github.com/E3SM-Project/scorpio<br />> cd scorpio<br />> git checkout tags/scorpio-v1.1.6<br /></div>If your NetCDF is built with DAP support, you will need to edit '''cmake/TryNetCDF_DAP.c''', changing line 8 from <div class="box">#if NC_HAS_DAP==1</div> to <div class="box">#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1</div>

*Configure '''SCORPIO''':<div class="box">> cd ..<br />> mkdir build_scorpio<br />> cd build_scorpio<br />> CC=mpicc CXX=mpicxx FC=mpifort cmake \<br /> -DNetCDF_PATH=/path/to/netcdf \<br /> -DPnetCDF_PATH=/path/to/pnetcdf \<br /> -DMPIEXEC=/path/submission/scipt \<br /> -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \<br /> -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \<br /> -DCMAKE_INSTALL_PREFIX=/scorpio/install/path ../scorpio</div>{{note}} Note that ''-DMPIEXEC'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Compile, test, and install:<div class="box">> make -j 6<br />> make -j 6 tests<br />> ctests<br />> make install</div>

Getting Started

2024-12-27T20:46:34Z

Robertson:

<div class="title">Getting Started</div>
ROMS is a very complex model with many options and capabilities. ROMS is composed of many Fortran files ('''.F'''), a few header files ('''.h'''), various input script files ('''.in'''), a metadata variable definition file ([[varinfo.yaml]]), and a single [[makefile]]. The ROMS algorithms are distributed with the following directory structure:

<span class="twilightBlue">Compilers/ make configuration files
Data/ Input data root directory
/ROMS/ ROMS data root directory
/CDL ROMS NetCDF metadata design
Master/ Main standalone and coupling programs
ROMS/ ROMS root directory
/Adjoint/ Adjoint model
/Biology Adjoint biology/ecosystem models
/Bin Executable scripts
/Drivers Computational drivers
/External Standard input scripts
/Functionals Analytical expression header files
/Include Test cases configuration header files
/Modules Declaration modules
/Nonlinear/ Nonlinear model
/BBL Bottom Boundary Layer formulation
/Biology Nonlinear biology/ecosystem models
/SeaIce SeaIce model
/Sediment Nonlinear sediment transport model
/Obsolete Discontinued files
/Programs Support programs
/Representer/ Representer model
/Biology Representer biology/ecosystem models
/Tangent/ Tangent linear model
/Biology Tangent linear biology/ecosystem models
/Utility Generic utility files
User/ ROMS User interface root directory
/External User standard input scripts
/Functionals User analytical expressions templates
/Include User application header files</span>

ROMS uses [[C-preprocessing]] extensively to activate and/or deactivate
the various numerical and physical algorithm [[Options | options]]. It also uses
[[NetCDF]] to manage input and output data streams. It is highly
recommended that first time users learn the basics of NetCDF
before working with ROMS.

==Basic Steps==

* Register at the ROMS website ([http://www.myroms.org www.myroms.org]) to get access to the algorithms and other user privileges. Select a '''username''' and '''password''' which will be used in the future to login and post messages on the ROMS forum, algorithm downloads, and contributing to wikiROMS.

* Make sure that you have the following required software in your computer before attempting to compile and run an application:
** [https://git-scm.com/ Git] client to download source
** Fortran 90 or Fortran 95 compiler
** [http://gcc.gnu.org/onlinedocs/cpp/ cpp] program for [[C-preprocessing]] ROMS source code.
** [http://www.gnu.org/software/make/ GNU make] version '''3.81''' or higher to compile ROMS
** [http://www.perl.org/ Perl] interpreter program
** [http://www.unidata.ucar.edu/software/netcdf/ NetCDF] library, with Fortran 90 interface
** Message Passing Interface (MPI) library to run in parallel on a distributed-memory system

* Use a [[Git]] client to '''clone''' the latest version of the ROMS '''develop''' branch. It is highly recommended that you obtain the code on the same system that you will be compiling and running the code on to avoid file format issues.

* We recommend that you use the included '''[[build Script|build script]]''' to compile and link ROMS. This will set up your build environment and execute the [[gmake|make]] command to build the default ROMS [[UPWELLING_CASE | upwelling]] application. This process allows you to avoid editing the [[makefile]]. Please visit the '''[[build Script|build script]]''' page for more detailed instructions.
** You can also type <span class="red">make</span> at the top of the directory structure where the [[makefile]] is located but we do not recommend this process because it requires changing the ROMS files which can cause conflicts when you update you ROMS code.

{{note}}'''Note:''' To speed compilation, you may want to add the '''-j <n>''' flag to the build command (i.e. '''build.sh -j 4''') where '''<n>''' is the number of processors you wish to compile with. Even single processor machines can benefit from the -j flag with '''<n> = 2'''.

{{note}}'''Note:''' To make sure your application can compile successfully, you might want to set '''USE_DEBUG''' to '''on''' in the [[build Script|build script]] since it will compile faster. Once your application can compile you can unset '''USE_DEBUG''' in order to create an optimized executable. Please visit the '''[[build Script|build script]]''' page for more information

{{note}}'''Note:''' In SVN revision 933 (January 26, 2019) all <span class="red">ocean_*.in</span> files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. If you are working with a ROMS release prior to revision 933 you will need to replace <span class="red">roms_upwelling.in</span> with <span class="red">ocean_upwelling.in</span> and <span class="red">romsS</span>, <span class="red">romsM</span>, or <span class="red">romsO</span> with <span class="red">oceanS</span>, <span class="red">oceanM</span>, or <span class="red">oceanO</span> in all commands below.

* To run ROMS in serial, just type: <div class="box"><span class="red">romsS < ROMS/External/roms_upwelling.in > & log &</span></div>or to run in parallel (distributed-memory) on two processors: <div class="box"><span class="red">mpirun -np 2 romsM ROMS/External/roms_upwelling.in > & log &</span></div>or to run in parrallel (shared-memory) on two processors:<div class="box"><span class="red">setenv OMP_NUM_THREAD 2<br>romsO < ROMS/External/roms_upwelling.in > & log &</span></div>Here, the the file <span class="red">ROMS/External/roms_upwelling.in</span> contains all the input parameters to required by this application. Visit [[roms.in]] for more information.

{{note}}Notice that in distributed-memory, the leading <span class="red"><</span> is omitted so all parallel threads can read and process this input script without any communications in between.

External Libraries

2024-12-27T20:46:18Z

Robertson:

<div class="title">External Libraries</div>

Depending on the ROMS configuration, several third-party libraries are required for linking and compiling an application.

==ARPACK (ARnoldi PACKage)==
Serial and parallel legacy libraries applied to solve large eigenvalue problems using either the Implicitly Restarted Arnoldi Method ('''IRAM''') for sparse matrices or the Lanczos algorithm for symmetric matrices. It includes a subset of the '''BLAS''' and '''LAPACK''' libraries. Some of its functions are used in 4D-Var and the adjoint-based stability analysis propagators.

To obtain '''ARPACK''', execute the following:
<div class="box">> git clone https://github.com/myroms/roms_libraries<br />> cd roms_libraries/ARPACK</div>
To compile, customize '''ARmake.inc''' for the desired compiler and its flags for the serial ('''FC''' and '''FFLAGS''') and parallel ('''PFC''' and '''PFFLAGS''') versions of the library. Once configured type:<div class="box">> make lib plib</div> to build '''libarpack.a''' and '''libparpack.a'''. Once the libraries are built you can move them anywhere you like to make it easier to tell ROMS where to find them through environmental variables or with the '''[[my_build_paths]]''' script.

{{note}} '''Note:''' Prior to SVN revision 1080 (July 23, 2021), '''ARPACK''' was distributed in the ROMS '''Lib/ARPACK''' subdirectory. For more information about why this was changed see TRAC ticket [https://www.myroms.org/projects/src/ticket/891 #891]

==ESMF (Earth System Modeling Framework)==
High-performance, open-source library for grid interpolation, remapping, and data exchange between coupled Earth System Model ('''ESM''') components. It includes the '''NUOPC''' (National Unified Operation Prediction Capability) layer that provides templates and protocols for sequential and concurrent coupling between ESM components. The '''NUOPC''' ''cap'' file is a Fortran module layer that sits on top of each '''ESM''' component that provides the protocols and methods to interact and share data in a coupled system.

ROMS supports version 8.0 or higher since the '''NUOPC''' layer includes the native CoupleSets for '''ESM''' applications with nested grids. Various types of connections are possible, like '''nest-to-nest''', '''fine-to-coarse''', or '''coarse-to-fine'''. The user decides which nested grids to connect.

*Download from github:<div class="box">> git clone https://github.com/esmf-org/esmf.git esmf<br />> cd esmf<br />> git checkout tags/ESMF_8_1_0</div>

*Configure with environment variables:<div class="box">> export ESMF_DIR=/projects/dmcs_1/src/esmf<br />> export ESMF_COMM=mvapich2<br />> export ESMF_COMPILER=intel<br />> export ESMF_BOPT=O<br />> export ESMF_OPTLEVEL=3<br />> export ESMF_ABI=64<br />> export ESMF_INSTALL_PREFIX=/ESMF/install/dir ESMF_NETCDF="/path/to/netcdf/bin/nc-config"<br />> export ESMF_NETCDF_LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl -lhdf5_hl -lhdf5 -lm -lz -lcurl"<br />> export ESMF_NETCDF_LIBPATH="/path/to/netcdf/lib /path/to/hdf5/lib"<br />> export ESMF_SHARED_LIB_BUILD=OFF<br />> export ESMF_MPIRUN=/path/to/cluster/submission/script</div>{{note}} Note that ''ESMF_MPIRUN'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-np''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -np N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-np" ]<br />then<br /> echo "Usage: mpirun -np # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install:<div class="box">> make -j 10<br />> make all_tests<br />> make install</div>

==MCT (Model Coupling Toolkit)==

Open-source library distributed as a set of Fortran-90 modules for constructing a coupled model system from Earth System Model (ESM) components ([http://www-unix.mcs.anl.gov/mct Larson et al., 2004]). Each component model has its own grid and runs on its own set of processors. The MCT library provides protocols for decomposition and allocation of model grids among different processors, efficient transfer of data fields between the different models, and interpolation algorithms for the data fields that are transferred.

The MCT library is used to couple ROMS atmosphere models (like WRF) and wave models (like SWAN and WW3). For example, SWAN sends to ROMS arrays of wave height, wavelength, average wave periods at the surface and near the bottom, wave propagation direction, near-bottom orbital velocity, and wave-energy dissipation rate. ROMS provides SWAN arrays of water depth, sea-surface elevation, and current velocity. Data exchange between SWAN and ROMS occurs at user-defined synchronization intervals. The frequency of data exchange depends on the application. If the exchanged fields fluctuate rapidly, more frequent synchronization is required. However, data exchange increases run time, so experience is required to determine the optimum synchronization interval for each application.

MCT is available via github:

<div class="box">> git clone https://github.com/MCSclimate/MCT<br />> cd MCT<br />> ./configure<br />> make</div>

==NetCDF (Network Common Data Form)==

Open-source, machine-independent libraries for storing self-describing scientific data into portable, scalable, appendable, and sharable computer files. It provides '''API''' in several computer languages to create files, to write data into a file, to read data from the file, and access information about the dataset dimensions, variables, and attributes.

ROMS uses '''NetCDF''' for all its input and output data management. Its '''NetCDF3''' or '''NetCDF4''' type files can be processed using the standard library developed and distributed [http://doi.org/10.5065/D6H70CW6 UCAR/Unidata], the Parallel-IO ('''PIO''') developed at ANL/NCAR ([https://doi.org/10.1177/1094342011428143 Dennis ''et al''., 2012]; [https://www.researchgate.net/publication/348169990_THE_PARALLELIO_PIO_CFORTRAN_LIBRARIES_FOR_SCALABLE_HPC_PERFORMANCE Hartnett and Edwards, 2021]), or the Software for Cashing Output and Reads for Parallel I/O ('''SCORPIO''') library available in [https://e3sm.org/scorpio-parallel-io-library E3SM]. We have migrated to the Fortran-90 '''API''' interface, which consists of a library and one or two module files. See the [[makefile]] and [[build_roms| build script]] for how to tell ROMS where these are located. You will have to use a copy of the library and module files that were compiled using the same compiler you are using to compile ROMS - this is especially true for the module files.

===Building NetCDF-4/HDF5===
NetCDF-4 uses HDF5 for it's underlying data format so you first need to build HDF5. We recommend 1.10.6 because 1.10.7 causes errors in the NetCDF-C test suite. Here we will describe how to build parallel versions for use on an HPC cluster. We will note what changes are necessary if you are building for a system that doesn't have parallel I/O.

====Building HDF5====
* First, download '''HDF5''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://support.hdfgroup.org/ftp/HDF5/releases/hdf5-1.10/hdf5-1.10.6/src/hdf5-1.10.6.tar.gz</div>or<div class="box">> git clone https://github.com/HDFGroup/hdf5<br />> cd hdf5<br />> git checkout tags/hdf5-1_10_6<br />> autoconf -i</div>

* Configure '''HDF5''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort \<br /> RUNPARALLEL="srun --nodes=1 --ntasks=6 --cpus-per-task=1 --time=00:13:00 --export=ALL " \<br /> --prefix=/path/to/parallel/hdf5/1.10.6 --enable-static-exec --enable-shared=no \<br /> --enable-parallel --disable-silent-rules --enable-fortran</div>{{note}} '''Note:''' ''RUNPARALLEL'' is only needed if your system uses a scheduler to run MPI jobs. The example here is for a SLURM system but you will need to adjust this for your cluster.<br /><br />{{note}} '''Note:''' To compile '''serial''', remove ''RUNPARALLEL=...'' and ''--enable-parallel'' and change the CC, CXX, and FC to the corresponding serial compilers (e.g. gcc, g++, and gfortran).<br /><br />{{note}} '''Note:''' If you would like '''shared libraries''' to be built, also remove ''--enable-shared=no''.<br /><br />

* Build, test, and install '''HDF5''':<div class="box">> make -j 5<br />> make all_tests<br />> make install</div>

====Building NetCDF-C====
The C and Fortran APIs for NetCDF are in separate packages and the C version must be built first.

* Download '''NetCDF-C''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-c-4.7.4.tar.gz</div>or<div class="box">> git clone https://github.com/Unidata/netcdf-c<br />> cd netcdf-c<br />> git checkout tags/v4.7.4<br />> autoconf -i</div>

* Configure '''NetCDF-C''':<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>{{note}} '''Note:''' To compile '''serial''', remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

====Building NetCDF-Fortran====
Now we can build the Fortran API.

* Download '''NetCDF-Fortran''' source code as a gzipped tarball or via git clone:<div class="box">> wget https://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-fortran-4.5.3.tar.gz</div>or<div class="box">git clone https://github.com/Unidata/netcdf-fortran<br />> cd netcdf-fortan<br />> git checkout tags/v4.5.3<br />> autoconf -i</div>

* There is a bug/typo in '''configure''' that makes ''nf-config'' report the wrong linking information to ROMS. To fix the error, edit the '''configure''' file (and '''configure.ac''' if you plan to run ''autoconf -i'' again) and change the line:<div class="box">NC_FLIBS="-lnetcdff $NC_FLIBS"</div>to<div class="box">NC_FLIBS="-lnetcdff $NC_LIBS"</div> more details can be found [https://github.com/Unidata/netcdf-fortran/issues/270 here].<br /><br />

* Configure '''NetCDF-Fortran'''. Note that we set ''prefix'' to the same as '''NetCDF-C''' so the C and Fortran libraries live together:<div class="box">> ./configure CC=mpicc FC=mpifort CFLAGS=-O3 FFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/parallel/hdf5/1.10.6/include -I/path/to/parallel/netcdf/4.7.4/include" \<br /> LDFLAGS="-L/path/to/parallel/hdf5/1.10.6/lib -L/path/to/parallel/netcdf/4.7.4/lib" \<br /> LIBS="-lnetcdf -lhdf5_hl -lhdf5 -lm -lz -lcurl" \<br /> --with-mpiexec=/path/to/job/submistion/script \<br /> --prefix=/path/to/parallel/netcdf/4.7.4 \<br /> --enable-parallel-tests --disable-shared --disable-silent-rules</div>{{note}} Note that ''--with-mpiexec'' should point to the submision script you created while building '''NetCDF-C''' and is only needed for running test on clusters that use scheduling systems like SLURM.<br /><br />{{note}} '''Note:''' To compile serial remove ''--with-mpiexec'' and ''--enable-parallel-tests'' and change the CC, and FC to the corresponding serial compilers (e.g. gcc and gfortran).<br /><br />{{note}} '''Note:''' If you would like shared libraries to be built, also remove ''--disable-shared''.<br /><br />

* If you are running on clusters that use scheduling systems like SLURM, you will need to edit 4 testing scripts replacing all occurrences of '''mpiexec''' with the full path to the submision script you created for '''NetCDF-C''':<div class="box">> vi examples/F90/run_f90_par_examples.sh<br />> vi nf03_test4/run_f90_par_test.sh<br />> vi nf_test4/run_f77_par_test_03.sh<br />> vi nf_test4/run_f77_par_test.sh</div>

* Build, test, and install '''NetCDF-C''':<div class="box">> make -j 5<br />> make check<br />> make install</div>

==PIO (Parallel I/O)==

Open-source parallel library for reading and writing distributed arrays to several scientific data formats like '''pNetCDF''', '''NetCDF3''', and '''NetCDF4/HDF5'''. '''PIO''' provides the flexibility to regulate the number of I/O tasks through data rearrangement between computational and I/O processes to improve performance and memory usage. It allows both '''synchronous''' and '''asynchronous''' I/O. In '''synchronous mode''', a subset of processes performs both I/O and computations (MPI intra-communications). Alternatively, in '''asynchronous mode''', the I/O is carried out by a set of disjointed and dedicated processes (MPI inter-communications).

The '''PIO''' or '''SCORPIO''' library is intended for ROMS Message Passing Interface (MPI) applications running on a large number of processes in an HPC computer with a '''Parallel File System''' (Lustre, GPFS, and so on). It uses the '''MPI-IO''' interface to facilitate the partitioning of data across computational or dedicated I/O processes. Both [https://github.com/NCAR/ParallelIO PIO] and [https://github.com/E3SM-Project/scorpio SCORPIO] will work with ROMS [https://www.myroms.org/projects/src/ticket/884 svn revision 1064] (May 10, 2021) or newer but we recommend '''PIO''' for its better performance. We recommend you read the manuals for all the build options, but below we show an example build process for both '''PIO''' and '''SCORPIO'''.

===Building PIO (autotools)===

'''PIO''' can use NetCDF (version 4.6.1+) and/or PnetCDF (version 1.9.0+) for I/O. NetCDF may be built with or without netCDF-4 features. NetCDF is required for '''PIO''', PnetCDF is optional. {{note}}However, the ROMS build system assumes that '''PIO''' is built with PnetCDF support and will need customization if '''PIO''' is built without it.

The NetCDF C library must be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP.

'''PIO''' recently introduced a NetCDF integration option that allows standard NetCDF library calls to be implemented with the '''PIO''' library but ROMS Parallel I/O is not currently implemented this way. In order to build '''PIO''' with NetCDF integration, you will need NetCDF version 4.7.4 or higher. However, the Fortran API integration has not been implemented in ROMS.

The '''PIO''' library can be built with the classic Autotools or Cmake. For '''PIO''' we show the Autotools procedure but will show Cmake for '''SCORPIO''' in the next section.

*First download the '''PIO''' source code as a gzipped tarball or via ''git clone'':<div class="box">> wget https://github.com/NCAR/ParallelIO/releases/download/pio2_5_4/pio-2.5.4.tar.gz</div>or<div class="box">> git clone https://github.com/NCAR/ParallelIO<br />> cd ParallelIO<br />> git checkout tags/pio2_5_4<br />> autoconf -i</div>

*If you downloaded the gzipped tarball, you should run ''make clean'' because some Fortran .mod files are included and need to be removed:<div class="box">> make clean</div>

*Configure '''PIO''':<div class="box">> ./configure CC=mpicc CXX=mpicxx FC=mpifort CFLAGS=-O3 FFLAGS=-O3 FCFLAGS=-O3 \<br /> CPPFLAGS="-I/path/to/netcdf/include -I/path/to/hdf5/include -I/path/to/pnetcdf/include" \<br /> LDFLAGS="-L/path/to/netcdf/lib -L/path/to/hdf5/lib -L/path/to/pnetcdf/lib" \<br /> LIBS="-lhdf5_hl -lhdf5 -lm -lcurl -lz" \<br /> --with-mpiexec=/path/to/cluster/submission/script \<br /> --prefix=/pio/install/path --enable-fortran --disable-timing \<br /> --disable-silent-rules --disable-shared</div>{{note}} Note that ''--with-mpiexec'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Build, test, and install '''PIO''':<div class="box">> make -j 5<br />> make check<br />> make install</div>{{warning}} '''Warning:''' If you are using the Intel compilers, you will probably need to remove "'''test_darray_async_many'''" from line 50 of '''test/cunit/run_tests.sh''' for ''make check'' to complete without errors. Details [https://github.com/NCAR/ParallelIO/issues/1867 here].

===Building SCORPIO (Cmake)===

{{note}} '''Note:''' '''SORPIO''' recently released version 1.2.0 and 1.2.1 but ROMS has not been tested with these versions. It is likely that the new version will work fine with ROMS but use these new versions at your own risk.

'''SCORPIO''' can use NetCDF (version 4.3.3+), PnetCDF (version 1.6.0+) or ADIOS (version 2.6.0+) for I/O. {{note}} ADIOS is not implemented in ROMS.

Ideally, the NetCDF version should be built with MPI, which requires that it be linked with an MPI-enabled version of HDF5. Optionally, NetCDF can be built with DAP support, which introduces a dependency on CURL. Additionally, HDF5, itself, introduces dependencies on LIBZ and (optionally) SZIP. If your NetCDF is built with DAP support, you will need to edit '''cmake/TryNetCDF_DAP.c''', changing line 8 from <div class="box">#if NC_HAS_DAP==1</div> to <div class="box">#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1</div>

*Download the '''SCORPIO''' source code:<div class="box">> git clone https://github.com/E3SM-Project/scorpio<br />> cd scorpio<br />> git checkout tags/scorpio-v1.1.6<br /></div>If your NetCDF is built with DAP support, you will need to edit '''cmake/TryNetCDF_DAP.c''', changing line 8 from <div class="box">#if NC_HAS_DAP==1</div> to <div class="box">#if NC_HAS_DAP==1 || NC_HAS_DAP2==1 || NC_HAS_DAP4==1</div>

*Configure '''SCORPIO''':<div class="box">> cd ..<br />> mkdir build_scorpio<br />> cd build_scorpio<br />> CC=mpicc CXX=mpicxx FC=mpifort cmake \<br /> -DNetCDF_PATH=/path/to/netcdf \<br /> -DPnetCDF_PATH=/path/to/pnetcdf \<br /> -DMPIEXEC=/path/submission/scipt \<br /> -DMPIEXEC_MAX_NUMPROCS=8 -DMPIEXEC_NUMPROC_FLAG=-n \<br /> -DPIO_FILESYSTEM_HINTS=gpfs -DPIO_ENABLE_TIMING=OFF -DPIO_ENABLE_TESTS=ON \<br /> -DCMAKE_INSTALL_PREFIX=/scorpio/install/path ../scorpio</div>{{note}} Note that ''-DMPIEXEC'' is only needed for running test on clusters that use scheduling systems like SLURM. Your submission script should take '''-n''', '''N''' (number of processes requested), and '''programName''' (in that order) as arguments to construct the appropriate command to run using the scheduling system on your cluster. A SLURM example is provided below:<div class="box"># 1) <this_script> -n N prog<br /># runs N copies of executable "prog" in parallel. The script must hide all<br /># of the system specific details, such as going through a queueing system<br /># and/or calling a system specific mpirun script with modified arguments.<br />#<br /># 2) The output of the application must arrive at the calling shell via<br /># stdout and stderr.<br /><br />if [ "$1" != "-n" ]<br />then<br /> echo "Usage: mpirun -n # prog"<br /> exit 1<br />fi<br /><br />num_procs=$2<br />shift 2<br />prog=$*<br /><br />echo "srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog"<br />srun --mpi=pmi2 --partition=my_partition --nodes=1 --ntasks=${num_procs} --cpus-per-task=1 --time=00:10:00 --exclusive --export=ALL $prog</div>

*Compile, test, and install:<div class="box">> make -j 6<br />> make -j 6 tests<br />> ctests<br />> make install</div>

Git

2024-11-13T16:59:20Z

Robertson:

<div class="title">Git</div>
ROMS source code is now distributed using [https://git-scm.com Git]. The [https://subversion.apache.org Subversion](SVN) repository is being phased out and will no longer be updated starting January 1<sup>st</sup>, 2025. The GitHub repository includes the full history of changes to the ROMS source code. There are command line and GUI Git clients available for nearly every operating system and a list of popular clients can be found [https://git-scm.com/downloads/guis here]. This page will help you get started with downloading ROMS with Git.

{{note}}'''Note:''' For instructions using the deprecated and soon to be discontinued myroms.org git repository, click [[Talk:Git|here]].

{{note}}'''Note:''' Details on using the deprecated and soon to be discontinued Subversion repository can be found [[Subversion|here]].

==Git Overview==
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. All the ROMS/TOMS files are stored in a Git repository on '''github.com'''.

This Git repository is the official version of the code which only the developers are allowed to change. Users should download the ROMS code to their local machines using a '''git client'''. Don't attempt to use a regular web browser to browse or download files from the Git repository - there are much better tools for interacting with the code repository.

We strongly recommend users always check out the current ''develop'' version since this has the most recent updates and bug fixes. The ''tags'' are kept largely as a historical record of stable releases at the conclusion of major code upgrades.

If you are making changes of your own, keep them in a separate branch, leaving the ''develop'' branch to track changes from the source. Git makes it ''so'' much easier to manage your own modifications than svn for those of us without write permission on the repository.

Below is a general description of how Git works. Please look at the [https://git-scm.com/book/en/v2 Pro Git] book for more detailed information. We have not yet tried any GUI clients but may add brief how-tos for the most popular GUIs at a later date.

==Configuring GIT==

Before downloading any of '''ROMS''' repositories, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading some '''''roms_test''''' input and observation NetCDF files. Otherwise, the '''Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box">more ~/.gitconf<br /><br /> [User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br /> [credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br /> [filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you clone the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a git repository, '''git client''' software must be installed on your local machine. Most Linux distributions come with git, so shell commands may be used without installing additional software. The general form of git commands is:

<span class="red">git</span> action <repository>

To check-out the files from the ROMS repository '''develop''' (latest version), enter (notice https instead of http):

> <span class="red">git clone <nowiki>https://github.com/myroms/roms.git</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If '''MyDir''' is omitted the code will be downloaded to a directory named '''roms'''. You only clone once because Git will keep track of the source, destination and a bunch of other information. For more detail on command line use and syntax, see the [https://git-scm.com/book/en/v2 Pro Git] book.

The idealized and realistic ROMS Test Cases and the Matlab processing software can be downloaded from:

> <span class="red">git clone <nowiki>https://github.com/myroms/roms_test.git </nowiki></span>
> <span class="red">git clone <nowiki>https://github.com/myroms/roms_matlab.git</nowiki></span>

We highly recommend that Users clone all the ROMS repositories from the same root directory in their computer and define the '''ROMS_ROOT_DIR''' variable in their computer shell login environment, specifying where the User cloned/downloaded the ROMS source code, Test Cases, and Matlab processing software. That is, use the value of pwd from the computer directory where you executed git clone https://github.com/myroms/roms.git

For bash shells:
export ROMS_ROOT_DIR=MyDownlodLocationDirectory

For csh/tcsh shells:
setenv ROMS_ROOT_DIR MyDownlodLocationDirectory

The build scripts will use this environmental variable when compiling any of the ROMS Test Cases without the need to customize the location of the ROMS source code. Also, it is used for loading the path of Matlab scripts in the startup.m configuration file.

===Git Hash Revision Information===
{{note}}'''Note:''' While this is not a required step, we highly recommend it to make bug reporting and tracking easier.

In order to get Git hash revision information in the header for each file, we have created a Git filter that will insert the git hash of the last time a file was modified, similar to the Subversion '''''$Id$''''' keyword. In order for the filter to work, it has to be explicitly enabled by the user. After the git clone command from above completes change into <span class="limeGreen">MyDir</span> and execute the following:

> <span class="red">git config filter.id.smudge ".git_filters/id.smudge %f"</span>
> <span class="red">git config filter.id.clean ".git_filters/id.clean %f"</span>

In order to get the Git Id hashes to fill in, you will need to force the smudge filter to run on the source code you have already downloaded. This takes several minutes but only needs to be done once.

For '''bash''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "$(git rev-parse --show-toplevel)"</span>

For '''csh/tcsh''' shells execute:

> <span class="red">rm .git/index</span>
> <span class="red">git checkout HEAD -- "`git rev-parse --show-toplevel`"</span>

If you look at the top of any ROMS source file, you will see the Git commit hash for the last time that file was modified. For example, the first few lines of <span class="limeGreen">ROMS/Modules/mod_ncparam.F</span> look like this:<div class="box">#include "cppdefs.h"<br /> MODULE mod_ncparam<br />!<br />!git $Id: caf970e3 2024-08-09 17:15:26 UTC arango $<br />!================================================== Hernan G. Arango ===<br />! Copyright (c) 2002-2024 The ROMS/TOMS Group !<br />! Licensed under a MIT/X style license !<br />! See License_ROMS.md !<br />!=======================================================================</div>

You can then use the 8 character (hexidecimal) Git hash to show more information about that code update:<div class="box">> <span class="red">git show --name-only caf970e3</span></div>

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

> <span class="red">git pull</span>

Git will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

==Managing Your Own Modifications==
This assumes that you have a fresh clone of the myroms repository on the '''develop''' branch. You want to keep '''develop''' as a pure copy of the source version and keep your own changes in say the '''arctic''' branch. Start by creating a branch and switching to it:

> <span class="red">git branch arctic<br /></span>> <span class="red">git checkout arctic</span>

Now you can make whatever modifications you like (and test them out). To see what changed, you can use <span class="red">git status</span> and <span class="red">git diff</span>. To save your changes, do a:

> <span class="red">git commit -a</span>

Though if you add new files you will have to <span class="red">git add</span> them first.

===Getting the Updates===
It is easy to fetch and merge the updates. Start by making sure your directory has been cleanly checked in with <span class="red">git status</span>. Then you can update your '''develop''' branch:

> <span class="red">git checkout develop</span><br />> <span class="red">git pull</span>

Then bring the changes into your '''arctic''' branch:

> <span class="red">git checkout arctic</span><br />> <span class="red">git merge develop</span>

This will bring in everything that changed since your last <span class="red">git pull</span>, so you might find it easier to keep on top of things by doing this often, not putting it off for years. You can also bring in changes one at a time with <span class="red">git cherry-pick</span>. Again, check the Pro Git book for much more information about all of these operations.

Note that this will save your '''arctic''' branch locally, under the .git directory. You can back this up as you would any other important files you have.

==Useful Git Commands==
{| border="1" cellspacing="0" cellpadding="5" align="center"
|'''Command'''
|'''Description'''
|-
| '''git add''' <_FileName_>
| Add a file to the repository staging area
|-
| '''git branch'''
| List all local branches (the asterisk denotes the current branch)
|-
| '''git branch -a'''
| List all local and remote branches
|-
| '''git branch -d''' <_BranchName_>
| Delete a local branch
|-
| '''git branch -D''' <_BranchName_>
| Delete a local branch forcefully
|-
| '''git branch -m''' <_OldName_> <_NewName_>
| Rename a local branch
|-
| '''git checkout -b''' <_BranchName_>
| Create a new local branch and switch to it
|-
| '''git checkout''' <_BranchName_>
| Switch to an existing local or remote branch
|-
| '''git clone''' <_RepositoryURL_>
| Clone a public repository
|-
| '''git commit -am''' "<_Message_>"
| Commit changes to all files
|-
| '''git commit -am''' "<_Message_>" '''-m''' "<_MessageURL_>"
| Commit changes to all files with two message lines
|-
| '''git commit --amend -m''' "<_NewMessage_>"
| Amend previous commit message
|-
| '''git diff'''
| Show all changes between HEAD and working branch
|-
| '''git diff''' <_FileName_>
| Show changes for a specific file between HEAD and working branch
|-
| '''git difftool -d''' <_BranchName1_> <_BranchName2_>
| Compare the difference between two branches with '''`KDIFF3`'''
|-
| '''git fetch'''
| Retrieve new work done by other people
|-
| '''git log'''
| View repository changes
|-
| '''git log --oneline'''
| Show the list of commits in one-line format
|-
| '''git log --summary'''
| View repository detailed changes
|-
| '''git merge --no-ff --no-commit''' <_BranchName_>
| Merge a branch into the active branch
|-
| '''git pull'''
| Update local repository or branch to the newest origin commit
|-
| '''git push origin --delete''' <_BranchName_>
| Delete a remote branch
|-
| '''git push -u origin''' <_BranchName_>
| Push changes to the remote repository (and remember the branch)
|-
| '''git reset --hard HEAD~1'''
| Undo last commit
|-
| '''git revert''' <_CommitID_>
| Revert commit changes
|-
| '''git status'''
| Check/display changes to the repository or particular branch
|-
| '''git tag'''
| List all tags
|}

==Complicated Commands==

If a branch is renamed in a repository on GitHub, the local clone on a computer needs to updated:<div class="box">git branch -m OldName NewName<br />git fetch origin<br />git branch -u origin/NewName NewName<br />git remote set-head origin -a</div>

How to merge changes in the '''feature/name1''' branch into the '''feature/name2''' branch:<div class="box">git checkout feature/name1<br />git pull<br /><br />git checkout feature/name2<br />git pull<br />git merge --no-ff --no-commit feature/name1<br />git commit -am "Merging feature/name1 into feature/name2" -m "MessageURL"<br />git difftool -d feature/name1 feature/name2<br />git push -u origin feature/name2</div>

MediaWiki:Monobook.css

2024-10-23T02:47:22Z

Robertson: not switching to mathML so the indentation is no longer needed

/* CSS placed here will affect users of the Monobook skin */

.AdjointColor {
font-weight: bold;
color: #9900CC;
}
.TangentColor {
font-weight: bold;
color: #FF0000;
}

.aquamarine {
font-weight: bold;
color: #45AB74;
}
.darkTurquoise {
font-weight: bold;
color: #00CED1;
}
.forestGreen {
font-weight: bold;
color: #22AB22;
}
.limeGreen {
font-weight: bold;
color: #32CD32;
}
.darkGoldenrod {
font-weight: bold;
color: #BAA60B;
}
.peru {
font-weight: bold;
color: #CDA53F;
}
.sandyBrown{
font-weight: bold;
color: #F4A460;
}
.salmon {
font-weight: bold;
color: #FAA072;
}
.hotPink {
font-weight: bold;
color: #FF69B4;
}
.mediumOrchid {
font-weight: bold;
color: #BA55D3;
}

.red {
font-weight: bold;
color: #FF5555;
}
.orange {
font-weight: bold;
color: #FF6600;
}
.brown {
font-weight: bold;
color: #663333;
}
.green {
font-weight: bold;
color: #009900;
}
.olive {
font-weight: bold;
color: #929205;
}
.cyan {
font-weight: bold;
color: #02CBC9;
}
.blue {
font-weight: bold;
color: #0000FF;
}
.twilightBlue {
font-weight: bold;
color: #6666CC;
}
.purple {
font-weight: bold;
color: #9900CC;
}
.violet {
font-weight: bold;
color: #FF00FF;
}
.black {
font-weight: bold;
color: black;
}

.alphabet {
font-weight: bold;
font-family: "Courier New",monospace;
border: solid black 1px;
padding: 2px 6px 0 5px;
background-color: #F1E8CA;
}

.source {
background-color: #FFF;
}

div.para {
font-size: 10px;
line-height: 10px;
padding: 0;
margin: 0;
}

/* get the block equations out of the center*/

math[mode="display"], math[display="block"] {
text-align: left;
}
.mwe-math-mathml-display{
padding-left: 7em;
text-align: left;
}

.mwe-math-fallback-image-display{
padding-left: 10em;
text-align: left;
margin-left: 0 !important;
margin-right: 0 !important;
}

/* Support where MediaWiki lazy loaded images */
/*.mwe-math-mathml-inline ~ .lazy-image-placeholder,
.mwe-math-mathml-display ~ .lazy-image-placeholder,*/
/* Support where MediaWiki doesn't lazy load images */
/*.mwe-math-mathml-inline + .mwe-math-fallback-image-inline,
.mwe-math-mathml-display + .mwe-math-fallback-image-display {
display: none !important;
}*/

/* equation numbers */
table.eqno {
width: 100%;
}

table.eqno tr td:first-child + td {
text-align: right;
vertical-align: middle;
}

#bodyContent table {
background-color: transparent;
}

table.matlab td {
min-width: 95px;
padding-right: 15px;
vertical-align: top;
}

img.thumbborder {
border: 1px solid #dddddd;
}

/*#toolbar {float: left;}*/

#wikiPreview {
height: 400px;
overflow: auto !important;
padding-bottom: 10px;
margin-bottom: 10px;
}

div#column-content {
width: 100%;
height:100%;
float: none;
margin: 0 0 .6em 12.2em;
padding: 0;
}

#content {
margin: 0;
position: absolute;
top: 2.8em;
left: 12.2em;
right: 0;
bottom: 38px;
padding: 0 1em 1.5em 1em;
background: #f9f9f9 url('https://www.myroms.org/wiki/images/content-bg.jpg') 0 0 no-repeat fixed;
color: black;
overflow: auto;
border: 1px solid #aaa;
border-right: none;
border-bottom: none;
line-height: 1.5em;
/*height: 100%;*/
z-index: 2;
}

div#sidebar {
position: absolute;
left: 3px;
top: 175px;
}

body {
font: x-small sans-serif;
background: #f9f9f9 url('https://www.myroms.org/wiki/images/bg.jpg') 0 0 no-repeat;
color: black;
margin: 0;
padding: 0;
height: 100%;
max-height: 100%;
width: 100%;
overflow: hidden;
}

/* scale back up to a sane default */
div#globalWrapper {
font-size: 127%;
width: 100%;
height: 100%;
max-height: 100%;
margin: 0;
padding: 0;
overflow: hidden;
position: absolute;
}

a {
text-decoration: none;
color: #0098CD;
background: none;
}
a:visited {
color: #9932CC;
}
a.new {
color: #45AB74 !important;
}
a.new:visited {
color: #45AB74 !important;
}
#p-personal a.new {
color: #03EB13;
}
#p-personal a.new:visited {
color: #03EB13;
}

h1, h2, h3, h4, h5, h6 {
color: black;
background: none;
font-weight: bold;
margin: 0;
padding-top: .5em;
padding-bottom: .17em;
border-bottom: 2px solid #aaa;
}
h1 .editsection { font-size: 53%; font-weight: normal; }
h2 .editsection { font-size: 67%; font-weight: normal; }

/* I don't think we ever used this so I am replacing it with the more
** specific div.code for the light green background 2022-04-29 - Robertson
code {
background-color: transparent;
margin-left: 3em;
}*/
div.code {
white-space: pre;
font-family: monospace;
margin: 1em 0 1em 0;
padding: .5em;
border: 1px dashed #2f6fab;
color: black;
background-color: #F0FFF0;
line-height: 11.5pt;
}
div.box {
white-space: pre;
font-family: monospace;
margin: 1em 0 1em 0;
padding: .5em;
border: 1px dashed #2f6fab;
color: black;
background-color: transparent;
line-height: 11.5pt;
}
pre {
padding: 1em;
border: 1px dashed #2f6fab;
color: black;
background-color: transparent;
line-height: 1.1em;
}

.firstHeading {
display: none;
font-weight: bold;
margin-bottom: 0;
}

/*
** content styles
*/
.alphabetlist {
width:15px;
text-align: center;
}

/* thumbnails */
div.thumb {
margin-bottom: .5em;
border-style: solid;
border-color: white;
border-width: 0px;
width: auto;
}

div.tleft {
float: left;
margin-right: .5em;
border-width: 0;
}

.portlet h3 {
background: transparent;
padding: 0 1em 0 .5em;
display: inline;
height: 1em;
text-transform: none;
font-size: 91%;
font-weight: bold;
white-space: nowrap;
border: none;
}

.pBody {
font-size: 95%;
background-color: transparent;
color: black;
border-collapse: collapse;
border: 1px solid #000;
padding: 0 .8em .3em .5em;
}
#p-cactions .pBody {
border: none;
}
.pBody a {
font-weight: bold;
color: #000000;
}
.pBody a:visited {
font-weight: bold;
color: #000;
}

#lastmod {
color: #000000;
font-weight: bold;
}
#viewcount {
color: #000000;
font-weight: bold;
}
#privacy a{
color: #2E0000;
font-weight: bold;
}
#about a{
color: #2E0000;
font-weight: bold;
}
#disclaimer a{
color: #2E0000;
font-weight: bold;
}

.portlet ul {
line-height: 1.5em;
list-style-type: square;
list-style-image: none;
font-size: 95%;
}

/*
** the navigation portlet
*/

#p-navigation {
position: relative;
z-index: 3;
}

#p-navigation .pBody {
padding-right: 0;
}

#p-navigation a {
font-weight: bold;
display: block;
}

#p-navigation li.active a, #p-navigation li.active a:hover {
text-decoration: none;
font-weight: bold;
}

#p-personal li a {
text-decoration: none;
color: #13F3F3;
padding-bottom: .2em;
background: none;
}
#p-personal li a:hover {
background-color: black;
padding-bottom: .2em;
text-decoration: none;
}

/*
** footer
*/
div#footer {
width: 800px;
position: absolute;
bottom: 0;
left: 0;
overflow: auto;
background-color: transparent;
border-top: none;
border-bottom: none;
margin: .6em 0 0 12.2em;
padding: .4em 0 0 0;
text-align: center;
font-size: 90%;
z-index: 10;
}

div.bib { text-indent: -2em; margin-left: 2em; }

BODY.page-Bibliography p { text-indent: -2em; margin-left: 2em; }

.title {
font-size: 155%;
font-weight: bold;
margin-bottom: .5em;
padding-top: .5em;
padding-bottom: .17em;
border-bottom: 2px solid #aaa;
}

table.main {
background-color: transparent;
font-size: 13px;
}

#bodyContent {
font-size: 16px;
}

#bodyContent ol li {
padding-bottom: 10px;
}

#bodyContent ol li ul li {
padding-bottom: 0px;
}

/* Fix special pages output */
.mw-specialpages-table td {
padding-top: 1em;
}

div#p-personal {
padding-top: 8px;
}

div#p-logo {
padding-top: 8px;
}

#column-one{
padding-top: unset;
}

Equations of Motion

2024-10-21T15:03:14Z

Robertson: /* Vertical Boundary Conditions */

<div class="title">Equations of Motion</div>
__TOC__
The primitive equations in Cartesian coordinates are shown here. The momentum balance in the <math>x</math>- and <math>y</math>-directions are:

{| class="eqno"
|<math display="block">\frac{\partial u}{\partial t} + \vec{v} \cdot \nabla u - fv = - \frac{\partial \phi}{\partial x} - {\partial \over \partial z} \left( \overline{u'w'} - \nu {\partial u \over \partial z} \right)+ {\cal F}_u + {\cal D}_u </math>
|(1)
|}

{| class="eqno"
|<math display="block">\frac{\partial v}{\partial t} + \vec{v} \cdot \nabla v + fu = - \frac{\partial \phi}{\partial y} - {\partial \over \partial z} \left( \overline{v'w'} - \nu {\partial v \over \partial z} \right) + {\cal F}_v + {\cal D}_v </math>
|(2)
|}

The time evolution of a scalar concentration field, <math>C(x,y,z,t)</math> (e.g. salinity, temperature, or nutrients), is governed by the advective-diffusive equation:

{| class="eqno"
|<math display="block">\frac{\partial C}{\partial t} + \vec{v} \cdot \nabla C = -{\partial \over \partial z} \left( \overline{C'w'} - \nu_\theta {\partial C \over \partial z} \right) + {\cal F}_C + {\cal D}_C </math> 
|(3)
|}
The equation of state is given by:

{| class="eqno"
|<math display="block">\rho = \rho(T,S,P) </math>
|(4)
|}

In the Boussinesq approximation, density variations are neglected in the momentum equations except in their
contribution to the buoyancy force in the vertical momentum equation.
Under the hydrostatic approximation, it is further
assumed that the vertical pressure gradient balances the buoyancy
force:

{| class="eqno"
|<math display="block">\frac{\partial \phi}{\partial z} = -\frac{\rho g}{\rho_o} </math>
|(5)
|}

The final equation expresses the continuity equation
for an incompressible fluid:

{| class="eqno"
|<math display="block">\frac{\partial u}{\partial x} + \frac{\partial v}{\partial y} + \frac{\partial w}{\partial z} = 0 </math>
|(6)
|}

For the moment, the effects of forcing
and horizontal dissipation will be represented by the schematic terms <math>{\cal F}</math>
and <math>{\cal D}</math>, respectively. The horizontal and vertical mixing will
be described more fully in [[Horizontal Mixing]] and [[Vertical Mixing Parameterizations]].
The variables used are shown here:
{| border="1" cellspacing="0" cellpadding="5" align="center"
| <math>{\cal D}_u, {\cal D}_v, {\cal D}_C</math>
|style="width: 350px;"| horizontal diffusive terms
|-
| <math>{\cal F}_u, {\cal F}_v, {\cal F}_C</math>
| forcing terms
|-
| <math>f(x,y)</math>
| Coriolis parameter
|-
| <math>g</math>
| acceleration of gravity
|-
| <math>h(x,y)</math>
| bottom depth
|-
| <math>\nu, \nu_\theta</math>
| molecular viscosity and diffusivity
|-
| <math>K_m, K_C</math>
| vertical eddy viscosity and diffusivity
|-
| <math>P</math>
| total pressure <math>P \approx -\rho_o gz</math>
|-
| <math>\phi(x,y,z,t)</math>
| dynamic pressure <math>\phi = \left(P/\rho_o \right)</math>
|-
| <math>\rho_o + \rho(x,y,z,t)</math>
| total ''in situ'' density
|-
| <math>S(x,y,z,t)</math>
| salinity
|-
| <math>t</math>
| time
|-
| <math>T(x,y,z,t)</math>
| potential temperature
|-
| <math>u,v,w</math>
| the (<math>x,y,z</math>) components of vector velocity <math>\vec{v}</math>
|-
| <math>x,y</math>
| horizontal coordinates
|-
| <math>z</math>
| vertical coordinate
|-
| <math>\zeta(x,y,t)</math>
| the surface elevation
|}

In the ocean, vertical mixing due to molecular viscosity is extremely weak compared to the turbulent mixing, so the terms involving <math>\nu</math> and <math>\nu_\theta</math> can be neglected.

To close these equations, parameterizations of the Reynolds stresses and turbulent tracer fluxes are introduced as functions of the other variables and vertical turbulent eddy viscosity and eddy diffusivity coefficients, <math>K_M</math> and <math>K_C</math>, respectively.

{| class="eqno"
|<math display="block">\overline{u'w'} = -K_M {\partial u \over \partial z};\qquad
\overline{v'w'} = -K_M {\partial v \over \partial z};\qquad
\overline{C'w'} = -K_C {\partial C \over \partial z}. </math> 
|(7)
|}

An overbar represents a time average and a prime represents a fluctuation about the mean.

Details on ROMS options for computing <math>K_M</math> and <math>K_C</math> are described in [[Vertical Mixing Parameterizations]].

==Vertical Boundary Conditions==
The vertical boundary conditions can be prescribed as follows:

:top (<math>z = \zeta(x,y,t))</math>:

{| class="eqno"
|<math display="block">\begin{align}&K_m \, \frac{\partial u}{\partial z} = \tau^x_s (x,y,t) \\
&K_m \, \frac{\partial v}{\partial z} = \tau_s^y(x,y,t) \\
&K_C \, \frac{\partial C}{\partial z} = {Q_C \over \rho_o c_P} \\
&w = {\partial \zeta \over \partial t} \end{align}</math> 
|(8)
|}

:and bottom (<math>z = -h(x,y)</math>):

{| class="eqno"
|<math display="block">\begin{align}&K_m \, \frac{\partial u}{\partial z} = \tau_b^x (x,y,t) \\
&K_m \, \frac{\partial v}{\partial z} = \tau_b^y(x,y,t) \\
&K_C \, \frac{\partial C}{\partial z} = 0 \\
&-w + \vec{v} \cdot \nabla h = 0 \end{align}</math> 
|(9)
|}

The new variables above are:
{| border="1" cellspacing="0" cellpadding="5" align="center"
! Variable
! Description
|-
| <math>Q_C</math>
| surface concentration flux
|-
| <math>\tau_s^x , \tau_s^y</math>
| surface wind stress
|-
| <math>\tau_b^x , \tau_b^y</math>
| bottom stress
|-
|}

The surface boundary condition variables are defined in the table above.
Since <math>Q_T</math> is a strong function of the surface
temperature, we usually choose to compute <math>Q_T</math> using the surface
temperature and the atmospheric fields in an atmospheric bulk flux
parameterization. This bulk flux routine also computes the wind
stress from the winds.

On the variable bottom, <math>z = -h(x,y)</math>, the horizontal velocity components are constrained to
accommodate a prescribed bottom stress which is a choice of linear, quadratic, or a log layer, depending on the [[Options]]. The vertical scalar concentration fluxes may also be prescribed at the bottom, although they are usually set to zero.

==Horizontal Boundary Conditions==
As distributed, the model can easily be configured for a periodic
channel, a doubly periodic domain, or a closed basin. Code is also
included for open boundaries which may or may not work for your
particular application. Appropriate boundary conditions are
provided for <math>u,v,C</math>, and <math>\zeta</math>, as described in [[Boundary Conditions]].

The model domain is logically rectangular, but it is possible to mask out land areas on the boundary and in the interior. Boundary conditions on these masked regions are straightforward, with a choice of no-slip or free-slip walls.

If biharmonic friction is used, a higher order boundary condition must also be provided. The model currently has this built into the code where the biharmonic terms are calculated. The high order boundary conditions used for <math>u</math> are <math>\frac{\partial}{\partial x} \left(
\frac{h\nu}{mn} \frac{\partial ^2 u}{\partial x^2} \right) = 0</math> on the eastern and western boundaries and <math>\frac{\partial}{\partial y} \left(\frac{h\nu}{mn} \frac{\partial ^2 u}{\partial y^2} \right) = 0</math> on the northern and southern boundaries. The boundary conditions for <math>v</math> and <math>C</math> are similar. These boundary conditions were chosen because they preserve the property of no gain or loss of volume-integrated momentum or scalar concentration.

MediaWiki:Monobook.css

2024-10-21T15:00:52Z

Robertson:

/* CSS placed here will affect users of the Monobook skin */

.AdjointColor {
font-weight: bold;
color: #9900CC;
}
.TangentColor {
font-weight: bold;
color: #FF0000;
}

.aquamarine {
font-weight: bold;
color: #45AB74;
}
.darkTurquoise {
font-weight: bold;
color: #00CED1;
}
.forestGreen {
font-weight: bold;
color: #22AB22;
}
.limeGreen {
font-weight: bold;
color: #32CD32;
}
.darkGoldenrod {
font-weight: bold;
color: #BAA60B;
}
.peru {
font-weight: bold;
color: #CDA53F;
}
.sandyBrown{
font-weight: bold;
color: #F4A460;
}
.salmon {
font-weight: bold;
color: #FAA072;
}
.hotPink {
font-weight: bold;
color: #FF69B4;
}
.mediumOrchid {
font-weight: bold;
color: #BA55D3;
}

.red {
font-weight: bold;
color: #FF5555;
}
.orange {
font-weight: bold;
color: #FF6600;
}
.brown {
font-weight: bold;
color: #663333;
}
.green {
font-weight: bold;
color: #009900;
}
.olive {
font-weight: bold;
color: #929205;
}
.cyan {
font-weight: bold;
color: #02CBC9;
}
.blue {
font-weight: bold;
color: #0000FF;
}
.twilightBlue {
font-weight: bold;
color: #6666CC;
}
.purple {
font-weight: bold;
color: #9900CC;
}
.violet {
font-weight: bold;
color: #FF00FF;
}
.black {
font-weight: bold;
color: black;
}

.alphabet {
font-weight: bold;
font-family: "Courier New",monospace;
border: solid black 1px;
padding: 2px 6px 0 5px;
background-color: #F1E8CA;
}

.source {
background-color: #FFF;
}

div.para {
font-size: 10px;
line-height: 10px;
padding: 0;
margin: 0;
}

/* get the block equations out of the center*/

math[mode="display"], math[display="block"] {
text-align: left;
}
.mwe-math-mathml-display{
padding-left: 7em;
text-align: left;
}

.mwe-math-fallback-image-display{
padding-left: 10em;
text-align: left;
margin-left: 0 !important;
margin-right: 0 !important;
}

/* Support where MediaWiki lazy loaded images */
/*.mwe-math-mathml-inline ~ .lazy-image-placeholder,
.mwe-math-mathml-display ~ .lazy-image-placeholder,*/
/* Support where MediaWiki doesn't lazy load images */
/*.mwe-math-mathml-inline + .mwe-math-fallback-image-inline,
.mwe-math-mathml-display + .mwe-math-fallback-image-display {
display: none !important;
}*/

/* equation numbers */
table.eqno {
width: 100%;
padding-left: 1em;
}

table.eqno tr td:first-child + td {
text-align: right;
vertical-align: middle;
}

#bodyContent table {
background-color: transparent;
}

table.matlab td {
min-width: 95px;
padding-right: 15px;
vertical-align: top;
}

img.thumbborder {
border: 1px solid #dddddd;
}

/*#toolbar {float: left;}*/

#wikiPreview {
height: 400px;
overflow: auto !important;
padding-bottom: 10px;
margin-bottom: 10px;
}

div#column-content {
width: 100%;
height:100%;
float: none;
margin: 0 0 .6em 12.2em;
padding: 0;
}

#content {
margin: 0;
position: absolute;
top: 2.8em;
left: 12.2em;
right: 0;
bottom: 38px;
padding: 0 1em 1.5em 1em;
background: #f9f9f9 url('https://www.myroms.org/wiki/images/content-bg.jpg') 0 0 no-repeat fixed;
color: black;
overflow: auto;
border: 1px solid #aaa;
border-right: none;
border-bottom: none;
line-height: 1.5em;
/*height: 100%;*/
z-index: 2;
}

div#sidebar {
position: absolute;
left: 3px;
top: 175px;
}

body {
font: x-small sans-serif;
background: #f9f9f9 url('https://www.myroms.org/wiki/images/bg.jpg') 0 0 no-repeat;
color: black;
margin: 0;
padding: 0;
height: 100%;
max-height: 100%;
width: 100%;
overflow: hidden;
}

/* scale back up to a sane default */
div#globalWrapper {
font-size: 127%;
width: 100%;
height: 100%;
max-height: 100%;
margin: 0;
padding: 0;
overflow: hidden;
position: absolute;
}

a {
text-decoration: none;
color: #0098CD;
background: none;
}
a:visited {
color: #9932CC;
}
a.new {
color: #45AB74 !important;
}
a.new:visited {
color: #45AB74 !important;
}
#p-personal a.new {
color: #03EB13;
}
#p-personal a.new:visited {
color: #03EB13;
}

h1, h2, h3, h4, h5, h6 {
color: black;
background: none;
font-weight: bold;
margin: 0;
padding-top: .5em;
padding-bottom: .17em;
border-bottom: 2px solid #aaa;
}
h1 .editsection { font-size: 53%; font-weight: normal; }
h2 .editsection { font-size: 67%; font-weight: normal; }

/* I don't think we ever used this so I am replacing it with the more
** specific div.code for the light green background 2022-04-29 - Robertson
code {
background-color: transparent;
margin-left: 3em;
}*/
div.code {
white-space: pre;
font-family: monospace;
margin: 1em 0 1em 0;
padding: .5em;
border: 1px dashed #2f6fab;
color: black;
background-color: #F0FFF0;
line-height: 11.5pt;
}
div.box {
white-space: pre;
font-family: monospace;
margin: 1em 0 1em 0;
padding: .5em;
border: 1px dashed #2f6fab;
color: black;
background-color: transparent;
line-height: 11.5pt;
}
pre {
padding: 1em;
border: 1px dashed #2f6fab;
color: black;
background-color: transparent;
line-height: 1.1em;
}

.firstHeading {
display: none;
font-weight: bold;
margin-bottom: 0;
}

/*
** content styles
*/
.alphabetlist {
width:15px;
text-align: center;
}

/* thumbnails */
div.thumb {
margin-bottom: .5em;
border-style: solid;
border-color: white;
border-width: 0px;
width: auto;
}

div.tleft {
float: left;
margin-right: .5em;
border-width: 0;
}

.portlet h3 {
background: transparent;
padding: 0 1em 0 .5em;
display: inline;
height: 1em;
text-transform: none;
font-size: 91%;
font-weight: bold;
white-space: nowrap;
border: none;
}

.pBody {
font-size: 95%;
background-color: transparent;
color: black;
border-collapse: collapse;
border: 1px solid #000;
padding: 0 .8em .3em .5em;
}
#p-cactions .pBody {
border: none;
}
.pBody a {
font-weight: bold;
color: #000000;
}
.pBody a:visited {
font-weight: bold;
color: #000;
}

#lastmod {
color: #000000;
font-weight: bold;
}
#viewcount {
color: #000000;
font-weight: bold;
}
#privacy a{
color: #2E0000;
font-weight: bold;
}
#about a{
color: #2E0000;
font-weight: bold;
}
#disclaimer a{
color: #2E0000;
font-weight: bold;
}

.portlet ul {
line-height: 1.5em;
list-style-type: square;
list-style-image: none;
font-size: 95%;
}

/*
** the navigation portlet
*/

#p-navigation {
position: relative;
z-index: 3;
}

#p-navigation .pBody {
padding-right: 0;
}

#p-navigation a {
font-weight: bold;
display: block;
}

#p-navigation li.active a, #p-navigation li.active a:hover {
text-decoration: none;
font-weight: bold;
}

#p-personal li a {
text-decoration: none;
color: #13F3F3;
padding-bottom: .2em;
background: none;
}
#p-personal li a:hover {
background-color: black;
padding-bottom: .2em;
text-decoration: none;
}

/*
** footer
*/
div#footer {
width: 800px;
position: absolute;
bottom: 0;
left: 0;
overflow: auto;
background-color: transparent;
border-top: none;
border-bottom: none;
margin: .6em 0 0 12.2em;
padding: .4em 0 0 0;
text-align: center;
font-size: 90%;
z-index: 10;
}

div.bib { text-indent: -2em; margin-left: 2em; }

BODY.page-Bibliography p { text-indent: -2em; margin-left: 2em; }

.title {
font-size: 155%;
font-weight: bold;
margin-bottom: .5em;
padding-top: .5em;
padding-bottom: .17em;
border-bottom: 2px solid #aaa;
}

table.main {
background-color: transparent;
font-size: 13px;
}

#bodyContent {
font-size: 16px;
}

#bodyContent ol li {
padding-bottom: 10px;
}

#bodyContent ol li ul li {
padding-bottom: 0px;
}

/* Fix special pages output */
.mw-specialpages-table td {
padding-top: 1em;
}

div#p-personal {
padding-top: 8px;
}

div#p-logo {
padding-top: 8px;
}

#column-one{
padding-top: unset;
}

Documentation Portal

2024-10-16T17:20:29Z

Robertson:

Documentation Portal

2024-07-25T17:08:26Z

Robertson:

Getting Started

2024-01-24T21:30:06Z

Robertson:

<div class="title">Getting Started</div>
ROMS is a very complex model with many options and capabilities. ROMS is composed of many Fortran files ('''.F'''), a few header files ('''.h'''), various input script files ('''.in'''), a metadata variable definition file ([[varinfo.dat]]), and a single [[makefile]]. The ROMS algorithms are distributed with the following directory structure:

<span class="twilightBlue">trunk/ Main trunk directory
/Atmosphere/ Atmosphere models root directory
/COAMPS COAMPS root directory (empty)
/WRF WRF root directory (empty)
/Compilers/ make configuration files
/Data/ Input data root directory
/ROMS/ ROMS data root directory
/CDL ROMS NetCDF metadata design
/Forcing Input test cases forcing NetCDF files
/Grid Input test cases grid NetCDF files
/Initial Input test cases initial conditions NetCDF files
/Lib/ External libraries
/ARPACK Arpack eigenvalue problems library
/MCT Modeling Coupling Tool library
/Master Main standalone and coupling programs
/ROMS/ ROMS root directory
/Adjoint Adjoint model
/Biology Adjoint biology/ecosystem models
/Bin Executable scripts
/Drivers Computational drivers
/External Standard input scripts
/Functionals Analytical expression header files
/Include Test cases configuration header files
/Modules Declaration modules
/Nonlinear Nonlinear model
/Biology Nonlinear biology/ecosystem models
/Sediment Nonlinear sediment transport model
/Obsolete Discontinued files
/Programs Support programs
/SeaIce Sea-ice model (empty)
/Representer Representer model
/Biology Representer biology/ecosystem models
/Tangent Tangent linear model
/Biology Tangent linear biology/ecosystem models
/Utility Generic utility files
License_ROMS.text Open source license
Version SVN Version information
/User/ ROMS User interface root directory
/External User standard input scripts
/Functionals User analytical expressions templates
/Include User application header files
/WAVES/ Waves models root directory
/SWAN SWAN root directory
/External SWAN input data and standard input files
/Src SWAN model</span>

ROMS uses [[C-preprocessing]] extensively to activate and/or deactivate
the various numerical and physical algorithm [[Options | options]]. It also uses
[[NetCDF]] to manage input and output data streams. It is highly
recommended that first time users learn the basics of NetCDF
before working with ROMS.

==Basic Steps==

* Register at the ROMS website ([http://www.myroms.org www.myroms.org]) to get access to the algorithms and other user privileges. Select a '''username''' and '''password''' which will be used in the future to login and post messages on the ROMS forum, algorithm downloads, and contributing to wikiROMS.

* Make sure that you have the following required software in your computer before attempting to compile and run an application:
** [https://git-scm.com/ Git] client to download source
** Fortran 90 or Fortran 95 compiler
** [http://gcc.gnu.org/onlinedocs/cpp/ cpp] program for [[C-preprocessing]] ROMS source code.
** [http://www.gnu.org/software/make/ GNU make] version '''3.81''' or higher to compile ROMS
** [http://www.perl.org/ Perl] interpreter program
** [http://www.unidata.ucar.edu/software/netcdf/ NetCDF] library, with Fortran 90 interface
** Message Passing Interface (MPI) library to run in parallel on a distributed-memory system

* Use a [[Git]] client to '''clone''' the latest version of the ROMS '''develop''' branch. It is highly recommended that you obtain the code on the same system that you will be compiling and running the code on to avoid file format issues.

* We recommend that you use the included '''[[build Script|build script]]''' to compile and link ROMS. This will set up your build environment and execute the [[gmake|make]] command to build the default ROMS [[UPWELLING_CASE | upwelling]] application. This process allows you to avoid editing the [[makefile]]. Please visit the '''[[build Script|build script]]''' page for more detailed instructions.
** You can also type <span class="red">make</span> at the top of the directory structure where the [[makefile]] is located but we do not recommend this process because it requires changing the ROMS files which can cause conflicts when you update you ROMS code.

{{note}}'''Note:''' To speed compilation, you may want to add the '''-j <n>''' flag to the build command (i.e. '''build.sh -j 4''') where '''<n>''' is the number of processors you wish to compile with. Even single processor machines can benefit from the -j flag with '''<n> = 2'''.

{{note}}'''Note:''' To make sure your application can compile successfully, you might want to set '''USE_DEBUG''' to '''on''' in the [[build Script|build script]] since it will compile faster. Once your application can compile you can unset '''USE_DEBUG''' in order to create an optimized executable. Please visit the '''[[build Script|build script]]''' page for more information

{{note}}'''Note:''' In SVN revision 933 (January 26, 2019) all <span class="red">ocean_*.in</span> files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. If you are working with a ROMS release prior to revision 933 you will need to replace <span class="red">roms_upwelling.in</span> with <span class="red">ocean_upwelling.in</span> and <span class="red">romsS</span>, <span class="red">romsM</span>, or <span class="red">romsO</span> with <span class="red">oceanS</span>, <span class="red">oceanM</span>, or <span class="red">oceanO</span> in all commands below.

* To run ROMS in serial, just type: <div class="box"><span class="red">romsS < ROMS/External/roms_upwelling.in > & log &</span></div>or to run in parallel (distributed-memory) on two processors: <div class="box"><span class="red">mpirun -np 2 romsM ROMS/External/roms_upwelling.in > & log &</span></div>or to run in parrallel (shared-memory) on two processors:<div class="box"><span class="red">setenv OMP_NUM_THREAD 2<br>romsO < ROMS/External/roms_upwelling.in > & log &</span></div>Here, the the file <span class="red">ROMS/External/roms_upwelling.in</span> contains all the input parameters to required by this application. Visit [[roms.in]] for more information.

{{note}}Notice that in distributed-memory, the leading <span class="red"><</span> is omitted so all parallel threads can read and process this input script without any communications in between.

Frequently Asked Questions

2024-01-24T18:16:19Z

Robertson:

<div class="title">Frequently Asked Questions</div>
__TOC__

== Installation and Configuration ==
{{note}}'''Note:''' In SVN revision 933 (January 26, 2019) all <span class="red">ocean_*.in</span> files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. If you are working with a ROMS release prior to revision 933 you will need to replace <span class="red">roms_upwelling.in</span> with <span class="red">ocean_upwelling.in</span> and <span class="red">romsS</span>, <span class="red">romsM</span>, or <span class="red">romsO</span> with <span class="red">oceanS</span>, <span class="red">oceanM</span>, or <span class="red">oceanO</span> in all commands below.

=== My build finished with no errors. Where is the ROMS executable? ===

It is either in '''romsS''' (serial), '''romsO''' (shared-memory, OpenMP), '''romsM''' (distributed-memory, MPI), or '''romsG''' (debug). Check the [[makefile]] to see which options are on. The makefile definition <span class="red">BINDIR</span> controls where to write the compiled ROMS executable.

=== What do I have to do to run an application? ===
There are basically two modes to run a ROMS application: serial or parallel
# Most of the canned [[Test Cases | applications]] that come with the ROMS distribution do not need input NetCDF files. The grid, initial conditions and forcing conditions are specified with analytical expressions in [[Functionals]]. Any of these test cases can be run by editing the [[makefile#ROMS_APPLICATION | ROMS_APPLICATION]] definition in the [[makefile]] or the [[build Script]]. A [[cppdefs.h#Options_for_idealized_test_problems | list]] of all pre-defined model applications can be found in header file [[cppdefs.h]]. The next step is to compile and link using the [[gmake | make]] tool and, if you are lucky, the program will build. It is good practice to execute '''make clean''' first. Then, the application should be run in serial with the command: <div class="box"><span class="red">romsS < roms_</span><span class="blue">APPLICATION</span><span class="red">.in > & log & </span></div>where <span class="blue">APPLICATION</span> is the lowercase of any of the CPP options defining a distributed test case. For example, the upwelling ([[UPWELLING]]) [[UPWELLING_CASE | test case]] uses the input script <span class="red">roms_upwelling.in</span> which is located in the '''ROMS/External''' directory.
# A few canned applications that come with ROMS require input NetCDF files which are located in the '''Data/ROMS''' directories. Same as above, but if you want run in parallel on a distributed-memory computer use: <div class="box"><span class="red">mpirun -np 2 romsM roms_test_head.in > & log & </span></div>or to run in parallel on a shared-memory machine:<div class="box"><span class="red">setenv OMP_THREAD_NUM 2<br>romsO < ROMS/External/roms_upwelling.in > & log &</span></div>for the [[TEST_HEAD_CASE | test headland case]], for example. You need to be sure that the product of [[Variables#NtileI | NtileI]] and [[Variables#NtileJ | NtileJ]] in '''roms_test_head.in''' is '''2''' since the above command specifies two processors to run ('''-np 2''' or '''OMP_THREAD_NUM=2'''). Notice that to compile the model in distributed-memory, you need to edit the [[makefile]] (or [[build Script]]) and activate [[makefile#USE_MPI | USE_MPI]]. Also, if you use '''MPICH''' rather than a native MPI library, define [[makefile#USE_MPIF90 | USE_MPIF90]] as well. The MPICH library uses a script called [[mpif90]] for compiling.

=== Why is only one node created when executing with mpirun? ===
This usually occurs when you are not using the correct mpirun. For Example, if you compiled with Open MPI and try to use the <span class="forestGreen">mpirun</span> command from MPICH, you will see this error. To see which mpirun you are executing type ''''which mpirun''''; the first part of the path should match what the ROMS output reports as your ''''Compiler command''''.

<div class="box"><span class="red">> which mpirun<br />/opt/pgisoft/mpich/bin/mpirun<br />> mpirun -np 2 ./romsM roms.in<br /> Process Information:<br /><br /> Node # 0 (pid= 150433) is active.<br /><br /> Model Input Parameters: ROMS/TOMS version 3.4<br /><br /> . . .<br /><br /> Compiler system : pgi<br /> Compiler command : /opt/pgisoft/openmpi/bin/mpif90<br /> Compiler flags : -O3 -Mfree<br /><br /> . . .<br /><br /> ERROR: Illegal domain partition.<br />></span></div>

Notice that only one node is active and the compiler command shows the model was compiled with Open MPI but <span class="forestGreen">mpirun</span> is from the MPICH directory. To fix this problem, update your path to use the mpirun from the <span class="forestGreen">openmpi</span> directory or re-compile with MPICH.

== Compiling and Linking ==

The compiling and linking information is all in [[Compiling and Linking]], [[makefile]], and [[build Script]]. Please let us know if any of this is unclear.

=== Errors at compile time ===
First of all, read the error messages from the compiler carefully to see if you can figure out what they are talking about. There are several classes of errors (add more as you see them):
* Errors from build_roms.sh(formerly build.bash):<div class="box">./build_roms.sh: line 40: $'\r': command not found<br/>./build_roms.sh: line 43: $'\r': command not found<br/>./build_roms.sh: line 46: syntax error near unexpected token '$'in\r'<br/>./build_roms.sh: line 46: ' case "$1" in</div>You need to run the dos2unix command on build_roms.sh.

* Errors from cpp: If there is a mismatch in the #ifdef/#endif pairs, cpp will be unhappy and will tell you so. There is a perl script called "ifdefs" which checks your #ifdefs/#endif pairings.

* Errors like this:<div class="box">cd Build_roms; -c -O3 mod_kinds.f90<br />/bin/sh: -c: not found</div>The command:<div class="box">FC := $(shell which ${FC})</div>in the Compilers/xxx-xxx.mk file has failed to find the compiler you specified, setting FC to the null string. You need to find out which compiler you were asking it to look for and to make sure it is in your path. If the flag USE_MPIF90 is set, it will be looking for "mpif90". What do you get from the command "which mpif90"? If you are trying for a serial run first, what does "which gfortran" give you?

* Failing to find netcdf.mod:<div class="box">cp -f /usr/local/include/netcdf.mod /home/kekejiayu/Projects/upwelling/Build<br/>cp: cannot stat ‘/usr/local/include/netcdf.mod’: No such file or directory</div>You need to have both the netcdf and netcdf-fortran libraries. It is strongly recommended that these be version 4.x, not in the 3 series and that you have USE_NETCDF4=on in your build. Even so, things can go wrong. For instance, recent versions of nc-config rely on finding nf-config in your PATH to work properly. In this case there are three options:
*#Add the full path to nf-config to you your PATH environment variable.
*#Change the appropriate NC_CONFIG declaration in your [[build.{sh|bash}|build_Script]] to the full path to your "nf-config".
*#Change "nc-config" to "nf-config" in the appropriate file in the Compilers directory. If "nf-config" is not in your path you will need to set this to the full path of your "nf-config" command.

* Syntax errors in the fortran: check the intermediate .f90 file for the exact code at the line numbers in the report. Be sure to edit the corresponding .F file once you determine what the trouble is.

* Trouble with the netcdf module file:<div class="box">PGF90-F-0004-Corrupt or Old Module file ./netcdf.mod (distribute.f90: 2750)</div> Netcdf-fortran must be compiled with the same version of the same compiler you are using now. If not, there will be trouble.

* Trouble with other modules: one instance of this is due to incorrect dependency information in MakeDepend. The ROMS code must be compiled in the correct order and failure to do so might give something like:<div class="box">cd Build_roms; ifort -c bc_2d.f90<br />fortcom: Error: bc_2d.f90, line 30: Error in opening the Library module file.<br />[MOD_PARAM]<br /> USE mod_param<br />----------^<br />fortcom: Error: bc_2d.f90, line 32: Error in opening the Library module file.</div> Here, it failed to compile mod_param.F before bc_2d.F, leading to trouble. The '--objdir=$(BUILD_DIR)' option in MDEPFLAGS is not optional.

=== Errors at link time ===
* One common problem is that differing options were used when compiling libraries vs. the ROMS code proper. For instance, you get many, many messages starting with:<div class="box">.../Build_roms/libUTIL.a(close_io.o):close_io.f90:(.text+0x1030): undefined reference to `__netcdf_MOD_nf90_strerror'</div> Check to make sure that the link statement above the errors includes both '-lnetcdff' and '-lnetcdf' with the correct path to the libraries: '-L/usr/local/pkg/netcdf/netcdf-4.3.0.gnu-4.7.3/lib', for instance.

* Old ROMS versions only linked to libnetcdf.a, while NetCDF 4.x also needs libnetcdff.a to be built from a separate netcdf-fortran package. The link statement needs to include both. See [https://www.myroms.org/forum/viewtopic.php?f=31&t=1296&p=4599&hilit=config#p4599 this thread] for a discussion on it.

* Also new in NetCDF is the option to create a library with OpenDAP capabilities. In that case, you need to also link to libcurl, in the makefile as the option USE_DAP. This option is needed if you are missing functions with "curl" in the name.

* For all of these issues, we strongly recommend the use of USE_NETCDF4 and its method of running 'nc-config --flibs' to find out the correct linking bits, though there are systems on which 'nf-config --flibs' works better. Run these things on the command line to find out what works for you:<div class="box">pacman13 396% nc-config --flibs<br> -L/usr/local/pkg/netcdf/netcdf-4.3.0.gnu-4.7.3/lib -lnetcdff \<br>-L/usr/local/pkg/hdf5/hdf5-1.8.10-p1.gnu-4.7.3/lib -L/usr/local/pkg/szip/szip-2.1.gnu/lib \<br> -L/usr/local/pkg/udunits/udunits-2.1.24.gnu-4.7.3/lib \<br>-L/usr/local/pkg/netcdf/netcdf-4.3.0.gnu-4.7.3/lib -lnetcdf -lnetcdf</div>or<div class="box">fish2 1487% nc-config --flibs<br>-L/opt/cray/netcdf/4.3.2/PGI/141/lib -lnetcdff -L/opt/cray/hdf5/1.8.13/PGI/141/lib -lnetcdf</div>

== Basic Usage ==
=== What are the ways that ROMS can receive the wind and other surface forcing? ===
# One option is to provide the stresses and fluxes via analytic expressions in [[Functionals#ana_smflux.h | ana_smflux.h]] and [[Functionals#ana_stflux.h | ana_stflux.h]].
# One option is to provide analytic expressions for the winds and other atmospheric fields in [[Functionals#ana_winds.h | ana_winds.h]] and friends, and define [[Options$BULK_FLUXES | BULK_FLUXES]].
# ROMS can also read fields from NetCDF files, also stresses or winds, depending on the state of [[Options$BULK_FLUXES | BULK_FLUXES]]. These can be provided as:
## Point measurements, in which ROMS will rotate to the correct directions from East (u) and North (v).
## Gridded fields on the ROMS grid, in which case the directions have to match the directions on your grid.
## Gridded fields on a larger, usually coarser grid. These are in East and North, with ROMS doing the rotation. ROMS picks up its cue on whether to interpolate+rotate or not based on the grid dimensions of these fields. If the dimensions exactly match that of your own grid, ROMS will assume that it is on your grid. If you plan on using global fields of dimension 180x90, say, don't build a grid with those dimensions. The cdl for a working example is shown here:
<div class="box">netcdf rain.1948-2006 {
dimensions:
rain_time = UNLIMITED ; // (708 currently)
lat = 94 ;
lon = 192 ;
variables:
float rain(rain_time, lat, lon) ;
rain:_FillValue = -1.e+34f ;
rain:units = "Kg_meter-2_sec-1\000Kg_meter-2_sec-1" ;
rain:long_name = "rain fall rate" ;
rain:coordinates = "lon lat" ;
double lat(lat) ;
lat:units = "degrees_north" ;
lat:point_spacing = "uneven" ;
lat:axis = "Y" ;
double lon(lon) ;
lon:units = "degrees_east" ;
lon:modulo = 360. ;
lon:point_spacing = "even" ;
lon:axis = "X" ;
float SNOW(rain_time, lat, lon) ;
SNOW:missing_value = -1.e+34f ;
SNOW:_FillValue = -1.e+34f ;
SNOW:long_name = "SNOW_FRAC[D=2,GXYT=PRC_MOD@ASN]*PRC_MOD" ;
SNOW:long_name_mod = "L=1:12" ;
double rain_time(rain_time) ;
rain_time:units = "days since 1900-01-01 00:00:00" ;
rain_time:axis = "T" ;
rain_time:bounds = "TIME_bnds" ;
rain_time:time_origin = "1-JAN-1948" ;
rain_time:calendar = "LEAP" ;
</div>
In this example, the variable names "rain" and "SNOW" would have to match with what's in the [[varinfo.dat]] file.

=== How can I create the initial/boundary/forcing NetCDF files for ROMS? ===
# Please don't use the old Fortran codes unless you are a glutton for punishment.
# Most use the Matlab scripts on the ROMS [https://www.myroms.org/svn/src/matlab/ svn server].
# There are Python tools at [https://github.com/kshedstrom/pyroms github]. A thorough description of how to get these running on a Mac is [http://houstonthereisaproblem.blogspot.com/2011/12/installation-of-pyroms-into-mac-os-1072.html here]. An alternate way to go is to install a VirtualBox with Linux and install them there.
# There are probably Ferret and other scripts out there. Ask on the ROMS forum if you have question about a specific data source.

=== What are the ways that ROMS can receive tides on the boundaries? ===
# You can specify a time-dependent boundary condition file that temporally resolves the tides and skip the tides file entirely.
# You can specify [[Options#SSH_TIDES | SSH_TIDES]] and/or [[Options#UV_TIDES | UV_TIDES]] and provide a tides file and skip the boundary condition file entirely - though this option might require you to tell ROMS in [[globaldefs.h]] that you really, really don't need an [[Options#OBC_FILE | OBC_FILE]]. Usual practice is to provide both surface elevation and currents in the tidal file and to use FSCHAPMAN and M2FLATHER on the open sides of your domain.
# You can specify other 2-d currents and surface elevation signals as coming in, plus have tides and use the [[Options#ADD_M2OBC | ADD_M2OBC]] and [[Options#ADD_FSOBC | ADD_FSOBC]] options.
=== How can I set up passive tracers? ===
You need to #define [[Options#T_PASSIVE | T_PASSIVE]] to get passive tracers. The flag [[Options#ANA_PASSIVE | ANA_PASSIVE]] determines whether or not the initial conditions are analytic (from [[Functionals#ana_passive.h | ana_passive.h]]) or from the initial condition file. As for boundary conditions, all tracers have either analytic boundary conditions or they all come from the same boundary condition file. If you need to read values from a file, the field names are dye_01, dye_02, etc. I successfully set up a uniform dye_01 using ANA_PASSIVE with boundary conditions from a file. The boundary file has u,v,T,S for many times and (analytic) biological values for the beginning and end times. I have a script to read one biology value, change the attributes and the value to my uniform value, then write it into the boundary file with the name dye_01_west, etc. I then had to edit [[varinfo.dat]] to reflect that it's using bio_time rather than ocean_time.

You probably also want [[Options#ANA_SPFLUX | ANA_SPFLUX]] and [[Options#ANA_BPFLUX | ANA_BPFLUX]] for analytic surface and bottom passive tracer fluxes.

=== What is the CFL criterion for ROMS? ===

From Sasha Shchepetkin:

1. Barotropic mode. The easiest to predict and control -- based on speed of the external waves, c_g=sqrt(g*h) where h is local depth, then <math display="block">Cg = c_g*dtfast*\sqrt{1/dx^2 +1/dy^2}</math> with dx and dy evaluated locally; Cg_max=max{Cg} over all horizontal grid points. This one is computed at the beginning of the run and reported by routine setup_grid2 and is is not monitored by diag thereafter. Its maximum allowed value depends on the particular time stepping algorithm and should be 0.87 for generalized FB stepping or 0.9-something for LF-AM3.

2. Internal wave speed. Essentially phase speed of the first baroclinic mode multiplied by dt (slow-mode time step) and by <math display="block">\sqrt{1/dx^2 +1/dy^2}</math> same as above. Once again, this is not monitored by diag because it is difficult to evaluate the phase speed, but one has to rely on oceanographic intuition to pick a reasonable value. Internal waves (first baroclinic mode is the fastest) are typically faster than the fastest horizontal advection speed, but is not too fast away from it. So internal phase speeds of about 2.5 m/sec is quite realistic.

3. Horizontal advection. The relevant number is Cu=dt*u/dx, however it should be kept in mind that for a 3D finite volume advection algorithm where fluxes in all three directions are applied simultaneously it is the combination off all three sets of fluxes what controls stability. So the most natural way to interpret Courant number for the 3D advection algorithm is the fraction of water within the control volume which being replaced in one time step. This CFL is monitored by diag.F (if proper CPP switches are activated).

4. Vertical advection. This is the one which is hard to predict in advance, and typically it becomes more and more restrictive as resolution becomes finer. This is both because your solution becomes more vivid, and all currents are intensified as well as due to the nature of hydrostatic equations (because hydrostatic pressure is computed individually in each vertical column, each horizontal velocity "feels" the effect of the nearest neighbors but not several grid intervals away as it would be in the case of nonhydrostatic model. As the result any free-travelling nonlinear internal wave tends to from a sharp shock-like front which has large vertical velocity. Eventually it becomes unrealisticly large if one chooses smaller and smaller horizontal dx. The threshold to ensure that it does not happen is the aspect ratio: thermocline depth / dx. As long as it is small (say less than unity, but does not have to be vanishingly small), vertical velocity is effectively averages over horizontal grid box area, and therefore cannot be large. Conversely, pushing dx smaller an smaller beyond this threshold makes model violate the applicability of hydrostatic approximation, and in practice result in unrealistically large vertical velocities. (Vice-versa, using nonhydrostatic code on a coarse grid such that the above ratio is vanishingly small is plain waste of resources because the non-hydrostatic effects are essentially non-existing on the scales resolved by such grid, and in this sense it is the fact that horizontal resolution is finite is what makes hydrostatic-approximation model applicable).

Vertical CFL is monitored by diag.F along with total advection CFL, and the latter should be kept under some limit to ensure model stability. The actual limiting value depends on the selection of advection algoritms and details of time stepping for 3D mode. But in any case the criterion is about 1 or so (0.8 or 0.9 or whatever).

Another complication comes from the fact that finer grid means less smoothed topography which translates into larger vertical velocities. Also having/not having tides typically makes major difference in this regard.

Another complication comes from the fact that some modern versions of ROMS have capability to circumvent vertical advection by monitoring overall 3D advective CFL budget and split vertical velocity into two parts, W=We+Wi, in such a way that We always stays within CFL allowed by the explicit advection scheme, and Wi is the "excess" portion vertical velocity which is treated implicitly for both advection of momentum and tracers equations. This offers some relief from vertical CFL.

Another complication comes from the fact that criteria 2 and 3 are actually add up to each other rather than to be constrained separately, i.e., internal wave speed must be increased by advection speed when considering stability budget.

5. Inertial period CFL. Because Coriolis term is treated explicitly it has its own CFL criterion. This is restrictive only if your time step is sufficiently large in absolute sense, that is 1 hour or more; the relevant nondimensioal number f*dt

=== How can I deal with "patch" files? ===
OK, so maybe no one actually asked this yet, but I posted something in "diff -u" format:
<div class="box">diff --git a/ROMS/Nonlinear/output.F b/ROMS/Nonlinear/output.F
index e4fb854..42ff7a8 100644
--- a/ROMS/Nonlinear/output.F
+++ b/ROMS/Nonlinear/output.F
@@ -1,4 +1,5 @@
#include "cppdefs.h"
+#define LONG_NUMS
#ifdef NONLINEAR
SUBROUTINE output (ng)
!
@@ -111,7 +119,11 @@
ifile=(iic(ng)-1)/ndefHIS(ng)+1
IF (Master) THEN
WRITE (HIS(ng)%name,10) TRIM(HIS(ng)%base), ifile
+#ifdef LONG_NUMS
+ 10 FORMAT (a,'_',i5.5,'.nc')
+#else
10 FORMAT (a,'_',i4.4,'.nc')
+#endif
END IF
# ifdef DISTRIBUTE
CALL mp_bcasts (ng, iNLM, HIS(ng)%name)</div>Do I honestly expect you to apply this by hand? Not at all. If you save it to a file, called say "diff.out", there's more than one way to automatically apply it:
* Because this was generated by git, you can use "git apply diff.out" to have these changes made to your output.F file. Shown here are two "hunks" starting with the "@@" lines. Git will check all the hunks before applying any so you won't end up with the file in a half-baked state.
* A much older tool for applying these patches is "patch". The proper invocation here is "patch -p1 < diff.out" from the top of your ROMS directory tree, where the "-p1" strips off the a/b pseudo-directories from the git output. Unlike git, patch will apply as many hunks as it can, dumping the rest into an output.F.rej file for you to deal with later.

== Errors at Run Time ==
=== Segmentation fault ===
There is more than one way to get a seg fault. Ways I know of include (a) running out of memory or some other system limit, (b) accessing an array out of bounds, (c) calling a function with too few arguments, (d) compiling on one system and running on a second with incompatible MPI libraries, (e) asking get_ngfld to read a field of all land values. For me, solving these typically involves recompiling with USE_DEBUG and running in a debugger. The array out of bounds can be found by using a bounds checking compiler flag (often included in USE_DEBUG).

One common example is shown here:
<div class="box">romsG 0000000000439EAC load_s2d_ 1564 inp_par.f90
romsG 0000000000481B8C read_phypar_ 1215 read_phypar.f90
romsG 000000000040F213 inp_par_ 88 inp_par.f90
romsG 000000000040666D ocean_control_mod 84 ocean_control.f90
romsG 000000000040635F MAIN__ 95 master.f90</div>load_s2d is used to read a list of forcing files in the trunk code. In the COAWST/ice codes, it is also used to read a list of CLM and BRY files. In the case where there can be multiple BRY files, you need to tell it how many BRY files so that the structure can be allocated. If you are using COAWST/ice, be sure your roms.in includes the setting of '''NCLMFILES''' and '''NBCFILES''', even if you have at most one of them.

Be sure your roms.in is consistent with your ROMS version!

=== Blowing up in diag.F ===
ROMS will check for extreme values of velocity and density at run time, including looking for NaN and Inf. If found, it will save a record to the restart file and die. If this happens early in the run, you might just try a shorter timestep. If it happens after many days/years of model time, you need to dig more deeply. Find out which field went bad and where in the domain it happened. The restart record can be viewed to look for this trouble. I've had runs go bad at the bottom with a bottom stress instability, at the top with a surface forcing problem, at the boundary with boundary condition issues. I've also had it go bad in the middle of the water column in no place special due to some other instability - it goes from fine to terrible in just a few timesteps. This last can often be run past with a shorter timestep, while the other problems call for more thoughtful responses.

=== NETCDF_OPEN - unable to open existing NetCDF file: ===
followed by some random junk. In the roms.in file, it is best not to comment out the lines with input filenames. For example, if ROMS wants to read a forcing file and doesn't know the name of it, it will have junk in the filename. I know there are times when one doesn't want to provide a forcing file, yet ROMS insists that it needs one. If you leave the filename intact, you will instead trigger a message about the field it can't read from file forcing.nc. You can check globaldefs.h for the conditions that trigger the need for FRC_FILE.

Also, if you tell it you have 13 forcing files, then only provide 12, you will get this message when it tries to read that file #13.

=== Other troubles ===
ROMS tries to check everything and print useful error messages if something isn't right. Many problems can be solved by carefully reading the ROMS output. Otherwise, carefully check to make sure your roms.in file is appropriate for the version of ROMS you are running. Are you adding variables that you forgot to include in your varinfo.dat? Is your bathymetry too steep? If it's still not clear, I then go back to the debugger - a very useful skill for serious ROMS hacking.

== Algorithm Design ==
=== Why the _r8 at the end of real constants? ===
Some computers use 32 bits for single precision real numbers and 64 bits for double precision, and some use 64 and 128, respectively. For consistent results on various machines, ROMS takes advantage of the intrinsic F90/95 function SELECTED_REAL_KIND( ). See ROMS/Modules/mod_kinds.F. This allows you to associate an integer parameter with a specific data type...in this case r8 is associated with 64-bit precision. RTFM (Read the Fortran90 Manual), or better yet, chapter 11 in [[Bibliography#ChapmanSJ_2004a | Chapman, 2004]].

=== Why we cannot have zero bathymetry (<span class="red">h=0</span>)? ===
In ROMS the the bathymetry, <span class="red">h</span>, is positive and represents the vertical water column thickness (meters) from the ocean rest state (<span class="red">zeta=0</span>). We '''cannot''' have <span class="red">h=0</span> in '''any''' horizontal grid point even if it is masked by land. Therefore, during grid generation the <span class="red">h</span> values are limited to a minimum value <span class="red">hmin</span>. Recall that in terrain-following vertical coordinates <span class="red">h</span> is used to compute the vertical level thickness (<span class="red">Hz</span>). If you examine the [[Curvilinear_Coordinates_Transformation | governing equations]], we divide by <span class="red">Hz</span> when computing the vertical viscosity and diffusion terms. In addition, in several places of the numerical kernel we also divide by <span class="red">Hz</span>. This is usually computed in the internal array <span class="red">oHz=1/Hz</span>. In mathematics, the '''division by zero''' is not defined and computers will return '''Inf''' or '''NaN'''. Users need to consider carefully what value of <span class="red">hmin</span> to use by taking into account the horizontal grid resolution, number of vertical levels, and vertical coordinates stretching function for a particular application. In ROMS, all the vertical levels are located above <span class="red">hmin</span>. This will limit the vertical time-step of the model due to the Currant-Friedrichs-Levy (CFL) condition. Notice that we also need a critical water column thickness (<span class="red">Dcrit</span>) in wetting and drying for the same reasons mentioned above.

== Input/Output NetCDF files ==
=== How to add a new variable to ROMS output files? ===
This question has been asked several times. ROMS IO design is very flexible and easy to expand. There are several steps that need to be followed to add new variables to any of the output files:
# The ROMS output NetCDF files are defined by any the routines with the prefix <span class="red">def_</span> in the <span class="blue">ROMS/Utilities</span> directory. For example, in [[def_his.F]] a variable can be defined in the history file by just adding:<div class="box"><span class="twilightBlue"> IF (Hout(idUvel,ng)) THEN<br /> Vinfo( 1)=Vname(1,idUvel)<br /> Vinfo( 2)=Vname(2,idUvel)<br /> Vinfo( 3)=Vname(3,idUvel)<br /> Vinfo(14)=Vname(4,idUvel)<br /> Vinfo(16)=Vname(1,idtime)<br /># if defined WRITE_WATER && defined MASKING<br /> Vinfo(20)='mask_u'<br /># endif<br /> Vinfo(22)='coordinates'<br /> Aval(5)=REAL(Iinfo(1,idUvel,ng),r8)<br /> status=def_var(ncHISid(ng),hisVid(idUvel,ng),NF_FOUT, &<br /> & nvd4,u3dgrd,Aval,Vinfo,ncname)<br /> END IF</span></div> Correspondingly, the field time records are written in routines with the prefix <span class="red">wrt_</span>. For example, a field can be written in [[wrt_his.F]] as:<div class="box"><span class="twilightBlue"> IF (Hout(idUvel,ng)) THEN<br /> scale=1.0_r8<br /> gtype=gfactor*u3dvar<br /> status=nf_fwrite3d(ng, iNLM, ncHISid(ng), hisVid(idUvel,ng), &<br /> & tHISindx(ng), gtype, &<br /> & LBi, UBi, LBj, UBj, 1, N(ng), scale, &<br /># ifdef MASKING<br /> & GRID(ng) % umask(LBi,LBj), &<br /># endif<br /> & OCEAN(ng) % u(LBi,LBj,1,NOUT))<br /> IF (status.ne.nf90_noerr) THEN<br /> IF (Master) THEN<br /> WRITE (stdout,10) TRIM(Vname(1,idUvel)), tHISindx(ng)<br /> END IF<br /> exit_flag=3<br /> ioerror=status<br /> RETURN<br /> END IF</span></div>Ass you may noticed, some knowledge of ROMS internal structure is required to define a new field. Please follow any of output fields examples in such files to define and write the new field.
# Notice that ROMS state variables are defined on a horizontal, [[Numerical_Solution_Technique#Horizontal_Discretization|staggered Arakawa C-grid]]. In addition, the variables are also [[Numerical_Solution_Technique#Vertical_Discretization|staggered in the vertical]]. Therefore, you need choose the appropriate identifier flag:
#* <span class="red">p2dvar</span>: 2D field at ψ-points
#* <span class="red">r2dvar</span>: 2D field at ρ-points
#* <span class="red">u2dvar</span>: 2D field at u-points
#* <span class="red">v2dvar</span>: 2D field at v-points
#* <span class="red">p3dvar</span>: 3D field at ψ-points
#* <span class="red">r3dvar</span>: 3D field at ρ-points
#* <span class="red">u3dvar</span>: 3D field at u-points
#* <span class="red">v3dvar</span>: 3D field at v-points
#* <span class="red">w3dvar</span>: 3D field at w-points
#* <span class="red">b3dvar</span>: 3D bed-sediment field
# Select a unique six-character, case sensitive, field identifier which is defined in file [[mod_ncparam.F | ROMS/Modules/mod_ncparam.F]]. Notice that all the identifiers are defined in alphabetic order to facilitate the search. Choose a meaningfull identifier prefixed by <span class="red">id</span>. This indentifier needs to be assigned in routine <span class="blue">initialize_ncparam</span> during the reading of variable metadata form file [[varinfo.dat]]. You need to add code similar to:<div class="box"><span class="twilightBlue"> CASE ('idUvel') <br /> idUvel=varid</span></div>Check the example above to see how the <span class="twilightBlue">idUvel</span> identifier is used in the internal field metadata arrays.
# Define new field metadata in file [[varinfo.dat | ROMS/External/varinfo.dat]]. Follow the examples to define the following field information:
#* Field variable name string. This is the variable name in the output NetCDF file. You need to choose a unique and meaninful variable name.
#* Field long-name string. This is the variable <span class="red">long_name</span> attribute in the output NetCDF file.
#* Field units string. This is the variable <span class="red">units</span> attribute in the output NetCDF file.
#* Field type string. This is the variable <span class="red">field</span> attribute in the output NetCDF file.
#* Associated time-variable name string. This is the variable <span class="red">time</span> attribute in the output NetCDF file.
#* Field identifier string. This is the field six-character unique index used in information arrays.
#* Field staggered C-grid variable type. Use one of flags defined above.
#* Field scale floating-point value. This value is only used in input data to scale field to model units, if necessary. Otherwise, use a unity value.<br />Notice that all string information above is specified within single quotes to facilitate free-format reading in [[mod_ncparam.F]]. For example:<div class="box"><span class="twilightBlue">'u' ! Input/Output<br /> 'u-momentum component'<br /> 'meter second-1' ! [m/s]<br /> 'u-velocity, scalar, series'<br /> 'ocean_time'<br /> 'idUvel'<br /> 'u3dvar'<br /> 1.0d0</span></div>
# Add logical switch <span class="red">Hout(id....)</span> in input script [[roms.in]] to activate the processing of the new field. For example:<div class="box"><span class="twilightBlue">Hout(idUvel) == T ! 3D U-velocity</span></div> This new switch needs to be read in file [[inp_par.F | ROMS/Utility/inp_par.F]], routine <span class="red">read_PhyPar</span>. For example:<div class="box"><span class="twilightBlue"> ELSE IF (TRIM(KeyWord).eq.'Hout(idUvel)') THEN<br /> IF (idUvel.eq.0) THEN<br /> WRITE (out,280) 'idUvel'<br /> STOP<br /> END IF<br /> Npts=load_l(Nval, Cval, Ngrids, Hout(idUvel,1))</span></div>It is also a good idea to report the new logical switch to standard output. For example:<div class="box"><span class="twilightBlue"> IF (Hout(idUvel,ng)) WRITE (out,170) Hout(idUvel,ng), &<br /> & 'Hout(idUvel)', &<br /> & 'Write out 3D U-momentum component.'</span></div>
# Follow the same steps to process time-averaged fields. In addition, you need to define, allocate, and initialize in [[mod_average.F | ROMS/Modules/mod_average.F]] the new ROMS variable used to accumulate the time records. For example:<div class="box"><span class="twilightBlue"> real(r8), pointer :: avgu3d(:,:,:)<br />...<br /> allocate ( AVERAGE(ng) % avgu3d(LBi:UBi,LBj:UBj,N(ng)) )<br />...<br /> AVERAGE(ng) % avgv3d(i,j,k) = IniVal</span></div>The time-averaged fields are computed in [[set_avg.F | ROMS/Nonlinear/set_avg.F]]. Check this file to see how this is done.

Applications

2024-01-24T18:12:22Z

Robertson:

<div class="title">Contributed User Applications</div>

'''User applications that are not distributed with the ROMS source code are invited here '''

Changes to the code structure now make it straightforward to confine application-specific options to a small number of files maintained separately from the central code.

This approach has the advantages that:

* users can update their central code, adding newly developed features and fixing reported bugs, without upsetting personal options for a specific project
* users can work on more than one project at once using a single up-to-date code stem
* applications can be shared with colleagues without redistributing the entire source code, thereby fostering user community use of the most up-to-date ROMS release

Most users configuring a typical application of the forward simulation model need only edit:

* a file containing the CPP definitions, e.g. ''project.h''. (The ''cppdefs.h'' file in previous ROMS versions has been re-structured to do this).
* an input [[roms.in]] script
* initial, forcing and/or boundary netcdf files
* application-specific ''analytical'' options following the templates for '''ana_*.F''' files in the '''User/Functionals''' directory. (The ''analytical.F'' subroutine in previous versions has been re-structured to do this).

__TOC__

== CBLAST ==

Configuration and input files for a model of the southeast New England shelf during the ONR Coupled Boundary Layers and Air-Sea Transfer (CBLAST) experiment are provided here as one example of how we recommend users configure realistic applications.

The files required to configure and run ROMS for CBLAST are available for download from '''www.myroms.org''' at the ''Datasets'' link on the left navigation panel (under the subheading '''Software'''), or follow this link [http://www.myroms.org/index.php?page=Datasets].

A brief description of the files follows:

=== cblast.h ===

This file defines CPP flags (previously in ''cppdefs.h'') that set options for physics, numerical schemes, vertical turbulence closure, open boundary conditions, biology, sediment transport ... and output options (averages, diagnostics, stations, floats) are set in ''cblast.h''.

=== roms_cblast.in ===

This file is the [[roms.in]] file for CBLAST that sets values that are defined at execution. Parameters of particular relevance to CBLAST include: timestep, output frequency, open boundary nudging timescales and input netcdf file paths (amongst many others).

In this example we use relative paths to input files that assume the following directory hierarchy:

* cblast/
** Data/
*** all initial, surface forcing, tides, and other open boundary data
*** the [[varinfo.dat]] file that matches model input/output variable names to corresponding netcdf data variables. A ''varinfo.dat'' is not included here because we use the default distributed with the code in ''ROMS/External/varinfo.dat''
** Forward
*** cblast.in
*** roms_cblast.in

Our choice of this directory hierarchy anticipates other applications with the Adjoint and 4DVAR codes for model sensitivity and data assimilation studies which would require distinct ''cblast.h'' files in each subdirectory (all controlled by a master driver script). You can change this organizational structure for files to suit your own requirements - just make the necessary changes in ''roms_cblast.in'' so that ROMS finds the files.

=== build_cblast.sh ===

''This script is a work in progress. We have already resolved some issues to do with the control of make depend. For now, the script provided here works but will be updated to use new features in the makefile as they evolve and are tested.''

This ''csh'' shell script serves to illustrate how a specific application can be configured without editing the [[makefile]] by using UNIX shell environment variables to over-ride the [[makefile]] defaults. If in the [[makefile]] a flag is set with the syntax '''VARIABLE ?= value''' this indicates its default value, but if '''VARIABLE''' is set in the shell that calls the script then the job inherits that value and over-rides the default. Of course, you can also choose to edit that line in the [[makefile]] itself and operate in that way. Just remember you did this.

In the example given here we set the variables:

* '''ROMS_APPLICATION''' is '''CBLAST''' which tells ''make'' to look for CPP definitions in a file called ''cblast.h''. See [[makefile]] for more information. The value of '''ROMS_APPLICATION''' must match the parameter '''MyAppCPP''' in [[roms.in]]
* '''MY_HEADER_DIR ''' tells ''make'' where to look for the ''cblast.h'' CPP defs file.
* '''MY_ANALYTICAL_DIR''' sets where '''ana_*.F''' files would be located. The CBLAST example here does not require any such files, but this line is left for information.
* '''BINDIR''': Sets where te executable should be placed following sucessful compilation.
** Not used in this example:
*** '''BUILD_DIR''' would set the path to a temporary ''Build'' subdirectory is created to hold '''*.f90''' and '''*.o''' files.
*** '''setenv MPI on''' would activiate compilation for parallel execution.
*** If your FORTRAN compiler is not the default in makefile, you could set it here with '''setenv FORT myF90compilername'''.

Executing the script '''build_cblast''' sets these environment variables to inform ''make'' what do when building the CBLAST application. The executable binary will be called ''romsM'' and the model would be run with command like:

cd MyProjects/cblast/Forward
./romsS < roms_cblast.in > roms_cblast.log

{{note}}'''Note:''' In SVN revision 933 (January 26, 2019) all <span class="red">ocean_*.in</span> files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. If you are working with a ROMS release prior to revision 933 you will need to replace <span class="red">roms_cblast.in</span> with <span class="red">ocean_cblast.in</span> and <span class="red">romsS</span> with <span class="red">oceanS</span> in the command above.

This model configuration was used to obtain the results described in:

Wilkin, J., 2006: The summertime heat budget and circulation of southeast New England shelf waters, Journal of Physical Oceanography, 36(11), 1997-2011.

The URL for full text article: http://ams.allenpress.com/perlserv/?request=get-document&doi=10.1175%2FJPO2968.1

A further study using this model set-up considers tidal circulation on the Nantucket Shoals:

He, R. and J. Wilkin, 2006: Barotropic tides on the southeast New England shelf: A view from a hybrid data assimilative modeling approach. Journal of Geophysical Research, 111, C08002, doi:10.1029/2005JC003254.

== LATTE ==

Coming soon.

makefile

2024-01-24T18:11:32Z

Robertson:

<div class="title">makefile</div>
{{warning}}<span class="red">Warning: Do not edit the '''makefile'''! Use the [[build Script]].</span> This page is for informational purposes only and is not a guide for compiling ROMS source code.

ROMS has a single '''makefile''' located in the top-level directory. Its generic structure facilitates the compiling of ROMS with other coupled models. The user usually needs to check/modify a few definitions in the '''makefile''' or the [[build Script]] before compiling a particular application. The list of the user options is given below.

This '''makefile''' builds the compiled objects in the specified build directory. It determines the compiling depedencies using the [[sfmakedepend]] script located in the '''ROMS/Bin''' directory. Notice that the '''makefile''' time modification in your computer is used by '''make''' to check if the dependencies need to be updated. If you set the definitions below through a [[build Script]], you need to execute '''make clean''' whenever changes to the user options are made.

All the other gory details of the '''makefile''' are covered in [[gmake]].

==Environment Variables in the '''makefile'''==

ROMS has a growing list of choices the user must make about the compilation before starting the compile process, set in user-defined variables. Since we now use '''gnu make''', it is possible to set the value of these variables in the Unix environment, rather than necessarily inside the '''Makefile''' (see [[gmake]]). The user-definable variables understood by the ROMS '''makefile''' are:

<section begin=ROMS_APPLICATION />;<span id="ROMS_APPLICATION"></span>[[ROMS_APPLICATION]]
:CPP option defining a particular application (for example, [[UPWELLING]]). The makefile will include its associated header file which is located in either the '''ROMS/Include''' directory or the path specified in the [[MY_HEADER_DIR]] definition. The header file name is the lowercase value of [[ROMS_APPLICATION]] with the '''.h''' extension (for example, [[upwelling.h]]). Recall the CPP option defining an application must be unique.<section end=ROMS_APPLICATION />

<section begin=MY_HEADER_DIR />;<span id="MY_HEADER_DIR"></span>[[MY_HEADER_DIR]]
: Application header file '''full''' directory path. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that it is included in [[cppdefs.h]]. Notice that if the header file is located in the '''ROMS/Include''' directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.<section end=MY_HEADER_DIR />

<section begin=MY_ANALYTICAL_DIR />;<span id="MY_ANALYTICAL_DIR"></span>[[MY_ANALYTICAL_DIR]]
: Directory containing the user's [[Functionals | analytic files]], if any. Templates of the analytic expressions are provided in the '''User/Functionals''' directory. If analytic expressions are activated, the makefile will load first the ones located in [[MY_ANALYTICAL_DIR]] and then the ones located in '''ROMS/Functionals'''. We recommended that you do not modify the analytical expressions in the '''ROMS/Functionals''' directory since these are the official version of these files and updates will fight with your changes. The user however can modify those in '''User/Functionals''' or some other directory and specify which directory with this variable.<section end=MY_ANALYTICAL_DIR />

<section begin=MY_CPP_FLAGS />;<span id="MY_CPP_FLAGS"></span>[[MY_CPP_FLAGS]]
: Additional CPP options to activate. Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the '''-D''' syntax. For example, to write out time-averaged NetCDF file use '''-D'''[[AVERAGES]].<section end=MY_CPP_FLAGS />

<section begin=NestedGrids />;<span id="NestedGrids"></span>[[NestedGrids]]
: Number of ROMS nested and/or composed grids, usually 1.<section end=NestedGrids />

===Compiler-Specific Options===

These flags are used by the files inside the '''Compilers''' directory.

<section begin=USE_DEBUG />;<span id="USE_DEBUG"></span>[[USE_DEBUG]]
: Used to turn on symbolic debug information with no optimization. Set it to '''on''' or leave it blank.<section end=USE_DEBUG />

<section begin=USE_MPI />;<span id="USE_MPI"></span>[[USE_MPI]]
: Used to turn on distributed-memory parallelization in ROMS and linking to the [http://www.llnl.gov/computing/tutorials/mpi/ Message Passing Interface (MPI)] library. Set it to '''on''' or leave it blank.<section end=USE_MPI />

<section begin=USE_MPIF90 />;<span id="USE_MPIF90"></span>[[USE_MPIF90]]
: Used to turn on the [http://www-unix.mcs.anl.gov/mpi/mpich1 MPICH] implementation of MPI via the compilation script '''mpif90''' which is required in some Linux operating systems without a native MPI library ('''-lmpi'''). This definition macro is also convenient when there are several Fortran-90 compilers ('''ifort''', '''pgf90''', '''pathf90''') in the system that use the '''mpif90''' compilation command. In this case, the user needs turn on both '''MPI''' and '''MPIF90''' definitions and specify the appropriate Fortran-90 compiler in '''FORT''' macro definition. Set it to '''on''' or leave it blank.<section end=USE_MPIF90 />

<section begin=USE_OPENMP />;<span id="USE_OPENMP"></span>[[USE_OPENMP]]
: Used to turn on shared-memory parallelization in ROMS using the [http://www.llnl.gov/computing/tutorials/openMP/ OpenMP] compiler directives. Set it to '''on''' or leave it blank.<section end=USE_OPENMP />

<section begin=USE_LARGE />;<span id="USE_LARGE"></span>[[USE_LARGE]]
: Used to turn on 64-bit floating-point number precision. Set it to '''on''' or leave it blank.<section end=USE_LARGE />

<section begin=NETCDF_INCDIR />;<span id="NETCDF_INCDIR"></span>[[NETCDF_INCDIR]]
: The location of the '''netcdf.mod''' and '''typesizes.mod''' files from [http://www.unidata.ucar.edu/software/netcdf/ NetCDF].<section end=NETCDF_INCDIR />

<section begin=NETCDF_LIBDIR />;<span id="NETCDF_LIBDIR"></span>[[NETCDF_LIBDIR]]
: The location of the [http://www.unidata.ucar.edu/software/netcdf/ NetCDF] library.<section end=NETCDF_LIBDIR />

<section begin=USE_NETCDF4 />;<span id="USE_NETCDF4"></span>[[USE_NETCDF4]]
: Set this to '''on''' if linking against the NetCDF4 library, which needs the [http://hdfgroup.com/HDF5/ HDF5] library and therefore:<section end=USE_NETCDF4 />

<section begin=HDF5_LIBDIR />;<span id="HDF5_LIBDIR"></span>[[HDF5_LIBDIR]]
: The location of the [http://hdfgroup.com/HDF5/ HDF5] library.<section end=HDF5_LIBDIR />

<section begin=FORT />;<span id="FORT"></span>[[FORT]]
The user also needs to specify the appropriate Fortran-90 compiler to use. The name of the Fortran-90 compiler depends on the operating system (see table below). This name is also used to determine the compiling configuration file to include in the '''makefile''' by concatenating the operating system ('''uname -s''') and the chosen compiler name.

{| border="1" cellspacing="0" cellpadding="5" align="center"
! Operating System
! Compiler(s)
|-
| AIX
| '''xlf'''
|-
| ALPHA
| '''f90'''
|-
| CYGWIN
| '''g95''', '''df''', '''ifort'''
|-
| Darwin
| '''f90''', '''gfortran''', '''xlf'''
|-
| Linux
| '''ftn''', '''ifc''', '''ifort''', '''pgi''', '''path''', '''g95''', '''gfortran'''
|-
| SunOS
| '''f95'''
|-
| SunOS/Linus
| '''ftn'''
|-
| UNICOS-mp
| '''ftn'''
|-
|}

Note that others can be added as needed, using the existing files as templates.<section end=FORT />

===Local File Options===
<section begin=BINDIR />;<span id="BINDIR"></span>[[BINDIR]]
: Directory path for the compiled executable.<section end=BINDIR />

<section begin=BUILD_DIR />;<span id="BUILD_DIR"></span>[[BUILD_DIR]] (Fromerly SCRATCH_DIR)
: Directory for '''make''' temporary objects. By default it is set to '''Build'''. If building several projects simultaneously, you need to specify different values for each project. Otherwise, you will compile the wrong files. This directory and the files within are removed when executing a '''make clean'''.<section end=BUILD_DIR />

==Providing the Environment==

Before compiling, you will need to find out some background information:
#What is the name of your compiler?
#What is returned by '''uname -s''' on your system?
#Is there a working NetCDF library?
#Where is it?
#Was it built with the above compiler?
#Do you have access to MPI or OpenMP?
As described more fully in [[gmake]], the '''makefile''' will be looking for a file in the '''Compilers''' directory with the combination of your operating system and your compiler. For instance, using Linux and the Pathscale compiler, the file would be called '''Linux-path.mk'''. Is the corresponding file for your system and compiler in the '''Compilers''' directory? If not, you will have to create it following the existing examples there.

Next, there are two ways to provide the location for the NetCDF files (and optional HDF5 library). One is by editing the corresponding lines in your system-compiler file. Another way is through the Unix environment variables. If you are always going to be using the same compiler on each system, you can edit your '''.profile''' or '''.login''' files to globally set them. Here is an example for '''tcsh''':
setenv NETCDF_INCDIR /usr/local/netcdf4/include
setenv NETCDF_LIBDIR /usr/local/netcdf4/lib
setenv HDF5_LIBDIR /usr/local/hdf5/lib
The '''ksh/bash''' equivalent is:
export NETCDF_INCDIR=/usr/local/netcdf4/include
export NETCDF_LIBDIR=/usr/local/netcdf4/lib
export HDF5_LIBDIR=/usr/local/hdf5/lib

For the rest, move on to [[build Script]].

gmake

2024-01-24T18:07:19Z

Robertson:

<div class="title">gmake</div>
This article first appeared in the [http://www.arsc.edu/support/news/HPCnews.shtml|ARSC HPC Newsletter]. It as been revised and updated to more closely match the ROMS [[makefile]].

Over the years, the community has moved from the stance of writing portable '''Makefiles''' to a stance of just using a powerful, portable '''make'''. The [[make]] section described a portable subset of '''make''' features. We now delve into some of the more powerful tools available in [http://www.gnu.org/software/make/ GNU make]. See also [http://www.oreilly.com/catalog/make3/ Managing projects with GNU Make] by Robert Mecklenburg, 2005.

__TOC__

==Pattern Rules==

The core of '''make''' hasn't changed in decades, but concentrating on '''gmake''' allows one to '''make''' use of its nifty little extras designed by real programmers to help with real projects. The first change that matters to my '''Makefiles''' is change from suffix rules to pattern rules. I've always found the '''.SUFFIXES''' list to be odd, especially since '''.f90''' is not in the default list. Good riddance to all of that! For a concrete example, the old way to provide a rule for going from '''file.f90''' to '''file.o''' is:
<span class="red">.SUFFIXES: .o .f90 .F .F90<br />
.f90.o:
<TAB> $(FC) -c $(FFLAGS) $<</span>
while the new way is:
<span class="red">%.o: %.f90
<TAB> $(FC) -c $(FFLAGS) $<</span>
In fact, going to a uniform '''make''' means that we can figure out what symbols are defined and use their standard values - in this case, '''$(FC)''' and '''$(FFLAGS)''' are the built-in default names for the compiler and its options. If you have any questions about this, you can always run '''make''' with the '''-p''' (or '''--print-data-base''') option. This prints out all the rules '''make''' knows about, such as:
<span class="red"># default<br/>
COMPILE.f = $(FC) $(FFLAGS) $(TARGET_ARCH) -c</span>
Printing the rules database will show variables that '''make''' is picking up from the '''environment''', from the '''Makefile''', and from its built-in rules - and which of these sources is providing each value.

==Assignments==

In the old days, I only used one kind of assignment: '''=''' (equals sign). '''Gmake''' has several kinds of assignment (other makes might as well, but I no longer have to know or care). An example of the power of '''GNU make''' is shown by an example from my Cray X1 '''Makefile'''. There is a routine which runs much more quickly if a short function in another file is inlined. The way to accomplish this is through the <span class="red">-O inlinefrom=file</span> directive to the compiler. I can't add this option to '''FFLAGS''', since the inlined routine won't compile with this directive - it is only the one file that needs it. I had:
<span class="red">FFLAGS = -O 3,aggress -e I -e m
FFLAGS2 = -O 3,aggress -O inlinefrom=lmd_wscale.f90 -e I -e m<br />
lmd_skpp.o:
<TAB> $(FC) -c $(FFLAGS2) $*.f90</span>
The first change I can make to this using other assignments is:
<span class="red">FFLAGS := -O 3,aggress -e I -e m
FFLAGS2 := $(FFLAGS) -O inlinefrom=lmd_wscale.f90</span>
The <span class="red">:=</span> assignment means to evaluate the right hand side immediately. In this case, there is no reason not to, as long as the second assignment follows the first one (since it's using the value of '''$(FFLAGS)'''. For the plain equals, '''make''' doesn't evaluate the right-hand side until its second pass through the '''Makefile'''. However, '''gmake''' supports an assignment that avoids the need for '''FFLAGS2''' entirely:
<span class="red">lmd_skpp.o: FFLAGS += -O inlinefrom=lmd_wscale.f90</span>
What this means is that for the target '''lmd_skpp.o''' only, append the inlining directive to '''FFLAGS'''. I think this is pretty cool!

One last kind of assignment is to set the value only if there is no value from somewhere else (the environment, for instance):
<span class="red">FC ?= mpxlf90_r</span>
If we use <span class="red">:=</span> or <span class="red">=</span>, we would override the value from the environment.

==Include and a Few Functions==

One reasonably portable '''make''' feature is the '''include''' directive. It can be used to clean up the '''Makefile''' by putting bits in an include file. The syntax is simply:
<span class="red">include file</span>
and we use it liberally to keep the project information neat. We can also include a file with the system/compiler information in it, assuming we have some way of deciding which file to include. We can use '''uname -s''' to find out which operating system we're using. We also need to know which compiler we're using.

One user-defined variable is called [[build Script#FORT | FORT]], the name of the Fortran compiler. This value is combined with the result of '''uname -s''' to provide a machine and compiler combination. For instance, '''ftn''' on Linux is the Cray cross-compiler. This would link to a different copy of the [[NetCDF]] library and use different compiler flags than the Intel compiler which might be on the same system.
<span class="red"># The user sets Fortran Compiler:<br />
FORT ?= ftn<br />
OS := $(shell uname -s| sed 's/[\/ ]/-/g'))<br />
include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk</span>
We pick one include file at compile time, here picking '''Linux-ftn.mk''', containing the Cray cross-compiler information. The value '''Linux''' comes from the '''uname''' command, the '''ftn''' comes from the user, and the two are concatenated. The sed command will turn the slash in '''UNICOS/mp''' into a dash; the native Cray include file is '''UNICOS-mp-ftn.mk'''. Strip is a built-in function to strip away any extra white space.

Another tricky system is '''CYGWIN''', which puts a version number in the '''uname''' output, such as '''CYGWIN_NT-5.1'''. GNU '''make''' has quite a few built-in functions plus allows the user to define their own functions. One of the built-in functions allows us to do string substitution:
<span class="red">MACHINE := $(patsubst CYGWIN_%,CYGWIN,$(MACHINE))</span>
In '''make''', the '''%''' symbol is a sort of wild card, much like '''*''' in the shell. Here, we match '''CYGWIN''' followed by an underscore and anything else, replacing the whole with simply '''CYGWIN'''. Another example of a built-in function is the substitution we do in:
<span class="red">objects = $(subst .F,.o,$(sources))</span>
where we build our list of objects from the list of sources. There are quite a few other functions, plus the user can defined their own. From the book:
:GNU make supports both built-in and user-defined functions.
:A function invocation looks much like a variable reference, but
:includes one or more parameters separated by commas. Most built-in
:functions expand to some value that is then assigned to a variable
:or passed to a subshell. A user-defined function is stored in a
:variable or macro and expects one or more parameters to be passed
:by the caller.

We will show some user-defined functions below.

==Conditionals==

We used to have way too many Makefiles, a separate one for each of the '''serial/MPI/OpenMP''' versions on each system (if supported). For instance, the name of the IBM compiler changes when using '''MPI'''; the options change for '''OpenMP'''. The compiler options also change when using '''64-bit''' addressing or for debugging, which we were manually setting. A better way to do this is to have the user select '''64-bit''' or not, '''MPI''' or not, etc, then use conditionals. A complete list of the user definable '''make''' variables is given in [[makefile]].

'''GNU make''' supports two kinds of '''if''' test, '''ifdef''' and '''ifeq''' (plus the negative versions '''ifndef''', '''ifneq'''). The example from the book is:
<span class="red">ifdef COMSPEC
# We are running Windows
else
# We are not on Windows
endif</span>

An example from the IBM include file is:
<span class="red">ifdef USE_DEBUG
FFLAGS += -g -qfullpath
else
FFLAGS += -O3 -qstrict
endif</span>
To test for equality, an example is:
<span class="red">ifeq ($(USE_MPI),on)
# Do MPI things
endif</span>
or
<span class="red">ifeq "$(USE_MPI)" "on"
# Do MPI things
endif</span>
The user has to set values for the '''USE_MPI''', '''USE_OPENMP''', '''USE_DEBUG''', and '''USE_LARGE''' switches in the
'''Makefile''' ''before'' the compiler-dependent piece is included:
<span class="red">USE_MPI ?= on
USE_OPENMP ?=
USE_DEBUG ?=
USE_LARGE ?= on</span>
The '''Makefile''' uses the conditional assign '''?=''' in case a build script is used to set them in the environment. Be sure to leave the switches meant to be off set to an empty string - the
string '''off''' will test true on an '''ifdef''' test.

==Multiple Source Directories the ROMS Way==

There's more than one way to divide your sources into separate directories. The choices we have made include non-recursive '''make''' and putting the temporary files in their own '''$(BUILD_DIR)''' directory. These include the '''.f90''' files which have been through the C preprocessor, object files, module files, and libraries.

===Directory Structure===

The directory structure of the source code has the top directory, a '''Master''' directory, a '''ROMS''' directory with a number of subdirectories, and several other directories. '''Master''' contains the main program while the rest contain sources for libraries and other files. Note that the bulk of the source code gets compiled into files that become libraries with the '''ar''' command, one library per directory. There is also a '''Compilers''' directory for the system- and compiler-specific '''Makefile''' components.

===Conditionally Including Components===

The '''makefile''' will build the lists of libraries to create and source files to compile. They start out empty at the top of the '''makefile''':
<span class="red">sources :=
libraries :=</span>
That's simple enough, but the list of directories to search for these sources will depend on the options chosen by the user, not just in the '''make''' [[makefile | options]], but inside the '''ROMS_HEADER''' file as well. How does this happen? Once '''make''' knows how to find the '''ROMS_HEADER''', it is used by '''cpp''' to generate an include file telling '''make''' about these other options.
<span class="red">MAKE_MACROS := Compilers/make_macros.mk
MACROS := $(shell cpp -P $(ROMS_CPPFLAGS) Compilers/make_macros.h > \
$(MAKE_MACROS); $(CLEAN) $(MAKE_MACROS))</span>
The '''make_macros.h''' file contains blocks such as:
<span class="red">#ifdef SWAN_COUPLING
USE_SWAN := on
#else
USE_SWAN :=
#endif</span>
The resulting '''make_macros.mk''' file will simply end up with either
<span class="red">USE_SWAN := on</span>
or
<span class="red">USE_SWAN :=</span>
This file can then be included by the '''makefile''' and the variable '''USE_SWAN''' will have the correct state for this particular compilation. We can now use it and all the similar flags to build a list of directories.

We initialize two lists:
<span class="red">modules :=
includes := ROMS/Include</span>
Add the adjoint bits:
<span class="red">ifdef USE_ADJOINT
modules += ROMS/Adjoint
endif
ifdef USE_ADJOINT
includes += ROMS/Adjoint
endif</span>
Add the bits we'll always need:
<span class="red">modules += ROMS/Nonlinear \
ROMS/Functionals \
ROMS/Utility \
ROMS/Modules
includes += ROMS/Nonlinear \
ROMS/Utility \
ROMS/Drivers</span>
Then we add in some more:
<span class="red">ifdef USE_SWAN
modules += Waves/SWAN/Src
includes += Waves/SWAN/Src
endif<br />
modules += Master
includes += Master Compilers</span>
Now that our lists are complete, let's put them to use:
<span class="red">vpath %.F $(modules)
vpath %.h $(includes)
vpath %.f90 $(BUILD_DIR)
vpath %.o $(BUILD_DIR)<br />
include $(addsuffix /Module.mk,$(modules))<br />
CPPFLAGS += $(patsubst %,-I%,$(includes))</span>
#'''vpath''' is a standard '''make''' feature for providing a list of directories for '''make''' to search for files of different types. Here we are saying that '''*.F''' files can be found in the directories provided in the '''$(modules)''' list, and so on for the others.
#For each directory in the '''$(modules)''' list, '''make''' will include the file '''Module.mk''' that is found there. More on these later.
#For each directory in the '''$(includes)''' list, add that directory to the list searched by '''cpp''' with the '''-I''' flag.

==User-defined '''make''' Functions==

The '''Module.mk''' fragments mentioned before call some user-defined functions. It's time to show these functions and talk about how they work. They get defined in the top Makefile:
<span class="red">#--------------------------------------------------------------------------
# Make functions for putting the temporary files in $(BUILD_DIR)
#--------------------------------------------------------------------------<br />
# $(call source-dir-to-binary-dir, directory-list)
source-dir-to-binary-dir = $(addprefix $(BUILD_DIR)/, $(notdir $1))<br />
# $(call source-to-object, source-file-list)
source-to-object = $(call source-dir-to-binary-dir, \
$(subst .F,.o,$(filter %.F,$1)))<br />
# $(call f90-source, source-file-list)
f90-source = $(call source-dir-to-binary-dir, \
$(subst .F,.f90,$1))<br />
# $(call make-library, library-name, source-file-list)
define make-library
libraries += $(BUILD_DIR)/$1
sources += $2<br />
$(BUILD_DIR)/$1: $(call source-dir-to-binary-dir, \
$(subst .F,.o,$2))
$(AR) $(ARFLAGS) $$@ $$^
$(RANLIB) $$@
endef<br />
# $(call one-compile-rule, binary-file, f90-file, source-files)
define one-compile-rule
$1: $2 $3
cd $$(BUILD_DIR); $$(FC) -c $$(FFLAGS) $(notdir $2)<br />
$2: $3
$$(CPP) $$(CPPFLAGS) $$(MY_CPP_FLAGS) $$< > $$@
$$(CLEAN) $$@
endef<br />
# $(compile-rules)
define compile-rules
$(foreach f, $(local_src), \
$(call one-compile-rule,$(call source-to-object,$f), \
$(call f90-source,$f),$f))
endef</span>
#We define a function to convert the path from the source directory to the '''Build''' directory, called '''source-dir-to-binary-dir'''. Note that the '''Build''' directory is called '''$(BUILD_DIR)''' here. All it does is strip off the leading directory with the the built-in function '''notdir''', then paste on the \code{Build} directory.
#Next comes '''source-to-object''', which calls the function above to return the object filename when given the source filename. It assumes that all sources have a '''.F''' extension.
#A similar function is '''f90-source''', which returns the name of the intermediate source which is created by '''cpp''' from our '''.F''' file.
#The '''Module.mk''' fragment in each library source directory invokes '''make-library''', which takes the library name and the list of sources as its arguments. The function adds its '''library''' to the global list of '''libraries''' and provides rules for building itself. The double dollar signs are to delay the variable substitution. Note that we call '''source-dir-to-binary-dir''' instead of '''source-to-object''' - this is a work-around for a make bug.
#The next, '''one-compile-rule''', takes three arguments: the '''.o''' filename, the '''.f90''' filename, and the '''.F''' filename. From these, it produces the '''make''' rules for running '''cpp''' and the compiler.<br /><br />A note on directories: '''make''' uses '''vpath''' to find the source file where it resides. It would be possible to compile from the top directory and put the '''.o''' file in '''Build''' with the appropriate arguments, but I don't know how to get the '''.mod''' file into '''Build''' short of a '''mv''' command. Likewise, if we compile in the top directory, we need to know the compiler option to tell it to look in '''Build''' for the '''.mod''' files it uses. Doing a '''cd''' to '''Build''' before compiling is just simpler.
#The last, '''compile-rules''', is given a list of sources, then calls '''one-compile-rule''' once per source file.

Again, you can invoke '''make -p''' to see how '''make''' internally transforms all this into actual targets and rules.

==Library '''Module.mk'''==

In each library directory, there is a file called '''Module.mk''' which gets included by the top level '''makefile'''. These '''Module.mk''' bits build onto the list of sources and libraries to be compiled and built, respectively. These '''Module.mk''' files look something like:
<span class="red">local_sub := ROMS/Nonlinear
local_lib := libNLM.a<br />
local_src := $(wildcard $(local_sub)/*.F)<br />
$(eval $(call make-library,$(local_lib),$(local_src)))<br />
$(eval $(compile-rules))</span>
First, we provide the name of the current directory and the library to be built from the resident sources. Next, we use the '''wildcard''' function to search the subdirectory for these sources. Note that every '''.F''' file found will be compiled. If you have half-baked files that you don't want used, make sure they have a different extension.

Each subdirectory is resetting the '''local_src''' variable. That's OK because we're saving the values in the global '''sources''' variable inside the '''make-library''' function, which also adds the local library
to the '''libraries''' list. The '''compile-rules''' function uses this '''local_src''' variable to generate the rules for compiling each file, placing the resulting files in the '''Build''' directory.

==Main Program==

The main program is in a directory called '''Master''' and its '''Module.mk''' is similar to the library one:
<span class="red">local_sub := Master
local_src := $(wildcard $(local_sub)/*.F)<br />
local_objs := $(subst .F,.o,$(local_src))
local_objs := $(addprefix $(BUILD_DIR)/, $(notdir $(local_objs)))<br />
sources += $(local_src)<br />
ifdef LD_WINDOWS
$(BIN): $(libraries) $(local_objs)
$(LD) $(FFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS_WIN32) $(LDFLAGS)
else
$(BIN): $(libraries) $(local_objs)
$(LD) $(FFLAGS) $(LDFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS)
endif<br />
$(eval $(compile-rules))</span>
Instead of a rule for building a library, we have a rule for building a binary. In this case, the name of the binary is defined elsewhere. The binary depends on the '''libraries''' getting compiled first, as well as the local sources. During the link, the '''$(libraries)''' are compiled from the sources in the other directories, while '''$(LIBS)''' are external libraries such as '''NetCDF'''.

==Top Level Makefile==

Now we get to the glue that holds it all together. We've covered many things so far, but there's still a few bits which might be confusing:
#There can be rare cases where you might have special code for some systems. You can check which system you are on in the '''.F''' file with:<div class="box"><span class="red">#ifdef X86_64<br />! special stuff<br />#endif</span></div>To be sure this is defined on each \code{X86\_64} system, it has to be passed to '''cpp''':<div class="box"><span class="red">CPPFLAGS += -D$(shell echo ${OS} | tr "-" "_" | tr [a-z] [A-Z])<br />CPPFLAGS += -D$(shell echo ${CPU} | tr "-" "_" | tr [a-z] [A-Z])<br />CPPFLAGS += -D$(shell echo ${FORT} | tr "-" "_" | tr [a-z] [A-Z])<br /><br />CPPFLAGS += -D'ROOT_DIR="$(ROOTDIR)"'<br />ifdef ROMS_APPLICATION<br /> CPPFLAGS += $(ROMS_CPPFLAGS)<br /> CPPFLAGS += -DNestedGrids=$(NestedGrids)<br /> MDEPFLAGS += -DROMS_HEADER="$(HEADER)"<br />endif</span></div>This guarantees that '''CPPFLAGS''' will have terms in it such as:<div class="box"><span class="red"> -DLINUX -DX86_64 -DPGI<br /> -D'ROOT_DIR="/export/staffdata/kate/roms/trunk"' -DSHOREFACE<br /> -D'HEADER="shoreface.h"' -D'ROMS_HEADER="shoreface.h"'<br /> -DNestedGrids=1</span></div>
#For [[mod_strings.F]], there is a special case:<div class="box"><span class="red">$(BUILD_DIR)/mod_strings.f90: CPPFLAGS += -DMY_OS='"$(OS)"' \<br /> -DMY_CPU='"$(CPU)"' -DMY_FORT='"$(FORT)"' \<br /> -DMY_FC='"$(FC)"' -DMY_FFLAGS='"$(FFLAGS)"'</span></div>allowing ROMS to report in its output:<div class="box"><span class="red"> Operating system : Linux<br /> CPU/hardware : x86_64<br /> Compiler system : pgi<br /> Compiler command : pgf90<br /> Compiler flags : -O3 -tp k8-64 -Mfree<br /><br /> Local Root : /export/staffdata/kate/roms/trunk<br /> Header Dir : ./ROMS/Include<br /> Header file : shoreface.h</span></div>Though this doesn't seem to work on the Mac.
#The very first '''makefile''' I showed had a set list of files to remove on '''make clean'''. We now build a list, called '''clean_list''':<div class="box"><span class="red"> clean_list := core *.ipo $(BUILD_DIR)<br /><br />ifeq "$(strip $(BUILD_DIR))" "."<br /> clean_list := core *.o *.oo *.mod *.f90 lib*.a *.bak<br /> clean_list += $(CURDIR)/*.ipo<br />endif</span></div>In other words, we want to clean up the '''Build''' directory unless it happens to be the top level directory, in which case we only want to remove specific files there.
#'''all''' is the first target that gets seen by '''make''', making it the default '''target'''. In this case, we know there is only the one binary, whose name we know - the book shows what to do with more than one binary. Both '''all''' and '''clean''' are phony targets in that no files of those names get generated - make has the '''.PHONY''' designation for such targets. Also, the '''clean''' target doesn't require any compiler information, so the compiler include doesn't happen if the target is '''clean''':<div class="box"><span class="red">ifneq "$(MAKECMDGOALS)" "clean"<br /> include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk<br />endif</span></div>'''$(MAKECMDGOALS)''' is a special variable containing the current '''make''' target.
#We'll be creating different executable names, depending on which options we've picked:<div class="box"><span class="red">BIN := $(BINDIR)/romsS<br />ifdef USE_DEBUG<br /> BIN := $(BINDIR)/romsG<br />else<br /> ifdef USE_MPI<br /> BIN := $(BINDIR)/romsM<br /> endif<br /> ifdef USE_OpenMP<br /> BIN := $(BINDIR)/romsO<br /> endif<br />endif</span></div>{{note}}'''Note:''' In SVN revision 933 (January 26, 2019) all <span class="red">ocean_*.in</span> files were renamed to <span class="red">roms_*.in</span> and all <span class="red">ocean*</span> ROMS executables were renamed to <span class="red">roms*</span> in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository [https://www.myroms.org/projects/src/ticket/794 Trac ticket #794]. If you are working with a ROMS release prior to revision 933, you will find that <span class="red">romsS</span>, <span class="red">romsM</span>, and <span class="red">romsO</span> will read <span class="red">oceanS</span>, <span class="red">oceanG</span>, <span class="red">oceanM</span>, and <span class="red">oceanO</span> instead.<br /><br />
# The [[NetCDF]] library gets included during the final link stage. However, we are now using the Fortran 90 version of it which requires its module information as well. We just copy the '''.mod''' files into the '''Build''' directory:<div class="box"><span class="red">NETCDF_MODFILE := netcdf.mod<br />TYPESIZES_MODFILE := typesizes.mod<br /><br />$(BUILD_DIR)/$(NETCDF_MODFILE): | $(BUILD_DIR)<br /> cp -f $(NETCDF_INCDIR)/$(NETCDF_MODFILE) $(BUILD_DIR)<br /><br />$(BUILD_DIR)/$(TYPESIZES_MODFILE): | $(BUILD_DIR)<br /> cp -f $(NETCDF_INCDIR)/$(TYPESIZES_MODFILE) $(BUILD_DIR)</span></div>Old versions of NetCDF do not have the '''typesizes.mod''' file, in which case it has to be removed from the following dependency list:<div class="box"><span class="red">$(BUILD_DIR)/MakeDepend: makefile \<br /> $(BUILD_DIR)/$(NETCDF_MODFILE) \<br /> $(BUILD_DIR)/$(TYPESIZES_MODFILE) \<br /> | $(BUILD_DIR)</span></div>

#Then there is [[MakeDepend]] itself. This file is created by the [[Perl]] script [[sfmakedepend]]. [[MakeDepend]] gets created by '''make depend''' and included on '''make''''s second pass through the '''makefile''':<div class="box"><span class="red">depend: $(BUILD_DIR)<br /> $(SFMAKEDEPEND) $(MDEPFLAGS) $(sources) > $(BUILD_DIR)/MakeDepend<br /><br />ifneq "$(MAKECMDGOALS)" "clean"<br /> -include $(BUILD_DIR)/MakeDepend<br />endif</span></div>The dash before the '''include''' tells '''make''' to ignore errors so that '''make depend''' will succeed before the file exists. The '''MakeDepend''' file will contain the include and module dependencies for each source file, such as:<div class="box"><span class="red">Build/mod_diags.o: tile.h cppdefs.h globaldefs.h shoreface.h<br />Build/mod_diags.f90: tile.h cppdefs.h globaldefs.h shoreface.h<br />Build/mod_diags.o: Build/mod_kinds.o Build/mod_param.o Build/mod_diags.f90</span></div>Note that the '''.h''' files are included by '''cpp''', so that both the '''.f90''' and '''.o''' files become out of date when an include file is modified. Without the module dependencies, '''make''' would try to build the sources in the wrong order and the compiler would fail with a complaint about not finding '''mod_param.mod''', for instance.

==Final Warnings==

The cost of this nifty '''make''' stuff is:
#We're a little closer to the '''gnu make''' bugs here, and we need a newer version of '''gnu make''' than before (version 3.81, 3.80 if you're lucky). Hence this stuff at the top of the '''makefile''':<div class="box"><span class="red">NEED_VERSION := 3.80 3.81<br>$(if $(filter $(MAKE_VERSION),$(NEED_VERSION)),, \<br> $(error This makefile requires one of GNU make version $(NEED_VERSION).))</span></div>
#The '''Makefile''' dependencies get just a little trickier every change we make. Note that '''F90''' has potentially both '''include''' and '''module''' use dependencies. The book example uses the C compiler to produce its own dependencies for each source file into a corresponding '''.d''' file to be included by '''make'''. Our Fortran compilers are not so smart. For these hairy compiles, it's critical to have accurate dependency information unless we're willing to '''make clean''' between compiles.

Git

2024-01-24T17:51:56Z

Robertson:

<div class="title">Git</div>
ROMS source code is now distributed using [https://git-scm.com Git]. The [https://subversion.apache.org Subversion](SVN) repository is being phased out and will no longer be updated starting January 1<sup>st</sup>, 2025. The GitHub repository includes the full history of changes to the ROMS source code. There are command line and GUI Git clients available for nearly every operating system and a list of popular clients can be found [https://git-scm.com/downloads/guis here]. This page will help you get started with downloading ROMS with Git.

{{note}}'''Note:''' For instructions using the deprecated and soon to be discontinued myroms.org git repository, click [[Talk:Git|here]].

{{note}}'''Note:''' Details on using the deprecated and soon to be discontinued Subversion repository can be found [[Subversion|here]].

==Git Overview==
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. All the ROMS/TOMS files are stored in a Git repository on '''github.com'''.

This Git repository is the official version of the code which only the developers are allowed to change. Users should download the ROMS code to their local machines using a '''git client'''. Don't attempt to use a regular web browser to browse or download files from the Git repository - there are much better tools for interacting with the code repository.

We strongly recommend users always check out the current ''develop'' version since this has the most recent updates and bug fixes. The ''tags'' are kept largely as a historical record of stable releases at the conclusion of major code upgrades.

If you are making changes of your own, keep them in a separate branch, leaving the ''develop'' branch to track changes from the source. Git makes it ''so'' much easier to manage your own modifications than svn for those of us without write permission on the repository.

Below is a general description of how Git works. Please look at the [https://git-scm.com/book/en/v2 Pro Git] book for more detailed information. We have not yet tried any GUI clients but may add brief how-tos for the most popular GUIs at a later date.

==Configuring GIT==

Before downloading any of '''ROMS''' repositories, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading some '''''roms_test''''' input and observation NetCDF files. Otherwise, the '''Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box">more ~/.gitconf<br /><br /> [User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br /> [credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br /> [filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you clone the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a git repository, '''git client''' software must be installed on your local machine. Most Linux distributions come with git, so shell commands may be used without installing additional software. The general form of git commands is:

<span class="red">git</span> action <repository>

To check-out the files from the ROMS repository '''develop''' (latest version), enter (notice https instead of http):

> <span class="red">git clone <nowiki>https://github.com/myroms/roms.git</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If '''MyDir''' is omitted the code will be downloaded to a directory named '''roms'''. You only clone once because Git will keep track of the source, destination and a bunch of other information. For more detail on command line use and syntax, see the [https://git-scm.com/book/en/v2 Pro Git] book.

The idealized and realistic ROMS Test Cases and the Matlab processing software can be downloaded from:

> <span class="red">git clone <nowiki>https://github.com/myroms/roms_test.git </nowiki></span>
> <span class="red">git clone <nowiki>https://github.com/myroms/roms_matlab.git</nowiki></span>

We highly recommend that Users clone all the ROMS repositories from the same root directory in their computer and define the '''ROMS_ROOT_DIR''' variable in their computer shell login environment, specifying where the User cloned/downloaded the ROMS source code, Test Cases, and Matlab processing software. That is, use the value of pwd from the computer directory where you executed git clone https://github.com/myroms/roms.git

For bash shells:
export ROMS_ROOT_DIR=MyDownlodLocationDirectory

For csh/tcsh shells:
setenv ROMS_ROOT_DIR MyDownlodLocationDirectory

The build scripts will use this environmental variable when compiling any of the ROMS Test Cases without the need to customize the location of the ROMS source code. Also, it is used for loading the path of Matlab scripts in the startup.m configuration file.

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

> <span class="red">git pull</span>

Git will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

==Managing Your Own Modifications==
This assumes that you have a fresh clone of the myroms repository on the '''develop''' branch. You want to keep '''develop''' as a pure copy of the source version and keep your own changes in say the '''arctic''' branch. Start by creating a branch and switching to it:

> <span class="red">git branch arctic<br /></span>> <span class="red">git checkout arctic</span>

Now you can make whatever modifications you like (and test them out). To see what changed, you can use <span class="red">git status</span> and <span class="red">git diff</span>. To save your changes, do a:

> <span class="red">git commit -a</span>

Though if you add new files you will have to <span class="red">git add</span> them first.

===Getting the Updates===
It is easy to fetch and merge the updates. Start by making sure your directory has been cleanly checked in with <span class="red">git status</span>. Then you can update your '''develop''' branch:

> <span class="red">git checkout develop</span><br />> <span class="red">git pull</span>

Then bring the changes into your '''arctic''' branch:

> <span class="red">git checkout arctic</span><br />> <span class="red">git merge develop</span>

This will bring in everything that changed since your last <span class="red">git pull</span>, so you might find it easier to keep on top of things by doing this often, not putting it off for years. You can also bring in changes one at a time with <span class="red">git cherry-pick</span>. Again, check the Pro Git book for much more information about all of these operations.

Note that this will save your '''arctic''' branch locally, under the .git directory. You can back this up as you would any other important files you have.

==Useful Git Commands==
{| border="1" cellspacing="0" cellpadding="5" align="center"
|'''Command'''
|'''Description'''
|-
| '''git add''' <_FileName_>
| Add a file to the repository staging area
|-
| '''git branch'''
| List all local branches (the asterisk denotes the current branch)
|-
| '''git branch -a'''
| List all local and remote branches
|-
| '''git branch -d''' <_BranchName_>
| Delete a local branch
|-
| '''git branch -D''' <_BranchName_>
| Delete a local branch forcefully
|-
| '''git branch -m''' <_OldName_> <_NewName_>
| Rename a local branch
|-
| '''git checkout -b''' <_BranchName_>
| Create a new local branch and switch to it
|-
| '''git checkout''' <_BranchName_>
| Switch to an existing local or remote branch
|-
| '''git clone''' <_RepositoryURL_>
| Clone a public repository
|-
| '''git commit -am''' "<_Message_>"
| Commit changes to all files
|-
| '''git commit -am''' "<_Message_>" '''-m''' "<_MessageURL_>"
| Commit changes to all files with two message lines
|-
| '''git commit --amend -m''' "<_NewMessage_>"
| Amend previous commit message
|-
| '''git diff'''
| Show all changes between HEAD and working branch
|-
| '''git diff''' <_FileName_>
| Show changes for a specific file between HEAD and working branch
|-
| '''git difftool -d''' <_BranchName1_> <_BranchName2_>
| Compare the difference between two branches with '''`KDIFF3`'''
|-
| '''git fetch'''
| Retrieve new work done by other people
|-
| '''git log'''
| View repository changes
|-
| '''git log --oneline'''
| Show the list of commits in one-line format
|-
| '''git log --summary'''
| View repository detailed changes
|-
| '''git merge --no-ff --no-commit''' <_BranchName_>
| Merge a branch into the active branch
|-
| '''git pull'''
| Update local repository or branch to the newest origin commit
|-
| '''git push origin --delete''' <_BranchName_>
| Delete a remote branch
|-
| '''git push -u origin''' <_BranchName_>
| Push changes to the remote repository (and remember the branch)
|-
| '''git reset --hard HEAD~1'''
| Undo last commit
|-
| '''git revert''' <_CommitID_>
| Revert commit changes
|-
| '''git status'''
| Check/display changes to the repository or particular branch
|-
| '''git tag'''
| List all tags
|}

==Complicated Commands==

If a branch is renamed in a repository on GitHub, the local clone on a computer needs to updated:<div class="box">git branch -m OldName NewName<br />git fetch origin<br />git branch -u origin/NewName NewName<br />git remote set-head origin -a</div>

How to merge changes in the '''feature/name1''' branch into the '''feature/name2''' branch:<div class="box">git checkout feature/name1<br />git pull<br /><br />git checkout feature/name2<br />git pull<br />git merge --no-ff --no-commit feature/name1<br />git commit -am "Merging feature/name1 into feature/name2" -m "MessageURL"<br />git difftool -d feature/name1 feature/name2<br />git push -u origin feature/name2</div>

Documentation Portal

2024-01-24T17:33:25Z

Robertson:

Amarel

2024-01-23T14:58:44Z

Robertson:

<div class="title">Using Amarel</div>
==Getting Started==
This wiki page is a brief "Getting Started" introduction to running the ROMS ocean model and analyzing model output on the Rutgers University Office of Advanced Research Computing (OARC) cluster computer ''Amarel''.

===Amarel help ===
There is an OARC supported user guide at https://sites.google.com/view/cluster-user-guide to which you can refer for a lot of detailed guidance.

===Working with UNIX===
To work effectively on ''Amarel'' you need at least a beginner level of competency with the UNIX operating system.
If you are a total newcomer to UNIX computers you might find it useful to work through a structured tutorial; for example [https://ryanstutorials.net/linuxtutorial/ Ryan's Linux Tutorial].

{| class="wikitable"
|+ Basic UNIX commands
|-
! Command !! What it does !! Example
|-
| ls || list contents of a directory </br> files ending in .nc </br> long format list and hidden files || ls </br> ls *.nc </br> ls -al
|-
| cd || change directory || cd /projects/dmcs_1/courses/cod
|-
| pwd || print working directory to see where you are || pwd
|-
| cp || copy a file || cp source_file target_file
|-
| mv || rename a file </br> move file to new directory || mv file_name new_name </br> mv file_name dir1/dir2/new_name
|-
| rm || delete (remove) a file || rm filename
|-
| more || type contents of a file to screen one page at a time || more .bashrc
|-
| less || like ''more'' only with more features (unix geeks are so funny) </br>Read the man page for help || less .bashrc
|-
| man || get the manual page (help) for a command || man less
|-
| . || the directory you are in || cp /projects/dmcs_1/courses/job.sh '''.'''
|-
| .. || the directory above the one you are in || cd '''..'''
|-
| grep || find a string in a file </br> ''get regular expression and print (!)'' || grep COD_1DMIX *.h
|}

Your default UNIX ''shell'' (the "flavor" of your UNIX environment) is '''bash''', so for more advanced help it is often good to include "bash" as one of your search terms in a google search for help.

===Editing files===
One of the things that stumps newcomers to UNIX is making simple edits to files. Ryan's Linux Tutorial goes over the basics of the '''vi''' editor [https://ryanstutorials.net/linuxtutorial/vi.php here], but it can be a beast to get comfortable with.

A more intuitive editor for new users is '''nano''', the basics of which you can read about [https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/ here].

There is also a way to edit files in the web browser based '''OnDemand''' service that is described later in this guide.

==Rutgers VPN==
If you are not on campus, you will need to connect to the Rutgers VPN. To use Rutgers VPN, you must first enroll in Duo 2 Factor Authentication (2FA) (NetID+). Most users have probably already done this. If not, navigate to https://netid.rutgers.edu/setupTwoFactorAuthentication.htm and follow the instructions.

Once you are enrolled in NetID+ you will need to activate the VPN service, if you have not already. Navigate to https://soc.rutgers.edu/vpn/ and click the gray button titled '''Service Activation'''. If you are not already logged in you may be asked to login and/or approve your login with the Duo Mobile app. To activate the VPN service, click the checkbox next to '''Remote Access VPN, Cisco AnyConnect Access for Rutgers''' and click the '''Activate Services''' button.

You are now ready to download the Cisco VPN client and connect to the Rutgers VPN. Complete instructions can be found [https://soc.rutgers.edu/vpn/ here] by clicking the red button titled '''General VPN Instructions''' to download the PDF. The instructions are geared towards Windows so if you are using a Mac you might find [https://soc.rutgers.edu/vpn/apple/ this page] more helpful. In most cases, regardless of your operating system, pointing your browser to https://vpn.rutgers.edu/ and logging in with your NetID will lead you to downloading the correct VPN client.

Once installed, open the '''Cisco AnyConnect''' client and type '''vpn.rutgers.edu''' in the box and click '''Connect''':

[[Image:AnyConnect_Connect.png]]

In the next window, shown below, '''Username''' will be your NetID.

In the second line '''Password''' is the password for your NetID.

The third line is the '''Duo Action''' to complete the 2FA, which can be accomplished in one of 4 possible ways:

# Enter a 6 digit Duo Passcode. These are generated either by a Hard Token, by showing the passcode in the Duo Mobile App, or from a previous “SMS” request. Simply type in the 6 numbers and hit OK.
# Type the word “push”. This will send a push notification to the primary device you have enrolled with Duo through NetID+ with the option to Accept or Deny.
# Type the word “phone”. You will receive a phone call to the primary device you have enrolled with Duo through NetID+ with touch tone options to Accept or Deny.
# Type the word “sms”. You will receive a text message to the primary device you have enrolled with Duo through NetID+ containing passcodes you can use to login.

[[Image:AnyConnect_Credentials.png]]

Click '''OK''' and you should be connected to Rutgers VPN. You should have a small AnyConnect icon with a lock on it in the task tray (Windows [[Image:AnyConnect_win_connected.jpg]]) or in the menu bar (Mac OS [[Image:AnyConnect_mac_connected.png|25px]]).

==Connecting to Amarel with SSH==
In order to connect to ''Amarel'' and compile and run ROMS you will need an SSH/terminal client. If you already use an SSH/terminal application you are comfortable with then stick with that and adapt the instructions below accordingly. For these instructions to work you need to either be on campus or connected to the Rutgers VPN.

===Mac OS===
If you are on a Mac, open the '''Terminal''' app (found in the Applications -> Utilities folder) or '''iTerm''' app and type:
:<div class="box">ssh fakeuser@amarel.rutgers.edu</div>
replacing <span class="blue">fakeuser</span> with your NetID. When it asks if you want to continue connecting enter '''yes''', then enter your NetID password.

Having to type the above command and netid password every time you login is moderately tedious, but this username/password authentication can become annoying and time consuming when repeatedly using the '''scp''' command to copy files to ''Amarel''. Passwordless access and file transfer can be enabled using '''SSH keys''' by following the instructions below, or for more detail see the [https://sites.google.com/view/cluster-user-guide#h.jgwrkm9e9rwg OARC User Guide on this topic].

# If you are still connected to ''Amarel'', disconnect by entering '''exit'''
# At the prompt for your local machine, enter '''<span style="font-family: monospace;">nano ~/.ssh/config</span>''' and paste the block below at the end of the file:<div class="box">Host amarel<br />Hostname amarel.rutgers.edu<br />HostKeyAlias amarel<br />User NetID</div> replacing <span class="blue">NetID</span> with your own NetID. Save and exit (Ctrl+o Ctrl+x). This allows you to type '''<span style="font-family: monospace;">ssh amarel</span>''' to connect.
# Enter '''<span style="font-family: monospace;">ls ~/.ssh</span>'''. If there is a file listed called <span class="blue">id_rsa.pub</span> skip to 5
# Enter '''<span style="font-family: monospace;">ssh-keygen -t rsa</span>''', then hit return to accept the default location, enter a passphase twice (or leave blank for no passphrase)
# Copy the public portion of the key to ''Amarel'' by entering '''<span style="font-family: monospace;">scp ~/.ssh/id_rsa.pub amarel:.</span>'''
# SSH to ''Amarel'' ('''<span style="font-family: monospace;">ssh amarel</span>''') and check if you already have a .ssh folder:<div class="box">[rjdave@amarel1 ~]$ ls -l .ssh<br />total 33<br />-rw------- 1 rjdave rjdave 1069 Sep 8 2017 authorized_keys<br />-rw-r--r-- 1 rjdave rjdave 434 Jun 4 2021 config<br />-rw-r--r-- 1 rjdave rjdave 5992 Dec 2 13:47 known_hosts</div> You probably won't have all the files above but if you instead get the message '''<span style="font-family: monospace;"><div class="box">ls: cannot access .ssh: No such file or directory</span>'''</div>then enter the following command:</br>'''<span style="font-family: monospace;">mkdir .ssh && chmod 700 .ssh</span>'''. </br>These two commands (combined on one line) will create the .ssh folder and set the required permissions. Setting private user-only permissions ('''chmod 700 ...''') is important, because the system will not allow you to have your keys viewable by others users.
# Once you have a .ssh folder with the proper permissions, enter the following commands</br> '''<span style="font-family: monospace;">cat ~/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 640 ~/.ssh/authorized_keys</span>'''</br>then exit back to your local machine and execute '''<span style="font-family: monospace;">ssh amarel</span>''' again. It should no longer ask for your password.

===Windows===
For Windows, we recommend [https://mobaxterm.mobatek.net/download-home-edition.html MobaXterm] installer edition.
# Once installed open it, choose light or dark theme and click the '''Session''' button in the upper left.
# Choose '''SSH''' and enter '''amarel.rutgers.edu''' for '''Remote host''', check the '''Specify username''' box and enter your NetID in the box and click '''OK'''.
# You will be asked to type your NetID password and then asked if you want to save the password. If you choose yes you will be asked to set a master password to encrypt all your saved passwords.
# In the future you can click '''Sessions''' (not Session) and select '''amarel.rutgers.edu''' and not have to enter your password.

==Setting up your .bashrc==
The default software setup by OARC on ''Amarel'' does not include everything needed to compile and run ROMS. Ocean Modeling Group computing specialist David Robertson has configured, and maintains and updates, a repository of what you need for ROMS and other models that will be automatically loaded for every login session once you add some lines to the login script (.bashrc, .cshrc, etc.) in your home directory. Log in to ''Amarel'' with '''Terminal''', '''iTerm2''' or '''MobaXterm''' and edit your login script as described below.

Unless otherwise requested your default shell will be bash and the following lines shown below in <span class="red">red</span> and <span class="forestGreen">green</span> should be added (with nano or your preferred editor) near the top of your .bashrc after the import of the global definitions as shown:
:<div class="box"># Source global definitions<br />if [ -f /etc/bashrc ]; then<br /> . /etc/bashrc<br />fi<br /><br /><span class="red">ulimit -s unlimited<br />export MODULEPATH=/projects/dmcs_1/sw/modulefiles/Core:${MODULEPATH}<br />export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.10l %.4C %.6D %R"</span><br /><br /><span class="forestGreen"># Automatically load the roms module<br />module load roms</span></div>

The line with <span class="red">ulimit</span> allows you to take full advantage of computing resources (may be obsolete now but it won't hurt anything).

The first <span class="red">export</span> line will tell '''Lmod''' where to find the modules that will load the custom software.

The second <span class="red">export</span> line makes the '''squeue''' command (explained later) more useful.

The lines in <span class="forestGreen">green</span> are optional and will load the '''roms''' module automatically every time you login to ''Amarel''. They must be '''after''' the line that sets the '''MODULEPATH'''.

Once you have added these lines, log out and back in for them to take effect.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... then you may need to logout and login again so that .bashrc sets MODULEPATH as instructed above. If the error persists, you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Accessing the ROMS Source Code==

===Register as a ROMS user===
If you are not already a ROMS user, you will need to fill out the registration form found [https://www.myroms.org/register here] and wait for approval. Once approved, you will be able to checkout the ROMS source code. It is highly recommended that you use the GitHub myroms repository to obtain the ROMS source code. The ''Subversion (SVN)'' repository is being phased out and will no longer be available after '''December 31, 2024'''. If you manage your own software projects using GitHub, the concepts for software version control may already be familiar to you. Detailed directions can be found on the WikiROMS [[Git#Downloading_ROMS|Git]] page.

===Obtaining the ROMS source code===
There is nothing special about obtaining the ROMS source code on ''Amarel''. The git commands are the same as on any UNIX host.
Navigate to the directory where you want your ROMS source code to reside and execute this git clone command:
:<div class="box">git clone https://github.com/myroms/roms.git <my_src_dir></div>
replacing <my_src_dir> with what you want the source code directory to be called. You can also omit <my_src_dir> and git will automatically name the folder '''roms'''.

It can take a couple of minutes to download all of ROMS, but you only do this once.

Many users simply run this command from their home directory and use <span class="forestGreen">'''myroms'''</span> for <my_src_dir>, but the choice is yours depending on how you want to manage your ''Amarel'' space and how many ROMS projects you think you might end up working on.

==Loading and Unloading Modules==
Like many clusters, ''Amarel'' uses environment modules to load and unload software and configure the environment. Some commands you will find useful are:
:<div class="box">'''module help''' Display help message<br />'''module help <m1'''> Show help information for module <m1><br />'''module available''' Show all modules currently available<br />'''module whatis <m1>''' Show brief information about module <m1><br />'''module spider''' List all possible modules even if not currently available<br />'''module list''' List currently loaded modules<br />'''module load''' Load the specified module(s)<br />'''module unload''' Unload the specified module(s)<br />'''module swap <m1> <m2>''' Unload <m1> and then load <m2> (for switching versions of the same software)<br />'''module purge''' Unload all loaded modules</div>

===Loading the ROMS Module===
If you '''did not''' add the optional lines from the [[#Setting_up_your_.bashrc|Setting up your .bashrc]] section, you can easily set up your environment to compile and run ROMS by manually loading the '''roms''' module.
:<div class="box">module load roms<br />module list<br />Currently Loaded Modules:<br /> 1) intel/17.0.4 2) mvapich2/2.2 3) mct-roms/2.6.0 4) netcdf/4.6.2<br /> 5) esmf/8.0.0_nc4 6) parpack-roms/2.1 7) hdf5/1.10.4 8) roms/intel_nc4</div>
Notice that loading the '''roms''' module will actually load 7 other modules.

This will setup your environment to use the Intel compiler so remember to set '''FORT''' to '''ifort''' in your build script.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Your project/coursework working directory space==

We recommend you create a subdirectory of your home directory in which to work, so you can keep each ROMS project separate and organized.

You might call this '''working directory''' <span class="forestGreen">CoastalDynamics</span>, or <span class="forestGreen">Projects</span>, or simply <span class="forestGreen">cod</span> (for coastal ocean dynamics). You will be typing this a lot, so keep it simple.

To make a new directory called <span class="forestGreen">Projects</span>, enter this command

:<div class="box">mkdir Projects </div>

Change to this working directory with the command

:<div class="box">cd Projects </div>

In instructions that follow, when we refer to your '''working directory''' we mean this workspace.

==Configuring and Compiling ROMS==
You will compile and run ROMS in your project/coursework '''working directory''', so change to that directory first.

:<div class="box">cd <span class="forestGreen">Projects</span> </div>

Running the build script <span class="blue">build_roms.sh</span> is the recommended method for compiling ROMS on ''Amarel''. For any given project you will likely only need to change <span class="blue">build_roms.sh</span> to reset <span class="sandyBrown">ROMS_APPLICATION</span> and <span class="sandyBrown">MY_ROMS_SRC</span> and make sure <span class="sandyBrown">FORT</span> is set to '''ifort'''.

{{warning}} <span class="red">It is important to make sure '''USE_MY_LIBS''' is set to ''no'' or your compilation will fail.</span>

A template <span class="blue">build_roms.sh</span> script is in subdirectory '''ROMS/Bin''' of '''<my_src_dir>''' that you created when you downloaded the code. For example, if you created <span class="forestGreen">myroms</span> in your home directory the copy command would be

:<div class="box">cp ~/myroms/ROMS/Bin/build_roms.sh . </div>

Now open <span class="blue">build_roms.sh</span> with an editor (''e.g.'' '''nano''') and modify the line <span class="sandyBrown">MY_ROMS_SRC</span> to point to your choice for <my_src_dir>. So, you will have something like ...

:<div class="box"><span class="sandyBrown">export MY_ROMS_SRC=</span>{$HOME}/<span class="forestGreen">myroms</span> </div>

if the name you chose for <my_src_dir> was <span class="forestGreen">myroms</span>.

If this is your first time working with ROMS, a good starting place is to compile the default UPWELLING test case that is indicated in build_roms.sh with the default setting <span class="sandyBrown">export ROMS_APPLICATION</span>=UPWELLING.

To compile the UPWELLING example, you need to copy the configuration file <span class="blue">upwelling.h</span> to your working directory. This is in subdirectory '''ROMS/Include''' of your source code. For example, the command might be:

:<div class="box">cp ~/myroms/ROMS/Include/upwelling.h .</div>

Once your build script is configured and you have upwelling.h in your working directory you can compile ROMS by typing:

:<div class="box">./build_roms.sh -j 4</div>
where the number after -j indicates the number of compute cores to use in parallel to execute the compilation. The greater the number, the faster it goes.

However, the login node you will be compiling on is shared for the entire ''Amarel'' system. If you use a number larger than '''4''', or omit it altogether (which says use ''all'' cores on the login node) your build might be terminated by an administrator. Be a considerate user and keep the number low.

If compilation is processing successfully you will see a lot of information typed to your screen as each component of the source code is compiled. It will take a minute or so to complete.

If compilation ends quickly with errors like this:

:<div class="box">$ ./build_roms.sh -j 4 </br>rm -f -r core *.ipo /home/jwilkin/cod/Projects/Build_roms...</br>which: <span class="red">no mpif90</span> in (/usr/lpp/mmfs/bin:/usr/local/bin:/us...</br>cp -f /opt/intelsoft/serial/netcdf3/include/netcdf.mod /h...</br>cp: cannot stat ‘/opt/intelsofo... No such file or directory</br>make: *** No rule to make target netcdf.mod needed by ... Stop. </div>

... this means you have not run the <span class="blue">module load roms</span> command. Go back to the notes on [[Amarel#Loading_the_ROMS_Module]].

If compilation was successful, there will be a file named '''romsM''' that is the model executable. You can check for the file with this command:

:<div class="box">ls -l *</div>

You should see:

:<div class="box"><span class="blue">Build_roms</span> <span class="forestGreen">build_roms.sh romsM</span> upwelling.h </div>

<span class="blue">Build_roms</span> is a folder that build_roms.sh created to hold all the temporary files that the FORTRAN compilation process makes to combine into the romsM executable file.

==Running on the Amarel Compute Nodes==
When you '''ssh''' to ''Amarel'' you are connected to one of the login nodes. These nodes are to be used for file editing, transfering output and input files to/from your local computer, and modest code compiling tasks and analysis, but not for compute intensive tasks. Here we explain how to connect to the compute nodes for these larger tasks.

Note: Consult the Amarel status page (https://oarc.rutgers.edu/amarel-system-status/) before scheduling a job. Amarel is down for maintenance monthly.

===What ''partitions'' can you run on?===

Access to Amarel resources is controlled by ''partitions''. You can find out the partitions you have permission to access with the command

:<div class="box">sinfo -s</div>

This will show the partitions that everyone has access to. See the cluster guide [https://sites.google.com/view/cluster-user-guide#h.oeejy9yf80e4 entry on partitions] for an explanation of these.

If you belong to a research group that own a partition you should also see that listed, for example '''p_omg_1''' for the Ocean Modeling Group. This is the partition you want to use most of the time as your job will not get pre-empted by other owner users.

===Using the SLURM batch job scheduler===

''Amarel'' uses SLURM workload manager to schedule compute intensive tasks. The user configures a SLURM job script for each model run and submits this script with the '''sbatch''' command. The job script declares the resources required, such as number of CPUs for parallel jobs, maximum memory required, etc.

We have configured a simple template job script (for the ROMS UPWELLING example) that you can copy from '''/projects/dmcs_1/courses/job.sh''' into your working directory.

The UNIX command to copy a file is '''cp source_file target_file'''. So, you would enter

:<div class="box">cp /projects/dmcs_1/courses/job.sh .</div>

The syntax here uses a ''dot'' ('''.''') for the target_file, which causes the source_file (job.sh) to be copied to the present working directory without changing its name.

When typing this, or any, UNIX command, you can start typing just the beginning few characters of the path then press TAB to autocomplete the filename. If the completion is ambiguous, you can continue typing more characters and press TAB again. Pressing TAB twice in a row will show you the set of possible completions. This trick can save you a lot of tedious (mis)typing.

The contents of '''job.sh''' are shown below:

:<div class="box">#!/bin/bash<br >#SBATCH --partition=p_omg_1 # Partition. If you are not a member of OMG enter your own partition<br />#SBATCH --job-name=upwelling # Assign a short name to your job<br/>#SBATCH --nodes=1 # Number of nodes you require<br />#SBATCH --ntasks=4 # Total number of tasks you'll launch<br />#SBATCH --ntasks-per-node=4 # Number of tasks you'll launch on each node<br />#SBATCH --cpus-per-task=1 # Cores per task (>1 if multithread tasks)<br />#SBATCH --mem=6400 # Real memory (RAM) required (MB) per node<br />#SBATCH --time=00-00:05 # Total run time limit (DD-HH:MM)<br />#SBATCH --output=out.%N.%j # STDOUT output file<br />#SBATCH --error=err.%N.%j # STDERR output file (optional but recommended)<br />#SBATCH --export=ALL # Export you current env to the job env<br /><br />## It is important to have --mpi=pmi2 here or ROMS will not run<br />srun --mpi=pmi2 ./romsM roms_upwelling.in</div>

Note: The maximum run time is 14-00:00 (14 days). It is tempting to set this to the maximum in order to avoid a job ending too early. (''i.e.'' Slurm stopping a ROMS run before it is finished) However, if there is maintenance scheduled during that time period the job will not start until maintenance is complete. So a runtime as close as possible to the actual run time is recommended.

====Requesting a specific class of compute node====

For large jobs it makes sense to use all available cores on a node, and for low priority jobs to use older less fast nodes. You can do this by requesting a specific node number, but it is better to request a particular class of nodes so you don't unwittingly pick a node already in use. The classes available from oldest to newest nodes are:

# skylake (32 cores) (Phase I nodes) (nodes hal0116 through hal0120)
# cascadelake (40 cores) (Phase II nodes) (nodes hal0159 through hal0163)
# icelake (64 cores) (Phase V.B nodes) (nodes hal0299 through hal0302)

Make the node class request in the SLURM script and choose the maximum number of cores to match, ''e.g.''

:<div class="box">#SBATCH --constraint=cascadelake # Run on Phase II nodes (40 cores) <br>#SBATCH --ntasks=40 # Total number of tasks to launch (like -np to mpirun) <br>#SBATCH --ntasks-per-node=40 # Number of tasks to launch on each node
</div>

If you don't specifically request a class of nodes the SLURM scheduler will make its own choice on the most appropriate resources to use given your request. We don't know the basis of that choice.

If you request a class of node that is fully assigned to other users in your partition, your job will wait for the others users jobs to complete. So, coordinate with your colleagues if resource use is at a premium. You can use the '''squeue -p partition_name''' command to see what resources are being used by other users, ''e.g.''

:<div class="box">$ squeue -p p_omg_1 <br> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON) <br> 26975473 p_omg_1 fwd levinj R 17:15:42 14-00:00:00 80 2 hal[0159,0162]
</div>

===Running the ROMS Upwelling Example===
You may have noticed that the srun command above includes a file named <span class="blue">roms_upwelling.in</span>. You will need to copy this file and <span class="blue">varinfo.dat</span> from the ROMS/External directory of your <my_src_dir> ROMS source code to the directory where you compiled ROMS.

If, for example, you chose '''myroms''' as the name of your <my_src_dir> when you downloaded the code, you would enter

:<div class="box">cp ~/myroms/ROMS/External/roms_upwelling.in .</div>

The syntax here uses ''tilde'' (~) as shorthand for your home directory,

After you copy the files, you will need to make a couple of small edits (using ''e.g.'' '''nano''') to <span class="blue">roms_upwelling.in</span> to get this to work.

# Change the line with VARNAME to read <span class="sandyBrown">VARNAME = varinfo.dat</span> (''i.e.'' delete the ROMS/External part)
# Set <span class="sandyBrown">NtileI == 2</span> and <span class="sandyBrown">NtileJ == 2</span>.

The product of NtileI and NtileJ, ''i.e.'' 2 x 2 = 4, is the number of cores the model will run on in parallel. This number must match the number in the SLURM job script options '''--ntasks''' and, in this case, '''--ntasks-per-node'''

Now you can submit your job to '''sbatch''' with the command '''<span style="font-family: monospace;">sbatch job.sh</span>'''

===ROMS configurations for ''Coastal Ocean Dynamics'' class===
The files that will configure the examples we will study in class will be located in <span class="blue">/projects/dmcs_1/courses/cod</span>.

You will find instructions for configuring and running the examples at a separate Wiki page [[COD_Class_Examples]]

===Checking Job Status with squeue===
Detailed documentation for monitoring your SLURM jobs can be found [https://sites.google.com/view/cluster-user-guide#h.4bsndqufii8p here]. The easiest way to check whether your job is running is with the command '''<span style="font-family: monospace;">squeue -p p_omg_1</span>''' (or your own partition name if it is not p_omg_1).

You should see output something like this:

:<div class="box"> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON)<br /> 17197219 dmcs_1 watl_psas rjdave R 3-04:07:00 3-20:00:00 4 1 hal0035<br /> 17146609 dmcs_1 upwelling rjdave R 00:01:10 00:05:00 4 1 hal0035</div>

You can see that the job is running (the ‘R’ under ‘ST’), has been running for 1 minute, 10 seconds, and is running on node 35.

===Checking Progress with tail===
You can check ROMS progress by using the tail command on the '''output''' file. For the upwelling job above the file would be called <span class="blue">out.hal0035.17146609</span> so the command '''<span style="font-family: monospace;">tail out.hal0035.17146609</span>''' will show you the most recent 10 lines written to the output log and will most likely tell you what time-step the model is on. Using the -f option ('''<span style="font-family: monospace;">tail -f out.hal0035.17146609</span>''') will output appended data as the file grows. Ctrl-C will escape the display.

===Did my UPWELLING job run?===
If the UPWELLING test case was successful there should be an output log file named something like <span class="blue">out.hal0035.17146609</span> that shows no errors, and there should be 4 output NetCDF files ...
:<div class="box">ls -l *.nc </br>roms_avg.nc </br>roms_dia.nc </br>roms_his.nc </br>roms_rst.nc </div>

===Cancelling a Job===
To safest way cancel a queued or running job is to use its jobid. A job can be canceled by name but that is not recommended. To cancel the upwelling job in the example above you would issue the command '''<span style="font-family: monospace;">scancel 17146609</span>'''. If the job has not yet started, it will be removed from the queue. If it is running, all child processes will be killed and the job will be removed from the queue. You are only able cancel jobs that you own.

===Running an interactive session on a compute node===
It is possible to conduct your work interactively on one of the compute nodes (instead of the login node). For most work we will be doing in class this is not necessary, but if for some reason you have a job that needs many processors or a large amount of memory, and you want to run it interactively - say, to simply check that everything is in order for it to start correctly - there are instructions in the Cluster User Guide [https://sites.google.com/view/cluster-user-guide#h.26x9sbburvsg here].

For compute intensive interactive work, such as model analysis using Python or MATLAB, we recommend using the OnDemand interface to launch an interactive session on a set of compute nodes.

==Using the OnDemand desktop environment==
The '''OnDemand''' service allows you to connect to ''Amarel'' using a graphical desktop environment in a web browser. One of the most useful features of '''OnDemand''' is that it allows you to analyze model output by Python or Matlab on the cluster compute nodes.

You need to be running a Rutgers VPN to use OnDemand.

Navigate your browser to https://ondemand.hpc.rutgers.edu/pun/sys/dashboard and login with your NetID and password. At the top of the page click '''Interactive Apps'''.

===Python Jupyter notebook:===
# Select '''Personal Jupyter''' from the list on the left
# Make your selections for '''number of hours''' your session can remain active (''e.g.'' 3 hours), the '''number of cores''' to run on (1), and '''memory''' (8 GB is an OK choice)
# For ''Partition'' enter ''e.g.'' <span class="blue">p_omg_1</span> if you are in OMG. Leave ''Reservation'' and ''Slurm feature'' blank. For ''conda path'' enter <span class="blue">/projects/dmcs_1/miniconda3</span> and for ''conda environment'' enter <span class="blue">/projects/dmcs_1/sw/packages/xroms/py38</span>. Click [[Media:Amarel Jupyter config.png | here]] to see what the completed launch page looks like.
# Click '''Launch''' and wait. This can take a couple minutes.
# You should see sub-window of the web page like this </br> [[Image:JupyterStarting.png|600px]] </br></br>When the session is ready it changes to this: </br> [[Image:JupyterRunning.png|600px]]
# Start your notebook by clicking the '''Connect to Jupyter, Anaconda version 5.1.0''' button: (If this does not appear, check that you correctly entered the '''conda path''' and '''conda environment''' at step 3 above.) </br>[[Image:JupyterConnect.png|400px]]
# This should place you in a file browse window from which you can navigate to one of your existing Jupyter notebooks, or, to start a new Untitled notebook, select ''Python 3 (ipykernel)'' from the ''New'' pull-down menu on the right side of the page [[Image:NewPythonNotebook.png]]
# Once your notebook is running, you can import the xroms package as you would any other toolbox, by entering '''<span style="font-family: monospace;">import xroms</span>''' then hold down the shift key and hit the return/enter key. You will see an asterisk ['''*'''] in the square brackets.
# Once that asterisk changes to a '''1''', the xroms python module has been loaded.

This '''conda''' environment built for DMCS_1 makes available many of the Python modules you are used to using.

===Matlab:===
# Select MATLAB from the list on the left
# As for Step 2 above for Python, choose time, cores, and memory
# Select MATLAB version, and enter <span class="blue">p_omg_1</span> (or your own group name) in partition.
# Click '''Launch''' and wait. This can take a couple minutes.
# Once the '''Launch noVNC in New Tab''' button appears, click it and a MatLab GUI will open.

You are running in a graphical desktop environment, so by default Matlab will look in Documents/MATLAB/startup.m for your personal Matlab configuration file.

Popular MATLAB utilities for working with ROMS output are in <span class="blue">/projects/dmcs_1/sw/packages/matlab-tools</span>. These include the myroms.org utilities in subdirectory <span class="blue">roms_matlab</span>, and John Wilkin's toolbox in <span class="blue">roms_wilkin</span>.

===Reconnecting to an interactive session===
A particularly nifty feature of '''OnDemand''' is that your interactive session is independent of the browser window instance.

Say you are working on your office computer using a Jupyter notebook via '''OnDemand''' but haven't finished all you want to do by the end of the day. If you leave that session running, you can go home and connect once again to '''OnDemand''' and you will see your session in the list of Interactive Sessions. Launch that session and you will be placed right back where you were. '''OnDemand''' is simply a web interface to code running on ''Amarel'', so you can connect from any browser.

Of course, this only works if you selected a '''number of hours''' in Step 2 above that will allow your session to remain open.

As long as the session is active the ''Amarel'' resources (cores and memory) are reserved for you and no one else. So, be respectful of other users and close out an interactive session once you are done.

Amarel

2024-01-22T15:38:51Z

Robertson:

<div class="title">Using Amarel</div>
==Getting Started==
This wiki page is a brief "Getting Started" introduction to running the ROMS ocean model and analyzing model output on the Rutgers University Office of Advanced Research Computing (OARC) cluster computer ''Amarel''.

===Amarel help ===
There is an OARC supported user guide at https://sites.google.com/view/cluster-user-guide to which you can refer for a lot of detailed guidance.

===Working with UNIX===
To work effectively on ''Amarel'' you need at least a beginner level of competency with the UNIX operating system.
If you are a total newcomer to UNIX computers you might find it useful to work through a structured tutorial; for example [https://ryanstutorials.net/linuxtutorial/ Ryan's Linux Tutorial].

{| class="wikitable"
|+ Basic UNIX commands
|-
! Command !! What it does !! Example
|-
| ls || list contents of a directory </br> files ending in .nc </br> long format list and hidden files || ls </br> ls *.nc </br> ls -al
|-
| cd || change directory || cd /projects/dmcs_1/courses/cod
|-
| pwd || print working directory to see where you are || pwd
|-
| cp || copy a file || cp source_file target_file
|-
| mv || rename a file </br> move file to new directory || mv file_name new_name </br> mv file_name dir1/dir2/new_name
|-
| rm || delete (remove) a file || rm filename
|-
| more || type contents of a file to screen one page at a time || more .bashrc
|-
| less || like ''more'' only with more features (unix geeks are so funny) </br>Read the man page for help || less .bashrc
|-
| man || get the manual page (help) for a command || man less
|-
| . || the directory you are in || cp /projects/dmcs_1/courses/job.sh '''.'''
|-
| .. || the directory above the one you are in || cd '''..'''
|-
| grep || find a string in a file </br> ''get regular expression and print (!)'' || grep COD_1DMIX *.h
|}

Your default UNIX ''shell'' (the "flavor" of your UNIX environment) is '''bash''', so for more advanced help it is often good to include "bash" as one of your search terms in a google search for help.

===Editing files===
One of the things that stumps newcomers to UNIX is making simple edits to files. Ryan's Linux Tutorial goes over the basics of the '''vi''' editor [https://ryanstutorials.net/linuxtutorial/vi.php here], but it can be a beast to get comfortable with.

A more intuitive editor for new users is '''nano''', the basics of which you can read about [https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/ here].

There is also a way to edit files in the web browser based '''OnDemand''' service that is described later in this guide.

==Rutgers VPN==
If you are not on campus, you will need to connect to the Rutgers VPN. To use Rutgers VPN, you must first enroll in Duo 2 Factor Authentication (2FA) (NetID+). Most users have probably already done this. If not, navigate to https://netid.rutgers.edu/setupTwoFactorAuthentication.htm and follow the instructions.

Once you are enrolled in NetID+ you will need to activate the VPN service, if you have not already. Navigate to https://soc.rutgers.edu/vpn/ and click the gray button titled '''Service Activation'''. If you are not already logged in you may be asked to login and/or approve your login with the Duo Mobile app. To activate the VPN service, click the checkbox next to '''Remote Access VPN, Cisco AnyConnect Access for Rutgers''' and click the '''Activate Services''' button.

You are now ready to download the Cisco VPN client and connect to the Rutgers VPN. Complete instructions can be found [https://soc.rutgers.edu/vpn/ here] by clicking the red button titled '''General VPN Instructions''' to download the PDF. The instructions are geared towards Windows so if you are using a Mac you might find [https://soc.rutgers.edu/vpn/apple/ this page] more helpful. In most cases, regardless of your operating system, pointing your browser to https://vpn.rutgers.edu/ and logging in with your NetID will lead you to downloading the correct VPN client.

Once installed, open the '''Cisco AnyConnect''' client and type '''vpn.rutgers.edu''' in the box and click '''Connect''':

[[Image:AnyConnect_Connect.png]]

In the next window, shown below, '''Username''' will be your NetID.

In the second line '''Password''' is the password for your NetID.

The third line is the '''Duo Action''' to complete the 2FA, which can be accomplished in one of 4 possible ways:

# Enter a 6 digit Duo Passcode. These are generated either by a Hard Token, by showing the passcode in the Duo Mobile App, or from a previous “SMS” request. Simply type in the 6 numbers and hit OK.
# Type the word “push”. This will send a push notification to the primary device you have enrolled with Duo through NetID+ with the option to Accept or Deny.
# Type the word “phone”. You will receive a phone call to the primary device you have enrolled with Duo through NetID+ with touch tone options to Accept or Deny.
# Type the word “sms”. You will receive a text message to the primary device you have enrolled with Duo through NetID+ containing passcodes you can use to login.

[[Image:AnyConnect_Credentials.png]]

Click '''OK''' and you should be connected to Rutgers VPN. You should have a small AnyConnect icon with a lock on it in the task tray (Windows [[Image:AnyConnect_win_connected.jpg]]) or in the menu bar (Mac OS [[Image:AnyConnect_mac_connected.png|25px]]).

==Connecting to Amarel with SSH==
In order to connect to ''Amarel'' and compile and run ROMS you will need an SSH/terminal client. If you already use an SSH/terminal application you are comfortable with then stick with that and adapt the instructions below accordingly. For these instructions to work you need to either be on campus or connected to the Rutgers VPN.

===Mac OS===
If you are on a Mac, open the '''Terminal''' app (found in the Applications -> Utilities folder) or '''iTerm''' app and type:
:<div class="box">ssh fakeuser@amarel.rutgers.edu</div>
replacing <span class="blue">fakeuser</span> with your NetID. When it asks if you want to continue connecting enter '''yes''', then enter your NetID password.

Having to type the above command and netid password every time you login is moderately tedious, but this username/password authentication can become annoying and time consuming when repeatedly using the '''scp''' command to copy files to ''Amarel''. Passwordless access and file transfer can be enabled using '''SSH keys''' by following the instructions below, or for more detail see the [https://sites.google.com/view/cluster-user-guide#h.jgwrkm9e9rwg OARC User Guide on this topic].

# If you are still connected to ''Amarel'', disconnect by entering '''exit'''
# At the prompt for your local machine, enter '''<span style="font-family: monospace;">nano ~/.ssh/config</span>''' and paste the block below at the end of the file:<div class="box">Host amarel<br />Hostname amarel.rutgers.edu<br />HostKeyAlias amarel<br />User NetID</div> replacing <span class="blue">NetID</span> with your own NetID. Save and exit (Ctrl+o Ctrl+x). This allows you to type '''<span style="font-family: monospace;">ssh amarel</span>''' to connect.
# Enter '''<span style="font-family: monospace;">ls ~/.ssh</span>'''. If there is a file listed called <span class="blue">id_rsa.pub</span> skip to 5
# Enter '''<span style="font-family: monospace;">ssh-keygen -t rsa</span>''', then hit return to accept the default location, enter a passphase twice (or leave blank for no passphrase)
# Copy the public portion of the key to ''Amarel'' by entering '''<span style="font-family: monospace;">scp ~/.ssh/id_rsa.pub amarel:.</span>'''
# SSH to ''Amarel'' ('''<span style="font-family: monospace;">ssh amarel</span>''') and check if you already have a .ssh folder:<div class="box">[rjdave@amarel1 ~]$ ls -l .ssh<br />total 33<br />-rw------- 1 rjdave rjdave 1069 Sep 8 2017 authorized_keys<br />-rw-r--r-- 1 rjdave rjdave 434 Jun 4 2021 config<br />-rw-r--r-- 1 rjdave rjdave 5992 Dec 2 13:47 known_hosts</div> You probably won't have all the files above but if you instead get the message '''<span style="font-family: monospace;"><div class="box">ls: cannot access .ssh: No such file or directory</span>'''</div>then enter the following command:</br>'''<span style="font-family: monospace;">mkdir .ssh && chmod 700 .ssh</span>'''. </br>These two commands (combined on one line) will create the .ssh folder and set the required permissions. Setting private user-only permissions ('''chmod 700 ...''') is important, because the system will not allow you to have your keys viewable by others users.
# Once you have a .ssh folder with the proper permissions, enter the following commands</br> '''<span style="font-family: monospace;">cat ~/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 640 ~/.ssh/authorized_keys</span>'''</br>then exit back to your local machine and execute '''<span style="font-family: monospace;">ssh amarel</span>''' again. It should no longer ask for your password.

===Windows===
For Windows, we recommend [https://mobaxterm.mobatek.net/download-home-edition.html MobaXterm] installer edition.
# Once installed open it, choose light or dark theme and click the '''Session''' button in the upper left.
# Choose '''SSH''' and enter '''amarel.rutgers.edu''' for '''Remote host''', check the '''Specify username''' box and enter your NetID in the box and click '''OK'''.
# You will be asked to type your NetID password and then asked if you want to save the password. If you choose yes you will be asked to set a master password to encrypt all your saved passwords.
# In the future you can click '''Sessions''' (not Session) and select '''amarel.rutgers.edu''' and not have to enter your password.

==Setting up your .bashrc==
The default software setup by OARC on ''Amarel'' does not include everything needed to compile and run ROMS. Ocean Modeling Group computing specialist David Robertson has configured, and maintains and updates, a repository of what you need for ROMS and other models that will be automatically loaded for every login session once you add some lines to the login script (.bashrc, .cshrc, etc.) in your home directory. Log in to ''Amarel'' with '''Terminal''', '''iTerm2''' or '''MobaXterm''' and edit your login script as described below.

Unless otherwise requested your default shell will be bash and the following lines shown below in <span class="red">red</span> and <span class="forestGreen">green</span> should be added (with nano or your preferred editor) near the top of your .bashrc after the import of the global definitions as shown:
:<div class="box"># Source global definitions<br />if [ -f /etc/bashrc ]; then<br /> . /etc/bashrc<br />fi<br /><br /><span class="red">ulimit -s unlimited<br />export MODULEPATH=/projects/dmcs_1/sw/modulefiles/Core:${MODULEPATH}<br />export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.10l %.4C %.6D %R"</span><br /><br /><span class="forestGreen"># Automatically load the roms module<br />module load roms</span></div>

The line with <span class="red">ulimit</span> allows you to take full advantage of computing resources (may be obsolete now but it won't hurt anything).

The first <span class="red">export</span> line will tell '''Lmod''' where to find the modules that will load the custom software.

The second <span class="red">export</span> line makes the '''squeue''' command (explained later) more useful.

The lines in <span class="forestGreen">green</span> are optional and will load the '''roms''' module automatically every time you login to ''Amarel''. They must be '''after''' the line that sets the '''MODULEPATH'''.

Once you have added these lines, log out and back in for them to take effect.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... then you may need to logout and login again so that .bashrc sets MODULEPATH as instructed above. If the error persists, you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Accessing the ROMS Source Code==

===Register as a ROMS user===
If you are not already a ROMS user, you will need to fill out the registration form found [https://www.myroms.org/register here] and wait for approval. Once approved, you will be able to checkout the ROMS source code. It is highly recommended that you use the GitHub myroms repository to obtain the ROMS source code. The ''Subversion (SVN)'' repository is being phased out and will no longer be available after '''December 31, 2024'''. If you manage your own software projects using GitHub, the concepts for software version control may already be familiar to you. Detailed directions can be found on the WikiROMS [[Git#Downloading_ROMS|Git]] page.

===Obtaining the ROMS source code===
There is nothing special about obtaining the ROMS source code on ''Amarel''. The git commands are the same as on any UNIX host.
Navigate to the directory where you want your ROMS source code to reside and execute this git clone command:
:<div class="box">git clone https://github.com/myroms/roms.git <my_src_dir></div>
replacing <my_src_dir> with what you want the source code directory to be called. You can also omit <my_src_dir> and git will automatically name the folder '''roms'''.

It can take a couple of minutes to download all of ROMS, but you only do this once.

Many users simply run this command from their home directory and use <span class="forestGreen">'''myroms'''</span> for <my_src_dir>, but the choice is yours depending on how you want to manage your ''Amarel'' space and how many ROMS projects you think you might end up working on.

==Loading and Unloading Modules==
Like many clusters, ''Amarel'' uses environment modules to load and unload software and configure the environment. Some commands you will find useful are:
:<div class="box">'''module help''' Display help message<br />'''module help <m1'''> Show help information for module <m1><br />'''module available''' Show all modules currently available<br />'''module whatis <m1>''' Show brief information about module <m1><br />'''module spider''' List all possible modules even if not currently available<br />'''module list''' List currently loaded modules<br />'''module load''' Load the specified module(s)<br />'''module unload''' Unload the specified module(s)<br />'''module swap <m1> <m2>''' Unload <m1> and then load <m2> (for switching versions of the same software)<br />'''module purge''' Unload all loaded modules</div>

===Loading the ROMS Module===
If you '''did not''' add the optional lines from the [[#Setting_up_your_.bashrc|Setting up your .bashrc]] section, you can easily set up your environment to compile and run ROMS by manually loading the '''roms''' module.
:<div class="box">module load roms<br />module list<br />Currently Loaded Modules:<br /> 1) intel/17.0.4 2) mvapich2/2.2 3) mct-roms/2.6.0 4) netcdf/4.6.2<br /> 5) esmf/8.0.0_nc4 6) parpack-roms/2.1 7) hdf5/1.10.4 8) roms/intel_nc4</div>
Notice that loading the '''roms''' module will actually load 7 other modules.

This will setup your environment to use the Intel compiler so remember to set '''FORT''' to '''ifort''' in your build script.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Your project/coursework working directory space==

We recommend you create a subdirectory of your home directory in which to work, so you can keep each ROMS project separate and organized.

You might call this '''working directory''' <span class="forestGreen">CoastalDynamics</span>, or <span class="forestGreen">Projects</span>, or simply <span class="forestGreen">cod</span> (for coastal ocean dynamics). You will be typing this a lot, so keep it simple.

To make a new directory called <span class="forestGreen">Projects</span>, enter this command

:<div class="box">mkdir Projects </div>

Change to this working directory with the command

:<div class="box">cd Projects </div>

In instructions that follow, when we refer to your '''working directory''' we mean this workspace.

==Configuring and Compiling ROMS==
You will compile and run ROMS in your project/coursework '''working directory''', so change to that directory first.

:<div class="box">cd <span class="forestGreen">Projects</span> </div>

Running the build script <span class="blue">build_roms.sh</span> is the recommended method for compiling ROMS on ''Amarel''. For any given project you will likely only need to change <span class="blue">build_roms.sh</span> to reset <span class="sandyBrown">ROMS_APPLICATION</span> and <span class="sandyBrown">MY_ROMS_SRC</span> and make sure <span class="sandyBrown">FORT</span> is set to '''ifort'''.

{{warning}} <span class="red">It is important to make sure '''USE_MY_LIBS''' is set to ''no'' or your compilation will fail.</span>

A template <span class="blue">build_roms.sh</span> script is in subdirectory '''ROMS/Bin''' of '''<my_src_dir>''' that you created when you downloaded the code. For example, if you created <span class="forestGreen">myroms</span> in your home directory the copy command would be

:<div class="box">cp ~/myroms/ROMS/Bin/build_roms.sh . </div>

Now open <span class="blue">build_roms.sh</span> with an editor (''e.g.'' '''nano''') and modify the line <span class="sandyBrown">MY_ROMS_SRC</span> to point to your choice for <my_src_dir>. So, you will have something like ...

:<div class="box"><span class="sandyBrown">export MY_ROMS_SRC=</span>{$HOME}/<span class="forestGreen">myroms</span> </div>

if the name you chose for <my_src_dir> was <span class="forestGreen">myroms</span>.

If this is your first time working with ROMS, a good starting place is to compile the default UPWELLING test case that is indicated in build_roms.sh with the default setting <span class="sandyBrown">export ROMS_APPLICATION</span>=UPWELLING.

To compile the UPWELLING example, you need to copy the configuration file <span class="blue">upwelling.h</span> to your working directory. This is in subdirectory '''ROMS/Include''' of your source code. For example, the command might be:

:<div class="box">cp ~/myroms/ROMS/Include/upwelling.h .</div>

Once your build script is configured and you have upwelling.h in your working directory you can compile ROMS by typing:

:<div class="box">./build_roms.sh -j 4</div>
where the number after -j indicates the number of compute cores to use in parallel to execute the compilation. The greater the number, the faster it goes.

However, the login node you will be compiling on is shared for the entire ''Amarel'' system. If you use a number larger than '''4''', or omit it altogether (which says use ''all'' cores on the login node) your build might be terminated by an administrator. Be a considerate user and keep the number low.

If compilation is processing successfully you will see a lot of information typed to your screen as each component of the source code is compiled. It will take a minute or so to complete.

If compilation ends quickly with errors like this:

:<div class="box">$ ./build_roms.sh -j 4 </br>rm -f -r core *.ipo /home/jwilkin/cod/Projects/Build_roms...</br>which: <span class="red">no mpif90</span> in (/usr/lpp/mmfs/bin:/usr/local/bin:/us...</br>cp -f /opt/intelsoft/serial/netcdf3/include/netcdf.mod /h...</br>cp: cannot stat ‘/opt/intelsofo... No such file or directory</br>make: *** No rule to make target netcdf.mod needed by ... Stop. </div>

... this means you have not run the <span class="blue">module load roms</span> command. Go back to the notes on [[Amarel#Loading_the_ROMS_Module]].

If compilation was successful, there will be a file named '''romsM''' that is the model executable. You can check for the file with this command:

:<div class="box">ls -l *</div>

You should see:

:<div class="box"><span class="blue">Build_roms</span> <span class="forestGreen">build_roms.sh romsM</span> upwelling.h </div>

<span class="blue">Build_roms</span> is a folder that build_roms.sh created to hold all the temporary files that the FORTRAN compilation process makes to combine into the romsM executable file.

==Running on the Amarel Compute Nodes==
When you '''ssh''' to ''Amarel'' you are connected to one of the login nodes. These nodes are to be used for file editing, transfering output and input files to/from your local computer, and modest code compiling tasks and analysis, but not for compute intensive tasks. Here we explain how to connect to the compute nodes for these larger tasks.

Note: Consult the Amarel status page (https://oarc.rutgers.edu/amarel-system-status/) before scheduling a job. Amarel is down for maintenance monthly.

===What ''partitions'' can you run on?===

Access to Amarel resources is controlled by ''partitions''. You can find out the partitions you have permission to access with the command

:<div class="box">sinfo -s</div>

This will show the partitions that everyone has access to. See the cluster guide [https://sites.google.com/view/cluster-user-guide#h.oeejy9yf80e4 entry on partitions] for an explanation of these.

If you belong to a research group that own a partition you should also see that listed, for example '''p_omg_1''' for the Ocean Modeling Group. This is the partition you want to use most of the time as your job will not get pre-empted by other owner users.

===Using the SLURM batch job scheduler===

''Amarel'' uses SLURM workload manager to schedule compute intensive tasks. The user configures a SLURM job script for each model run and submits this script with the '''sbatch''' command. The job script declares the resources required, such as number of CPUs for parallel jobs, maximum memory required, etc.

We have configured a simple template job script (for the ROMS UPWELLING example) that you can copy from '''/projects/dmcs_1/courses/job.sh''' into your working directory.

The UNIX command to copy a file is '''cp source_file target_file'''. So, you would enter

:<div class="box">cp /projects/dmcs_1/courses/job.sh .</div>

The syntax here uses a ''dot'' ('''.''') for the target_file, which causes the source_file (job.sh) to be copied to the present working directory without changing its name.

When typing this, or any, UNIX command, you can start typing just the beginning few characters of the path then press TAB to autocomplete the filename. If the completion is ambiguous, you can continue typing more characters and press TAB again. Pressing TAB twice in a row will show you the set of possible completions. This trick can save you a lot of tedious (mis)typing.

The contents of '''job.sh''' are shown below:

:<div class="box">#!/bin/bash<br >#SBATCH --partition=p_omg_1 # Partition. If you are not a member of OMG enter your own partition<br />#SBATCH --job-name=upwelling # Assign a short name to your job<br/>#SBATCH --nodes=1 # Number of nodes you require<br />#SBATCH --ntasks=4 # Total number of tasks you'll launch<br />#SBATCH --ntasks-per-node=4 # Number of tasks you'll launch on each node<br />#SBATCH --cpus-per-task=1 # Cores per task (>1 if multithread tasks)<br />#SBATCH --mem=6400 # Real memory (RAM) required (MB) per node<br />#SBATCH --time=00-00:05 # Total run time limit (DD-HH:MM)<br />#SBATCH --output=out.%N.%j # STDOUT output file<br />#SBATCH --error=err.%N.%j # STDERR output file (optional but recommended)<br />#SBATCH --export=ALL # Export you current env to the job env<br /><br />## It is important to have --mpi=pmi2 here or ROMS will not run<br />srun --mpi=pmi2 ./romsM roms_upwelling.in</div>

Note: The maximum run time is 14-00:00 (14 days). It is tempting to set this to the maximum in order to avoid a job ending too early. (''i.e.'' Slurm stopping a ROMS run before it is finished) However, if there is maintenance scheduled during that time period the job will not start until maintenance is complete. So a runtime as close as possible to the actual run time is recommended.

====Requesting a specific class of compute node====

For large jobs it makes sense to use all available cores on a node, and for low priority jobs to use older less fast nodes. You can do this by requesting a specific node number, but it is better to request a particular class of nodes so you don't unwittingly pick a node already in use. The classes available from oldest to newest nodes are:

# skylake (32 cores) (Phase I nodes) (nodes hal0116 through hal0120)
# cascadelake (40 cores) (Phase II nodes) (nodes hal0159 through hal0163)
# icelake (64 cores) (Phase V.B nodes) (nodes hal0299 through hal0302)

Make the node class request in the SLURM script and choose the maximum number of cores to match, ''e.g.''

:<div class="box">#SBATCH --constraint=cascadelake # Run on Phase II nodes (40 cores) <br>#SBATCH --ntasks=40 # Total number of tasks to launch (like -np to mpirun) <br>#SBATCH --ntasks-per-node=40 # Number of tasks to launch on each node
</div>

If you don't specifically request a class of nodes the SLURM scheduler will make its own choice on the most appropriate resources to use given your request. We don't know the basis of that choice.

If you request a class of node that is fully assigned to other users in your partition, your job will wait for the others users jobs to complete. So, coordinate with your colleagues if resource use is at a premium. You can use the '''squeue -p partition_name''' command to see what resources are being used by other users, ''e.g.''

:<div class="box">$ squeue -p p_omg_1 <br> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON) <br> 26975473 p_omg_1 fwd levinj R 17:15:42 14-00:00:00 80 2 hal[0159,0162]
</div>

===Running the ROMS Upwelling Example===
You may have noticed that the srun command above includes a file named <span class="blue">roms_upwelling.in</span>. You will need to copy this file and <span class="blue">varinfo.dat</span> from the ROMS/External directory of your <my_src_dir> ROMS source code to the directory where you compiled ROMS.

If, for example, you chose '''myroms''' as the name of your <my_src_dir> when you downloaded the code, you would enter

:<div class="box">cp ~/myroms/ROMS/External/roms_upwelling.in .</div>

The syntax here uses ''tilde'' (~) as shorthand for your home directory,

After you copy the files, you will need to make a couple of small edits (using ''e.g.'' '''nano''') to <span class="blue">roms_upwelling.in</span> to get this to work.

# Change the line with VARNAME to read <span class="sandyBrown">VARNAME = varinfo.dat</span> (''i.e.'' delete the ROMS/External part)
# Set <span class="sandyBrown">NtileI == 2</span> and <span class="sandyBrown">NtileJ == 2</span>.

The product of NtileI and NtileJ, ''i.e.'' 2 x 2 = 4, is the number of cores the model will run on in parallel. This number must match the number in the SLURM job script options '''--ntasks''' and, in this case, '''--ntasks-per-node'''

Now you can submit your job to '''sbatch''' with the command '''<span style="font-family: monospace;">sbatch job.sh</span>'''

===ROMS configurations for ''Coastal Ocean Dynamics'' class===
The files that will configure the examples we will study in class will be located in <span class="blue">/projects/dmcs_1/courses/cod</span>.

You will find instructions for configuring and running the examples at a separate Wiki page [[COD_Class_Examples]]

===Checking Job Status with squeue===
Detailed documentation for monitoring your SLURM jobs can be found [https://sites.google.com/view/cluster-user-guide#h.4bsndqufii8p here]. The easiest way to check whether your job is running is with the command '''<span style="font-family: monospace;">squeue -p p_omg_1</span>''' (or your own partition name if it is not p_omg_1).

You should see output something like this:

:<div class="box"> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON)<br /> 17197219 dmcs_1 watl_psas rjdave R 3-04:07:00 3-20:00:00 4 1 hal0035<br /> 17146609 dmcs_1 upwelling rjdave R 00:01:10 00:05:00 4 1 hal0035</div>

You can see that the job is running (the ‘R’ under ‘ST’), has been running for 1 minute, 10 seconds, and is running on node 35.

===Checking Progress with tail===
You can check ROMS progress by using the tail command on the '''output''' file. For the upwelling job above the file would be called <span class="blue">out.hal0035.17146609</span> so the command '''<span style="font-family: monospace;">tail out.hal0035.17146609</span>''' will show you the most recent 10 lines written to the output log and will most likely tell you what time-step the model is on. Using the -f option ('''<span style="font-family: monospace;">tail -f out.hal0035.17146609</span>''') will output appended data as the file grows. Ctrl-C will escape the display.

===Did my UPWELLING job run?===
If the UPWELLING test case was successful there should be an output log file named something like <span class="blue">out.hal0035.17146609</span> that shows no errors, and there should be 4 output NetCDF files ...
:<div class="box">ls -l *.nc </br>roms_avg.nc </br>roms_dia.nc </br>roms_his.nc </br>roms_rst.nc </div>

===Cancelling a Job===
To safest way cancel a queued or running job is to use its jobid. A job can be canceled by name but that is not recommended. To cancel the upwelling job in the example above you would issue the command '''<span style="font-family: monospace;">scancel 17146609</span>'''. If the job has not yet started, it will be removed from the queue. If it is running, all child processes will be killed and the job will be removed from the queue. You are only able cancel jobs that you own.

===Running an interactive session on a compute node===
It is possible to conduct your work interactively on one of the compute nodes (instead of the login node). For most work we will be doing in class this is not necessary, but if for some reason you have a job that needs many processors or a large amount of memory, and you want to run it interactively - say, to simply check that everything is in order for it to start correctly - there are instructions in the Cluster User Guide [https://sites.google.com/view/cluster-user-guide#h.26x9sbburvsg here].

For compute intensive interactive work, such as model analysis using Python or MATLAB, we recommend using the OnDemand interface to launch an interactive session on a set of compute nodes.

==Using the OnDemand desktop environment==
The '''OnDemand''' service allows you to connect to ''Amarel'' using a graphical desktop environment in a web browser. One of the most useful features of '''OnDemand''' is that it allows you to analyze model output by Python or Matlab on the cluster compute nodes.

You need to be running a Rutgers VPN to use OnDemand.

Navigate your browser to https://ondemand.hpc.rutgers.edu/pun/sys/dashboard and login with your NetID and password. At the top of the page click '''Interactive Apps'''.

===Python Jupyter notebook:===
# Select '''Personal Jupyter''' from the list on the left
# Make your selections for '''number of hours''' your session can remain active (''e.g.'' 3 hours), the '''number of cores''' to run on (1), and '''memory''' (8 GB is an OK choice)
# For ''Partition'' enter ''e.g.'' <span class="blue">p_omg_1</span> if you are in OMG. Leave ''Reservation'' and ''Slurm feature'' blank. For ''conda path'' enter <span class="blue">/projects/dmcs_1/miniconda3</span> and for ''conda environment'' enter <span class="blue">/projects/dmcs_1/sw/packages/xroms/py38</span>. Click [[Media:Amarel Jupyter config.png | here]] to see what the completed launch page looks like.
# Click '''Launch''' and wait. This can take a couple minutes.
# You should see sub-window of the web page like this </br> [[Image:JupyterStarting.png|600px]] </br></br>When the session is ready it changes to this: </br> [[Image:JupyterRunning.png|600px]]
# Start your notebook by clicking the '''Connect to Jupyter, Anaconda version 5.1.0''' button: (If this does not appear, check that you correctly entered the '''conda path''' and '''conda environment''' at step 3 above.) </br>[[Image:JupyterConnect.png|400px]]
# This should place you in a file browse window from which you can navigate to one of your existing Jupyter notebooks, or, to start a new Untitled notebook, select ''Python 3 (ipykernel)'' from the ''New'' pull-down menu on the right side of the page [[Image:NewPythonNotebook.png]]
# Once your notebook is running, you can import the xroms package as you would any other toolbox, by entering '''<span style="font-family: monospace;">import xroms</span>''' then hold down the shift key and hit the return/enter key. You will see an asterisk ['''*'''] in the square brackets.
# Once that asterisk changes to a '''1''', the xroms python module has been loaded.

This '''conda''' environment built for DMCS_1 makes available many of the Python modules you are used to using.

===Matlab:===
# Select MATLAB from the list on the left
# As for Step 2 above for Python, choose time, cores, and memory
# Select MATLAB version, and enter <span class="blue">p_omg_1</span> (or your own group name) in partition.
# Click '''Launch''' and wait. This can take a couple minutes.
# Once the '''Launch noVNC in New Tab''' button appears, click it and a MatLab GUI will open.

You are running in a graphical desktop environment, so by default Matlab will look in Documents/MATLAB/startup.m for your personal Matlab configuration file.

Popular MATLAB utilities for working with ROMS output are in <span class="blue">/projects/dmcs_1/sw/packages/matlab-tools</span>. These include the myroms.org utilities in subdirectory <span class="blue">myroms</span>, and John Wilkin's toolbox in <span class="blue">roms_wilkin</span>.

===Reconnecting to an interactive session===
A particularly nifty feature of '''OnDemand''' is that your interactive session is independent of the browser window instance.

Say you are working on your office computer using a Jupyter notebook via '''OnDemand''' but haven't finished all you want to do by the end of the day. If you leave that session running, you can go home and connect once again to '''OnDemand''' and you will see your session in the list of Interactive Sessions. Launch that session and you will be placed right back where you were. '''OnDemand''' is simply a web interface to code running on ''Amarel'', so you can connect from any browser.

Of course, this only works if you selected a '''number of hours''' in Step 2 above that will allow your session to remain open.

As long as the session is active the ''Amarel'' resources (cores and memory) are reserved for you and no one else. So, be respectful of other users and close out an interactive session once you are done.

Amarel

2024-01-19T17:37:53Z

Robertson:

<div class="title">Using Amarel</div>
==Getting Started==
This wiki page is a brief "Getting Started" introduction to running the ROMS ocean model and analyzing model output on the Rutgers University Office of Advanced Research Computing (OARC) cluster computer ''Amarel''.

===Amarel help ===
There is an OARC supported user guide at https://sites.google.com/view/cluster-user-guide to which you can refer for a lot of detailed guidance.

===Working with UNIX===
To work effectively on ''Amarel'' you need at least a beginner level of competency with the UNIX operating system.
If you are a total newcomer to UNIX computers you might find it useful to work through a structured tutorial; for example [https://ryanstutorials.net/linuxtutorial/ Ryan's Linux Tutorial].

{| class="wikitable"
|+ Basic UNIX commands
|-
! Command !! What it does !! Example
|-
| ls || list contents of a directory </br> files ending in .nc </br> long format list and hidden files || ls </br> ls *.nc </br> ls -al
|-
| cd || change directory || cd /projects/dmcs_1/courses/cod
|-
| pwd || print working directory to see where you are || pwd
|-
| cp || copy a file || cp source_file target_file
|-
| mv || rename a file </br> move file to new directory || mv file_name new_name </br> mv file_name dir1/dir2/new_name
|-
| rm || delete (remove) a file || rm filename
|-
| more || type contents of a file to screen one page at a time || more .bashrc
|-
| less || like ''more'' only with more features (unix geeks are so funny) </br>Read the man page for help || less .bashrc
|-
| man || get the manual page (help) for a command || man less
|-
| . || the directory you are in || cp /projects/dmcs_1/courses/job.sh '''.'''
|-
| .. || the directory above the one you are in || cd '''..'''
|-
| grep || find a string in a file </br> ''get regular expression and print (!)'' || grep COD_1DMIX *.h
|}

Your default UNIX ''shell'' (the "flavor" of your UNIX environment) is '''bash''', so for more advanced help it is often good to include "bash" as one of your search terms in a google search for help.

===Editing files===
One of the things that stumps newcomers to UNIX is making simple edits to files. Ryan's Linux Tutorial goes over the basics of the '''vi''' editor [https://ryanstutorials.net/linuxtutorial/vi.php here], but it can be a beast to get comfortable with.

A more intuitive editor for new users is '''nano''', the basics of which you can read about [https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/ here].

There is also a way to edit files in the web browser based '''OnDemand''' service that is described later in this guide.

==Rutgers VPN==
If you are not on campus, you will need to connect to the Rutgers VPN. To use Rutgers VPN, you must first enroll in Duo 2 Factor Authentication (2FA) (NetID+). Most users have probably already done this. If not, navigate to https://netid.rutgers.edu/setupTwoFactorAuthentication.htm and follow the instructions.

Once you are enrolled in NetID+ you will need to activate the VPN service, if you have not already. Navigate to https://soc.rutgers.edu/vpn/ and click the gray button titled '''Service Activation'''. If you are not already logged in you may be asked to login and/or approve your login with the Duo Mobile app. To activate the VPN service, click the checkbox next to '''Remote Access VPN, Cisco AnyConnect Access for Rutgers''' and click the '''Activate Services''' button.

You are now ready to download the Cisco VPN client and connect to the Rutgers VPN. Complete instructions can be found [https://soc.rutgers.edu/vpn/ here] by clicking the red button titled '''General VPN Instructions''' to download the PDF. The instructions are geared towards Windows so if you are using a Mac you might find [https://soc.rutgers.edu/vpn/apple/ this page] more helpful. In most cases, regardless of your operating system, pointing your browser to https://vpn.rutgers.edu/ and logging in with your NetID will lead you to downloading the correct VPN client.

Once installed, open the '''Cisco AnyConnect''' client and type '''vpn.rutgers.edu''' in the box and click '''Connect''':

[[Image:AnyConnect_Connect.png]]

In the next window, shown below, '''Username''' will be your NetID.

In the second line '''Password''' is the password for your NetID.

The third line is the '''Duo Action''' to complete the 2FA, which can be accomplished in one of 4 possible ways:

# Enter a 6 digit Duo Passcode. These are generated either by a Hard Token, by showing the passcode in the Duo Mobile App, or from a previous “SMS” request. Simply type in the 6 numbers and hit OK.
# Type the word “push”. This will send a push notification to the primary device you have enrolled with Duo through NetID+ with the option to Accept or Deny.
# Type the word “phone”. You will receive a phone call to the primary device you have enrolled with Duo through NetID+ with touch tone options to Accept or Deny.
# Type the word “sms”. You will receive a text message to the primary device you have enrolled with Duo through NetID+ containing passcodes you can use to login.

[[Image:AnyConnect_Credentials.png]]

Click '''OK''' and you should be connected to Rutgers VPN. You should have a small AnyConnect icon with a lock on it in the task tray (Windows [[Image:AnyConnect_win_connected.jpg]]) or in the menu bar (Mac OS [[Image:AnyConnect_mac_connected.png|25px]]).

==Connecting to Amarel with SSH==
In order to connect to ''Amarel'' and compile and run ROMS you will need an SSH/terminal client. If you already use an SSH/terminal application you are comfortable with then stick with that and adapt the instructions below accordingly. For these instructions to work you need to either be on campus or connected to the Rutgers VPN.

===Mac OS===
If you are on a Mac, open the '''Terminal''' app (found in the Applications -> Utilities folder) or '''iTerm''' app and type:
:<div class="box">ssh fakeuser@amarel.rutgers.edu</div>
replacing <span class="blue">fakeuser</span> with your NetID. When it asks if you want to continue connecting enter '''yes''', then enter your NetID password.

Having to type the above command and netid password every time you login is moderately tedious, but this username/password authentication can become annoying and time consuming when repeatedly using the '''scp''' command to copy files to ''Amarel''. Passwordless access and file transfer can be enabled using '''SSH keys''' by following the instructions below, or for more detail see the [https://sites.google.com/view/cluster-user-guide#h.jgwrkm9e9rwg OARC User Guide on this topic].

# If you are still connected to ''Amarel'', disconnect by entering '''exit'''
# At the prompt for your local machine, enter '''<span style="font-family: monospace;">nano ~/.ssh/config</span>''' and paste the block below at the end of the file:<div class="box">Host amarel<br />Hostname amarel.rutgers.edu<br />HostKeyAlias amarel<br />User NetID</div> replacing <span class="blue">NetID</span> with your own NetID. Save and exit (Ctrl+o Ctrl+x). This allows you to type '''<span style="font-family: monospace;">ssh amarel</span>''' to connect.
# Enter '''<span style="font-family: monospace;">ls ~/.ssh</span>'''. If there is a file listed called <span class="blue">id_rsa.pub</span> skip to 5
# Enter '''<span style="font-family: monospace;">ssh-keygen -t rsa</span>''', then hit return to accept the default location, enter a passphase twice (or leave blank for no passphrase)
# Copy the public portion of the key to ''Amarel'' by entering '''<span style="font-family: monospace;">scp ~/.ssh/id_rsa.pub amarel:.</span>'''
# SSH to ''Amarel'' ('''<span style="font-family: monospace;">ssh amarel</span>''') and check if you already have a .ssh folder:<div class="box">[rjdave@amarel1 ~]$ ls -l .ssh<br />total 33<br />-rw------- 1 rjdave rjdave 1069 Sep 8 2017 authorized_keys<br />-rw-r--r-- 1 rjdave rjdave 434 Jun 4 2021 config<br />-rw-r--r-- 1 rjdave rjdave 5992 Dec 2 13:47 known_hosts</div> You probably won't have all the files above but if you instead get the message '''<span style="font-family: monospace;"><div class="box">ls: cannot access .ssh: No such file or directory</span>'''</div>the enter the following command:</br>'''<span style="font-family: monospace;">mkdir .ssh && chmod 700 .ssh</span>'''. </br>These two commands (combined on one line) will create the .ssh folder and set the required permissions. Setting private user-only permissions ('''chmod 700 ...''') is important, because the system will not allow you to have your keys viewable by others users.
# Once you have a .ssh folder with the proper permissions, enter the following commands</br> '''<span style="font-family: monospace;">cat ~/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 640 ~/.ssh/authorized_keys</span>'''</br>then exit back to your local machine and execute '''<span style="font-family: monospace;">ssh amarel</span>''' again. It should no longer ask for your password.

===Windows===
For Windows, we recommend [https://mobaxterm.mobatek.net/download-home-edition.html MobaXterm] installer edition.
# Once installed open it, choose light or dark theme and click the '''Session''' button in the upper left.
# Choose '''SSH''' and enter '''amarel.rutgers.edu''' for '''Remote host''', check the '''Specify username''' box and enter your NetID in the box and click '''OK'''.
# You will be asked to type your NetID password and then asked if you want to save the password. If you choose yes you will be asked to set a master password to encrypt all your saved passwords.
# In the future you can click '''Sessions''' (not Session) and select '''amarel.rutgers.edu''' and not have to enter your password.

==Setting up your .bashrc==
The default software setup by OARC on ''Amarel'' does not include everything needed to compile and run ROMS. Ocean Modeling Group computing specialist David Robertson has configured, and maintains and updates, a repository of what you need for ROMS and other models that will be automatically loaded for every login session once you add a some lines to the login script (.bashrc, .cshrc, etc.) in your home directory. Log in to ''Amarel'' with '''Terminal''', '''iTerm2''' or '''MobaXterm''' and edit your login script as described below.

Unless otherwise requested your default shell will be bash and the following lines shown below in <span class="red">red</span> and green<span class="forestGreen">Colored text here</span> should be added (with nano or your preferred editor) near the top of your .bashrc after the import of the global definitions as shown:
:<div class="box"># Source global definitions<br />if [ -f /etc/bashrc ]; then<br /> . /etc/bashrc<br />fi<br /><br /><span class="red">ulimit -s unlimited<br />export MODULEPATH=/projects/dmcs_1/sw/modulefiles/Core:${MODULEPATH}<br />export SQUEUE_FORMAT="%.18i %.9P %.8j %.8u %.2t %.10M %.10l %.4C %.6D %R"</span><br /><br /><span class="forestGreen"># Automatically load the roms module<br />module load roms</span></div>

The line with <span class="red">ulimit</span> allows you to take full advantage of computing resources (may be obsolete now but it won't hurt anything).

The first <span class="red">export</span> line will tell '''Lmod''' where to find the modules that will load the custom software.

The second <span class="red">export</span> line makes the '''squeue''' command (explained later) more useful.

The lines in <span class="forestGreen">green</span> are optional and will load the '''roms''' module automatically every time you login to ''Amarel''. They must be '''after''' the line that sets the '''MODULEPATH'''.

Once you have added these lines, log out and back in for them to take effect.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... then you may need to logout and login again so that .bashrc sets MODULEPATH as instructed above. If the error persists, you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Accessing the ROMS Source Code==

===Register as a ROMS user===
If you are not already a ROMS user, you will need to fill out the registration form found [https://www.myroms.org/register here] and wait for approval. Once approved, you will be able to checkout the ROMS source code. It is highly recommended that you use the GitHub myroms repository to obtain the ROMS source code. The ''Subversion (SVN)'' repository is being phased out and will no longer be available after '''December 31, 2024'''. If you manage your own software projects using GitHub, the concepts for software version control may already be familiar to you. Detailed directions can be found on the WikiROMS [[Git#Downloading_ROMS|Git]] page.

===Checking out the ROMS source code===
There is nothing special about checking out the ROMS source code on ''Amarel''. The git commands are the same as on any UNIX host.
Navigate to the directory where you want your ROMS source code to reside and execute this git checkout command:
:<div class="box">git clone https://github.com/myroms/roms.git <my_src_dir></div>
replacing <my_src_dir> with what you want the source code directory to be called.

It will take a couple of minutes to download all of ROMS, but you only do this once. You will see the list of all the files being downloaded typed to your screen.

Many users simply run this command from their home directory and use <span class="forestGreen">'''myroms'''</span> for <my_src_dir>, but the choice is yours depending on how you want to manage your ''Amarel'' space and how many ROMS projects you think you might end up working on.

==Loading and Unloading Modules==
Like many clusters, ''Amarel'' uses environment modules to load and unload software and configure the environment. Some commands you will find useful are:
:<div class="box">'''module help''' Display help message<br />'''module help <m1'''> Show help information for module <m1><br />'''module available''' Show all modules currently available<br />'''module whatis <m1>''' Show brief information about module <m1><br />'''module spider''' List all possible modules even if not currently available<br />'''module list''' List currently loaded modules<br />'''module load''' Load the specified module(s)<br />'''module unload''' Unload the specified module(s)<br />'''module swap <m1> <m2>''' Unload <m1> and then load <m2> (for switching versions of the same software)<br />'''module purge''' Unload all loaded modules</div>

===Loading the ROMS Module===
If you '''did not''' add the optional lines from the [[#Setting_up_your_.bashrc|Setting up your .bashrc]] section, you can easily set up your environment to compile and run ROMS by manually loading the '''roms''' module.
:<div class="box">module load roms<br />module list<br />Currently Loaded Modules:<br /> 1) intel/17.0.4 2) mvapich2/2.2 3) mct-roms/2.6.0 4) netcdf/4.6.2<br /> 5) esmf/8.0.0_nc4 6) parpack-roms/2.1 7) hdf5/1.10.4 8) roms/intel_nc4</div>
Notice that loading the '''roms''' module will actually load 7 other modules.

This will setup your environment to use the Intel compiler so remember to set '''FORT''' to '''ifort''' in your build script.

{{note}} '''Note:''' If you see the error ...
:<div class="box"><span class="red">Lmod has detected the following error:</span> The following module(s) are unknown: "roms"<br /><br />Please check the spelling or version number. Also try "module spider ..."<br />It is also possible your cache file is out-of-date; it may help to try:<br /> $ module --ignore_cache load "roms"<br /><br />Also make sure that all modulefiles written in TCL start with the string #%Module</div>
... you may not have been properly added to the '''dmcs_1''' Linux group. To check if you are part of '''dmcs_1''' type the command '''<span style="font-family: monospace;">id</span>''' and if '''dmcs_1''' is not listed, let us know and we will get you added to the group.

==Your project/coursework working directory space==

We recommend you create a subdirectory of your home directory in which to work, so you can keep each ROMS project separate and organized.

You might call this '''working directory''' <span class="forestGreen">CoastalDynamics</span>, or <span class="forestGreen">Projects</span>, or simply <span class="forestGreen">cod</span> (for coastal ocean dynamics). You will be typing this a lot, so keep it simple.

To make a new directory called <span class="forestGreen">Projects</span>, enter this command

:<div class="box">mkdir Projects </div>

Change to this working directory with the command

:<div class="box">cd Projects </div>

In instructions that follow, when we refer to your '''working directory''' we mean this workspace.

==Configuring and Compiling ROMS==
You will compile and run ROMS in your project/coursework '''working directory''', so change to that directory first.

:<div class="box">cd <span class="forestGreen">Projects</span> </div>

Running the build script <span class="blue">build_roms.sh</span> is the recommended method for compiling ROMS on ''Amarel''. For any given project you will likely only need to change <span class="blue">build_roms.sh</span> to reset <span class="sandyBrown">ROMS_APPLICATION</span> and <span class="sandyBrown">MY_ROMS_SRC</span> and make sure <span class="sandyBrown">FORT</span> is set to '''ifort'''.

{{warning}} <span class="red">It is important to make sure '''USE_MY_LIBS''' is set to ''no'' or your compilation will fail.</span>

A template <span class="blue">build_roms.sh</span> script is in subdirectory '''ROMS/Bin''' of '''<my_src_dir>''' that you created when you downloaded the code. For example, if you created <span class="forestGreen">myroms</span> in your home directory the copy command would be

:<div class="box">cp ~/myroms/ROMS/Bin/build_roms.sh . </div>

Now open <span class="blue">build_roms.sh</span> with an editor (e.g. '''nano''') and modify the line <span class="sandyBrown">MY_ROMS_SRC</span> to point to your choice for <my_src_dir>. So, you will have something like ...

:<div class="box"><span class="sandyBrown">export MY_ROMS_SRC=</span>{$HOME}/<span class="forestGreen">myroms</span> </div>

if the name you chose for <my_src_dir> was <span class="forestGreen">myroms</span>.

If this is your first time working with ROMS, a good starting place is to compile the default UPWELLING test case that is indicated in build_roms.sh with the default setting <span class="sandyBrown">export ROMS_APPLICATION</span>=UPWELLING.

To compile the UPWELLING example, you need to copy the configuration file <span class="blue">upwelling.h</span> to your working directory. This is in subdirectory '''ROMS/Include''' of your source code. For example, the command might be:

:<div class="box">cp ~/myroms/ROMS/Include/upwelling.h .</div>

Once your build script is configured and you have upwelling.h in your working directory you can compile ROMS by typing:

:<div class="box">./build_roms.sh -j 4</div>
where the number after -j indicates the number of compute cores to use in parallel to execute the compilation. The greater the number, the faster it goes.

However, the login node you will be compiling on is shared for the entire ''Amarel'' system. If you use a number larger than '''4''', or omit it altogether (which says use ''all'' cores on the login node) your build might be terminated by an administrator. Be a considerate user and keep the number low.

If compilation is processing successfully you will see a lot of information typed t your screen as each component of the source code is compiled. It will take a minute or so to complete.

If compilation ends quickly with errors like this:

:<div class="box">$ ./build_roms.sh -j 4 </br>rm -f -r core *.ipo /home/jwilkin/cod/Projects/Build_roms...</br>which: <span class="red">no mpif90</span> in (/usr/lpp/mmfs/bin:/usr/local/bin:/us...</br>cp -f /opt/intelsoft/serial/netcdf3/include/netcdf.mod /h...</br>cp: cannot stat ‘/opt/intelsofo... No such file or directory</br>make: *** No rule to make target netcdf.mod needed by ... Stop. </div>

... this means you have not run the <span class="blue">module load roms</span> command. Go back to the notes on [[Amarel#Loading_the_ROMS_Module]].

If compilation was successful, there will be a file named '''romsM''' that is the model executable. You can check for the file with this command:

:<div class="box">ls -l *</div>

You should see:

:<div class="box"><span class="blue">Build_roms</span> <span class="forestGreen">build_roms.sh romsM</span> upwelling.h </div>

<span class="blue">Build_roms</span> is a folder that build_roms.sh created to hold all the temporary files that the FORTRAN compilation process makes to combine into the romsM executable file

==Running on the Amarel Compute Nodes==
When you '''ssh''' to ''Amarel'' you are connected to one of the login nodes. These nodes are to be used for file editing, transfering output and input files to/from your local computer, and modest code compiling tasks and analysis, but not for compute intensive tasks. Here we explain how to connect to the compute nodes for these larger tasks.

Note: Consult the Amarel status page (https://oarc.rutgers.edu/amarel-system-status/) before scheduling a job. Amarel is down for maintenance monthly.

===What ''partitions'' can you run on?===

Access to Amarel resources is controlled by ''partitions''. You can find out the partitions you have permission to access with the command

:<div class="box">sinfo -s</div>

This will show the partitions that everyone has access to. See the cluster guide [https://sites.google.com/view/cluster-user-guide#h.oeejy9yf80e4 entry on partitions] for an explanation of these.

If you belong to a research group that own a partition you should also see that listed, for example '''p_omg_1''' for the Ocean Modeling Group. This is the partition you want to use most of the time as your job will not get pre-empted by other owner users.

===Using the SLURM batch job scheduler===

''Amarel'' uses SLURM workload manager to schedule compute intensive tasks. The user configures a SLURM job script for each model run and submits this script with the '''sbatch''' command. The job script declares the resources required, such as number of CPUs for parallel jobs, maximum memory required, etc.

We have configured a simple template job script (for the ROMS UPWELLING example) that you can copy from '''/projects/dmcs_1/courses/job.sh''' into your working directory.

The UNIX command to copy a file is '''cp source_file target_file'''. So, you would enter

:<div class="box">cp /projects/dmcs_1/courses/job.sh .</div>

The syntax here uses a ''dot'' ('''.''') for the target_file, which causes the source_file (job.sh) to be copied to the present working directory without changing its name.

When typing this, or any, UNIX command, you can start typing just the beginning few characters of the path then press TAB to autocomplete the filename. If the completion is ambiguous, you can continue typing more characters and press TAB again. Pressing TAB twice in a row will show you the set of possible completions. This trick can save you a lot of tedious (mis)typing.

The contents of '''job.sh''' are shown below:

:<div class="box">#!/bin/bash<br >#SBATCH --partition=p_omg_1 # Partition. If you are not a member of OMG enter your own partition<br />#SBATCH --job-name=upwelling # Assign a short name to your job<br/>#SBATCH --nodes=1 # Number of nodes you require<br />#SBATCH --ntasks=4 # Total number of tasks you'll launch<br />#SBATCH --ntasks-per-node=4 # Number of tasks you'll launch on each node<br />#SBATCH --cpus-per-task=1 # Cores per task (>1 if multithread tasks)<br />#SBATCH --mem=6400 # Real memory (RAM) required (MB) per node<br />#SBATCH --time=00-00:05 # Total run time limit (DD-HH:MM)<br />#SBATCH --output=out.%N.%j # STDOUT output file<br />#SBATCH --error=err.%N.%j # STDERR output file (optional but recommended)<br />#SBATCH --export=ALL # Export you current env to the job env<br /><br />## It is important to have --mpi=pmi2 here or ROMS will not run<br />srun --mpi=pmi2 ./romsM roms_upwelling.in</div>

Note: The maximum run time is 14-00:00 (14 days). It is tempting to set this to the maximum in order to avoid a job ending too early. (i.e. Slurm stopping a ROMS run before it is finished) However, if there is maintenance scheduled during that time period the job will not start until maintenance is complete. So a runtime as close as possible to the actual run time is recommended.

====Requesting a specific class of compute node====

For large jobs it makes sense to use all available cores on a node, and for low priority jobs to use older less fast nodes. You can do this by requesting a specific node number, but it is better to request a particular class of nodes so you don't unwittingly pick a node already in use. The classes available from oldest to newest nodes are:

# skylake (32 cores) (Phase I nodes) (nodes hal0116 through hal0120)
# cascadelake (40 cores) (Phase II nodes) (nodes hal0159 through hal0163)
# icelake (64 cores) (Phase V.B nodes) (nodes hal0299 through hal0302)

Make the node class request in the SLURM script and choose the maximum number of cores to match, e.g.

:<div class="box">#SBATCH --constraint=cascadelake # Run on Phase II nodes (40 cores) <br>#SBATCH --ntasks=40 # Total number of tasks to launch (like -np to mpirun) <br>#SBATCH --ntasks-per-node=40 # Number of tasks to launch on each node
</div>

If you don't specifically request a class of nodes the SLURM scheduler will make its own choice on the most appropriate resources to use given your request. We don't know the basis of that choice.

If you request a class of node that is fully assigned to other users in your partition, your job will wait for the others users jobs to complete. So, coordinate with your colleagues if resource use is at a premium. You can use the '''squeue -p partition_name''' command to see what resources are being used by other users, e.g.

:<div class="box">$ squeue -p p_omg_1 <br> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON) <br> 26975473 p_omg_1 fwd levinj R 17:15:42 14-00:00:00 80 2 hal[0159,0162]
</div>

===Running the ROMS Upwelling Example===
You may have noticed that the srun command above includes a file named <span class="blue">roms_upwelling.in</span>. You will need to copy this file and <span class="blue">varinfo.dat</span> from the ROMS/External directory of your <my_src_dir> ROMS source code to the directory where you compiled ROMS.

If, for example, you chose '''myroms''' as the name of your <my_src_dir> when you downloaded the code, you would enter

:<div class="box">cp ~/myroms/ROMS/External/roms_upwelling.in .</div>

The syntax here uses ''tilde'' (~) as shorthand for your home directory,

After you copy the files, you will need to make a couple of small edits (using e.g. '''nano''') to <span class="blue">roms_upwelling.in</span> to get this to work.

# Change the line with VARNAME to read <span class="sandyBrown">VARNAME = varinfo.dat</span> (i.e. delete the ROMS/External part)
# Set <span class="sandyBrown">NtileI == 2</span> and <span class="sandyBrown">NtileJ == 2</span>.

The product of NtileI and NtileJ, i.e. 2 x 2 = 4, is the number of cores the model will run on in parallel. This number must match the number in the SLURM job script options '''--ntasks''' and '''--ntasks-per-node'''

Now you can submit your job to '''sbatch''' with the command '''<span style="font-family: monospace;">sbatch job.sh</span>'''

===ROMS configurations for ''Coastal Ocean Dynamics'' class===
The files that will configure the examples we will study in class will be located in <span class="blue">/projects/dmcs_1/courses/cod</span>.

You will find instructions for configuring and running the examples at a separate Wiki page [[COD_Class_Examples]]

===Checking Job Status with squeue===
Detailed documentation for monitoring your SLURM jobs can be found [https://sites.google.com/view/cluster-user-guide#h.4bsndqufii8p here]. The easiest way to check whether your job is running is with the command '''<span style="font-family: monospace;">squeue -p p_omg_1</span>''' (or your own partition name if it is not p_omg_1).

You should see output something like this:

:<div class="box"> JOBID PARTITION NAME USER ST TIME TIME_LIMIT CPUS NODES NODELIST(REASON)<br /> 17197219 dmcs_1 watl_psas rjdave R 3-04:07:00 3-20:00:00 4 1 hal0035<br /> 17146609 dmcs_1 upwelling rjdave R 00:01:10 00:05:00 4 1 hal0035</div>

You can see that the job is running (the ‘R’ under ‘ST’), has been running for 1 minute, 10 seconds, and is running on node 35.

===Checking Progress with tail===
You can check ROMS progress by using the tail command on the '''output''' file. For the upwelling job above the file would be called <span class="blue">out.hal0035.17146609</span> so the command '''<span style="font-family: monospace;">tail out.hal0035.17146609</span>''' will show you the most recent 10 lines written to the output log and will most likely tell you what time-step the model is on. Using the -f option ('''<span style="font-family: monospace;">tail -f out.hal0035.17146609</span>''') will output appended data as the file grows. Ctrl-C will escape the display.

===Did my UPWELLING job run?===
If the UPWELLING test case was successful there should be an output log file named something like <span class="blue">out.hal0035.17146609</span> that shows no errors, and there should be 4 output NetCDF files ...
:<div class="box">ls -l *.nc </br>roms_avg.nc </br>roms_dia.nc </br>roms_his.nc </br>roms_rst.nc </div>

===Cancelling a Job===
To safest way cancel a queued or running job is to use its jobid. A job can be canceled by name but that is not recommended. To cancel the upwelling job in the example above you would issue the command '''<span style="font-family: monospace;">scancel 17146609</span>'''. If the job has not yet started, it will be removed from the queue. If it is running, all child processes will be killed and the job will be removed from the queue. You are only able cancel jobs that you own.

===Running an interactive session on a compute node===
It is possible to conduct your work interactively on one of the compute nodes (instead of the login node). For most work we will be doing in class this is not necessary, but if for some reason you have a job that needs many processors or a large amount of memory, and you want to run it interactively - say, to simply check that everything is in order for it to start correctly - there are instructions in the Cluster User Guide [https://sites.google.com/view/cluster-user-guide#h.26x9sbburvsg here].

For compute intensive interactive work, such as model analysis using Python or MATLAB, we recommend using the OnDemand interface to launch an interactive session on a set of compute nodes.

==Using the OnDemand desktop environment==
The '''OnDemand''' service allows you to connect to ''Amarel'' using a graphical desktop environment in a web browser. One of the most useful features of '''OnDemand''' is that it allows you to analyze model output by Python or Matlab on the cluster compute nodes.

You need to be running a Rutgers VPN to use OnDemand.

Navigate your browser to https://ondemand.hpc.rutgers.edu/pun/sys/dashboard and login with your NetID and password. At the top of the page click '''Interactive Apps'''.

===Python Jupyter notebook:===
# Select '''Personal Jupyter''' from the list on the left
# Make your selections for '''number of hours''' your session can remain active (e.g. 3 hours), the '''number of cores''' to run on (1), and '''memory''' (8 GB is an OK choice)
# For ''Partition'' enter e.g. <span class="blue">p_omg_1</span> if you are in OMG. Leave ''Reservation'' and ''Slurm feature'' blank. For ''conda path'' enter <span class="blue">/projects/dmcs_1/miniconda3</span> and for ''conda environment'' enter <span class="blue">/projects/dmcs_1/sw/packages/xroms/py38</span>. Click [[Media:Amarel Jupyter config.png | here]] to see what the completed launch page looks like.
# Click '''Launch''' and wait. This can take a couple minutes.
# You should see sub-window of the web page like this </br> [[Image:JupyterStarting.png|600px]] </br></br>When the session is ready it changes to this: </br> [[Image:JupyterRunning.png|600px]]
# Start your notebook by clicking the '''Connect to Jupyter, Anaconda version 5.1.0''' button: (If this does not appear, check that you correctly entered the '''conda path''' and '''conda environment''' at step 3 above.) </br>[[Image:JupyterConnect.png|400px]]
# This should place you in a file browse window from which you can navigate to one of your existing Jupyter notebooks, or, to start a new Untitled notebook, select ''Python 3 (ipykernel)'' from the ''New'' pull-down menu on the right side of the page [[Image:NewPythonNotebook.png]]
# Once your notebook is running, you can import the xroms package as you would any other toolbox, by entering '''<span style="font-family: monospace;">import xroms</span>''' then hold down the shift key and hit the return/enter key. You will see an asterisk ['''*'''] in the square brackets.
# Once that asterisk changes to a '''1''', the xroms python module has been loaded.

This '''conda''' environment built for DMCS_1 makes available many of the Python modules you are used to using.

===Matlab:===
# Select MATLAB from the list on the left
# As for Step 2 above for Python, choose time, cores, and memory
# Select MATLAB version, and enter <span class="blue">p_omg_1</span> (or your own group name) in partition.
# Click '''Launch''' and wait. This can take a couple minutes.
# Once the '''Launch noVNC in New Tab''' button appears, click it and a MatLab GUI will open.

You are running in a graphical desktop environment, so by default Matlab will look in Documents/MATLAB/startup.m for your personal Matlab configuration file.

Popular MATLAB utilities for working with ROMS output are in <span class="blue">/projects/dmcs_1/sw/packages/matlab-tools</span>. These include the myroms.org utilities in subdirectory <span class="blue">myroms</span>, and John Wilkin's toolbox in <span class="blue">roms_wilkin</span>.

===Reconnecting to an interactive session===
A particularly nifty feature of '''OnDemand''' is that your interactive session is independent of the browser window instance.

Say you are working on your office computer using a Jupyter notebook via '''OnDemand''' but haven't finished all you want to do by the end of the day. If you leave that session running, you can go home and connect once again to '''OnDemand''' and you will see your session in the list of Interactive Sessions. Launch that session and you will be placed right back where you were. '''OnDemand''' is simply a web interface to code running on ''Amarel'', so you can connect from any browser.

Of course, this only works if you selected a '''number of hours''' in Step 2 above that will allow your session to remain open.

As long as the session is active the ''Amarel'' resources (cores and memory) are reserved for you and no one else. So, be respectful of other users and close out an interactive session once you are done.

makefile

2023-12-01T14:40:29Z

Robertson:

<div class="title">makefile</div>
{{warning}}<span class="red">Warning: Do not edit the '''makefile'''! Use the [[build Script]].</span> This page is for informational purposes only and is not a guide for compiling ROMS source code.

ROMS has a single '''makefile''' located in the top-level directory. Its generic structure facilitates the compiling of ROMS with other coupled models. The user usually needs to check/modify a few definitions in the '''makefile''' or the [[build Script]] before compiling a particular application. The list of the user options is given below.

This '''makefile''' builds the compiled objects in the specified scratch directory. It determines the compiling depedencies using the [[sfmakedepend]] script located in the '''ROMS/Bin''' directory. Notice that the '''makefile''' time modification in your computer is used by '''make''' to check if the dependencies need to be updated. If you set the definitions below through a [[build Script]], you need to execute '''make clean''' whenever changes to the user options are made.

All the other gory details of the '''makefile''' are covered in [[gmake]].

==Environment Variables in the '''makefile'''==

ROMS has a growing list of choices the user must make about the compilation before starting the compile process, set in user-defined variables. Since we now use '''gnu make''', it is possible to set the value of these variables in the Unix environment, rather than necessarily inside the '''Makefile''' (see [[gmake]]). The user-definable variables understood by the ROMS '''makefile''' are:

<section begin=ROMS_APPLICATION />;<span id="ROMS_APPLICATION"></span>[[ROMS_APPLICATION]]
:CPP option defining a particular application (for example, [[UPWELLING]]). The makefile will include its associated header file which is located in either the '''ROMS/Include''' directory or the path specified in the [[MY_HEADER_DIR]] definition. The header file name is the lowercase value of [[ROMS_APPLICATION]] with the '''.h''' extension (for example, [[upwelling.h]]). Recall the CPP option defining an application must be unique.<section end=ROMS_APPLICATION />

<section begin=MY_HEADER_DIR />;<span id="MY_HEADER_DIR"></span>[[MY_HEADER_DIR]]
: Application header file '''full''' directory path. Recall that a ROMS header file contains all the CPP options defining a particular application. This is the header file that it is included in [[cppdefs.h]]. Notice that if the header file is located in the '''ROMS/Include''' directory, it is possible to include the same header file name located somewhere else because the full directory path is specified here. The user may copy the distributed header file(s) into a different directory and modify it to explore different CPP options.<section end=MY_HEADER_DIR />

<section begin=MY_ANALYTICAL_DIR />;<span id="MY_ANALYTICAL_DIR"></span>[[MY_ANALYTICAL_DIR]]
: Directory containing the user's [[Functionals | analytic files]], if any. Templates of the analytic expressions are provided in the '''User/Functionals''' directory. If analytic expressions are activated, the makefile will load first the ones located in [[MY_ANALYTICAL_DIR]] and then the ones located in '''ROMS/Functionals'''. We recommended that you do not modify the analytical expressions in the '''ROMS/Functionals''' directory since these are the official version of these files and updates will fight with your changes. The user however can modify those in '''User/Functionals''' or some other directory and specify which directory with this variable.<section end=MY_ANALYTICAL_DIR />

<section begin=MY_CPP_FLAGS />;<span id="MY_CPP_FLAGS"></span>[[MY_CPP_FLAGS]]
: Additional CPP options to activate. Sometimes it is desirable to activate one or more CPP options to run different variants of the same application without modifying its header file. If this is the case, specify such options here using the '''-D''' syntax. For example, to write out time-averaged NetCDF file use '''-D'''[[AVERAGES]].<section end=MY_CPP_FLAGS />

<section begin=NestedGrids />;<span id="NestedGrids"></span>[[NestedGrids]]
: Number of ROMS nested and/or composed grids, usually 1.<section end=NestedGrids />

===Compiler-Specific Options===

These flags are used by the files inside the '''Compilers''' directory.

<section begin=USE_DEBUG />;<span id="USE_DEBUG"></span>[[USE_DEBUG]]
: Used to turn on symbolic debug information with no optimization. Set it to '''on''' or leave it blank.<section end=USE_DEBUG />

<section begin=USE_MPI />;<span id="USE_MPI"></span>[[USE_MPI]]
: Used to turn on distributed-memory parallelization in ROMS and linking to the [http://www.llnl.gov/computing/tutorials/mpi/ Message Passing Interface (MPI)] library. Set it to '''on''' or leave it blank.<section end=USE_MPI />

<section begin=USE_MPIF90 />;<span id="USE_MPIF90"></span>[[USE_MPIF90]]
: Used to turn on the [http://www-unix.mcs.anl.gov/mpi/mpich1 MPICH] implementation of MPI via the compilation script '''mpif90''' which is required in some Linux operating systems without a native MPI library ('''-lmpi'''). This definition macro is also convenient when there are several Fortran-90 compilers ('''ifort''', '''pgf90''', '''pathf90''') in the system that use the '''mpif90''' compilation command. In this case, the user needs turn on both '''MPI''' and '''MPIF90''' definitions and specify the appropriate Fortran-90 compiler in '''FORT''' macro definition. Set it to '''on''' or leave it blank.<section end=USE_MPIF90 />

<section begin=USE_OPENMP />;<span id="USE_OPENMP"></span>[[USE_OPENMP]]
: Used to turn on shared-memory parallelization in ROMS using the [http://www.llnl.gov/computing/tutorials/openMP/ OpenMP] compiler directives. Set it to '''on''' or leave it blank.<section end=USE_OPENMP />

<section begin=USE_LARGE />;<span id="USE_LARGE"></span>[[USE_LARGE]]
: Used to turn on 64-bit floating-point number precision. Set it to '''on''' or leave it blank.<section end=USE_LARGE />

<section begin=NETCDF_INCDIR />;<span id="NETCDF_INCDIR"></span>[[NETCDF_INCDIR]]
: The location of the '''netcdf.mod''' and '''typesizes.mod''' files from [http://www.unidata.ucar.edu/software/netcdf/ NetCDF].<section end=NETCDF_INCDIR />

<section begin=NETCDF_LIBDIR />;<span id="NETCDF_LIBDIR"></span>[[NETCDF_LIBDIR]]
: The location of the [http://www.unidata.ucar.edu/software/netcdf/ NetCDF] library.<section end=NETCDF_LIBDIR />

<section begin=USE_NETCDF4 />;<span id="USE_NETCDF4"></span>[[USE_NETCDF4]]
: Set this to '''on''' if linking against the NetCDF4 library, which needs the [http://hdfgroup.com/HDF5/ HDF5] library and therefore:<section end=USE_NETCDF4 />

<section begin=HDF5_LIBDIR />;<span id="HDF5_LIBDIR"></span>[[HDF5_LIBDIR]]
: The location of the [http://hdfgroup.com/HDF5/ HDF5] library.<section end=HDF5_LIBDIR />

<section begin=FORT />;<span id="FORT"></span>[[FORT]]
The user also needs to specify the appropriate Fortran-90 compiler to use. The name of the Fortran-90 compiler depends on the operating system (see table below). This name is also used to determine the compiling configuration file to include in the '''makefile''' by concatenating the operating system ('''uname -s''') and the chosen compiler name.

{| border="1" cellspacing="0" cellpadding="5" align="center"
! Operating System
! Compiler(s)
|-
| AIX
| '''xlf'''
|-
| ALPHA
| '''f90'''
|-
| CYGWIN
| '''g95''', '''df''', '''ifort'''
|-
| Darwin
| '''f90''', '''gfortran''', '''xlf'''
|-
| Linux
| '''ftn''', '''ifc''', '''ifort''', '''pgi''', '''path''', '''g95''', '''gfortran'''
|-
| SunOS
| '''f95'''
|-
| SunOS/Linus
| '''ftn'''
|-
| UNICOS-mp
| '''ftn'''
|-
|}

Note that others can be added as needed, using the existing files as templates.<section end=FORT />

===Local File Options===
<section begin=BINDIR />;<span id="BINDIR"></span>[[BINDIR]]
: Directory path for the compiled executable.<section end=BINDIR />

<section begin=SCRATCH_DIR />;<span id="SCRATCH_DIR"></span>[[SCRATCH_DIR]]
: Directory for '''make''' temporary objects. By default it is set to '''Build'''. If building several projects simultaneously, you need to specify different values for each project. Otherwise, you will compile the wrong files. This directory and the files within are removed when executing a '''make clean'''.<section end=SCRATCH_DIR />

==Providing the Environment==

Before compiling, you will need to find out some background information:
#What is the name of your compiler?
#What is returned by '''uname -s''' on your system?
#Is there a working NetCDF library?
#Where is it?
#Was it built with the above compiler?
#Do you have access to MPI or OpenMP?
As described more fully in [[gmake]], the '''makefile''' will be looking for a file in the '''Compilers''' directory with the combination of your operating system and your compiler. For instance, using Linux and the Pathscale compiler, the file would be called '''Linux-path.mk'''. Is the corresponding file for your system and compiler in the '''Compilers''' directory? If not, you will have to create it following the existing examples there.

Next, there are two ways to provide the location for the NetCDF files (and optional HDF5 library). One is by editing the corresponding lines in your system-compiler file. Another way is through the Unix environment variables. If you are always going to be using the same compiler on each system, you can edit your '''.profile''' or '''.login''' files to globally set them. Here is an example for '''tcsh''':
setenv NETCDF_INCDIR /usr/local/netcdf4/include
setenv NETCDF_LIBDIR /usr/local/netcdf4/lib
setenv HDF5_LIBDIR /usr/local/hdf5/lib
The '''ksh/bash''' equivalent is:
export NETCDF_INCDIR=/usr/local/netcdf4/include
export NETCDF_LIBDIR=/usr/local/netcdf4/lib
export HDF5_LIBDIR=/usr/local/hdf5/lib

For the rest, move on to [[build Script]].

Git

2023-10-12T16:25:06Z

Robertson:

<div class="title">Git</div>
ROMS source code is distributed using both [https://git-scm.com Git] and [https://subversion.apache.org Subversion](SVN). The Git repository includes the full history of changes to the ROMS source code. Details on using Subversion can be found [[Subversion|here]]. There are command line and GUI Git clients available for nearly every operating system and a list of popular clients can be found [https://git-scm.com/downloads/guis here]. This page will help you get started with downloading ROMS with Git.

{{note}}'''Note:''' For instructions using the deprecated and soon to be discontinued myroms.org git repository, click [[Talk:Git|here]].

==Git Overview==
Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency. All the ROMS/TOMS files are stored in a Git repository on '''github.com'''.

This Git repository is the official version of the code which only the developers are allowed to change. Users should download the ROMS code to their local machines using a '''git client'''. Don't attempt to use a regular web browser to browse or download files from the Git repository - there are much better tools for interacting with the code repository.

We strongly recommend users always check out the current ''develop'' version since this has the most recent updates and bug fixes. The ''tags'' are kept largely as a historical record of stable releases at the conclusion of major code upgrades.

If you are making changes of your own, keep them in a separate branch, leaving the ''develop'' branch to track changes from the source. Git makes it ''so'' much easier to manage your own modifications than svn for those of us without write permission on the repository.

Below is a general description of how Git works. Please look at the [https://git-scm.com/book/en/v2 Pro Git] book for more detailed information. We have not yet tried any GUI clients but may add brief how-tos for the most popular GUIs at a later date.

==Configuring GIT==

Before downloading any of '''ROMS''' repositories, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading some '''''roms_test''''' input and observation NetCDF files. Otherwise, the '''Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box">more ~/.gitconf<br /><br /> [User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br /> [credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br /> [filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS==
{{warning}}'''WARNING:''' It is strongly suggested that you clone the ROMS source code using the same operating system you wish to compile and run ROMS on. If you download the code on a Windows machine and wish to run it on a non-Windows machine you will need convert the line endings with a utility like <span class="red">dos2unix</span> or <span class="red">recode</span>. Even with these utilities you may still have problems compiling ROMS.

In order download source code from a git repository, '''git client''' software must be installed on your local machine. Most Linux distributions come with git, so shell commands may be used without installing additional software. The general form of git commands is:

<span class="red">git</span> action <repository>

To check-out the files from the ROMS repository '''develop''' (latest version), enter (notice https instead of http):

> <span class="red">git clone <nowiki>https://github.com/myroms/roms.git</nowiki> MyDir</span>

where '''MyDir''' is the destination directory on your local computer. It will be created if not found. If '''MyDir''' is omitted the code will be downloaded to a directory named '''roms'''. You only clone once because Git will keep track of the source, destination and a bunch of other information. For more detail on command line use and syntax, see the [https://git-scm.com/book/en/v2 Pro Git] book.

The idealized and realistic ROMS Test Cases and the Matlab processing software can be downloaded from:

> <span class="red">git clone <nowiki>https://github.com/myroms/roms_test.git </nowiki></span>
> <span class="red">git clone <nowiki>https://github.com/myroms/roms_matlab.git</nowiki></span>

We highly recommend that Users clone all the ROMS repositories from the same root directory in their computer and define the '''ROMS_ROOT_DIR''' variable in their computer shell login environment, specifying where the User cloned/downloaded the ROMS source code, Test Cases, and Matlab processing software. That is, use the value of pwd from the computer directory where you executed git clone https://github.com/myroms/roms.git

For bash shells:
export ROMS_ROOT_DIR=MyDownlodLocationDirectory

For csh/tcsh shells:
setenv ROMS_ROOT_DIR MyDownlodLocationDirectory

The build scripts will use this environmental variable when compiling any of the ROMS Test Cases without the need to customize the location of the ROMS source code. Also, it is used for loading the path of Matlab scripts in the startup.m configuration file.

===Updates===

Now and again, you might feel the urge to get up to speed with the latest changes that have been made to the ROMS repository. When that happens, simply go to the directory that was "MyDir" above and type:

> <span class="red">git pull</span>

Git will remember where you checked out from before and see if a newer revision exists. If so, it will download and apply all the relevant changes.

==Managing Your Own Modifications==
This assumes that you have a fresh clone of the myroms repository on the '''develop''' branch. You want to keep '''develop''' as a pure copy of the source version and keep your own changes in say the '''arctic''' branch. Start by creating a branch and switching to it:

> <span class="red">git branch arctic<br /></span>> <span class="red">git checkout arctic</span>

Now you can make whatever modifications you like (and test them out). To see what changed, you can use <span class="red">git status</span> and <span class="red">git diff</span>. To save your changes, do a:

> <span class="red">git commit -a</span>

Though if you add new files you will have to <span class="red">git add</span> them first.

===Getting the Updates===
It is easy to fetch and merge the updates. Start by making sure your directory has been cleanly checked in with <span class="red">git status</span>. Then you can update your '''develop''' branch:

> <span class="red">git checkout develop</span><br />> <span class="red">git pull</span>

Then bring the changes into your '''arctic''' branch:

> <span class="red">git checkout arctic</span><br />> <span class="red">git merge develop</span>

This will bring in everything that changed since your last <span class="red">git pull</span>, so you might find it easier to keep on top of things by doing this often, not putting it off for years. You can also bring in changes one at a time with <span class="red">git cherry-pick</span>. Again, check the Pro Git book for much more information about all of these operations.

Note that this will save your '''arctic''' branch locally, under the .git directory. You can back this up as you would any other important files you have.

==Useful Git Commands==
{| border="1" cellspacing="0" cellpadding="5" align="center"
|'''Command'''
|'''Description'''
|-
| '''git add''' <_FileName_>
| Add a file to the repository staging area
|-
| '''git branch'''
| List all local branches (the asterisk denotes the current branch)
|-
| '''git branch -a'''
| List all local and remote branches
|-
| '''git branch -d''' <_BranchName_>
| Delete a local branch
|-
| '''git branch -D''' <_BranchName_>
| Delete a local branch forcefully
|-
| '''git branch -m''' <_OldName_> <_NewName_>
| Rename a local branch
|-
| '''git checkout -b''' <_BranchName_>
| Create a new local branch and switch to it
|-
| '''git checkout''' <_BranchName_>
| Switch to an existing local or remote branch
|-
| '''git clone''' <_RepositoryURL_>
| Clone a public repository
|-
| '''git commit -am''' "<_Message_>"
| Commit changes to all files
|-
| '''git commit -am''' "<_Message_>" '''-m''' "<_MessageURL_>"
| Commit changes to all files with two message lines
|-
| '''git commit --amend -m''' "<_NewMessage_>"
| Amend previous commit message
|-
| '''git diff'''
| Show all changes between HEAD and working branch
|-
| '''git diff''' <_FileName_>
| Show changes for a specific file between HEAD and working branch
|-
| '''git difftool -d''' <_BranchName1_> <_BranchName2_>
| Compare the difference between two branches with '''`KDIFF3`'''
|-
| '''git fetch'''
| Retrieve new work done by other people
|-
| '''git log'''
| View repository changes
|-
| '''git log --oneline'''
| Show the list of commits in one-line format
|-
| '''git log --summary'''
| View repository detailed changes
|-
| '''git merge --no-ff --no-commit''' <_BranchName_>
| Merge a branch into the active branch
|-
| '''git pull'''
| Update local repository or branch to the newest origin commit
|-
| '''git push origin --delete''' <_BranchName_>
| Delete a remote branch
|-
| '''git push -u origin''' <_BranchName_>
| Push changes to the remote repository (and remember the branch)
|-
| '''git reset --hard HEAD~1'''
| Undo last commit
|-
| '''git revert''' <_CommitID_>
| Revert commit changes
|-
| '''git status'''
| Check/display changes to the repository or particular branch
|-
| '''git tag'''
| List all tags
|}

==Complicated Commands==

If a branch is renamed in a repository on GitHub, the local clone on a computer needs to updated:<div class="box">git branch -m OldName NewName<br />git fetch origin<br />git branch -u origin/NewName NewName<br />git remote set-head origin -a</div>

How to merge changes in the '''feature/name1''' branch into the '''feature/name2''' branch:<div class="box">git checkout feature/name1<br />git pull<br /><br />git checkout feature/name2<br />git pull<br />git merge --no-ff --no-commit feature/name1<br />git commit -am "Merging feature/name1 into feature/name2" -m "MessageURL"<br />git difftool -d feature/name1 feature/name2<br />git push -u origin feature/name2</div>

ROMS-JEDI Intro

2023-08-30T20:41:18Z

Robertson:

<div class="title">ROMS-JEDI</div>

<div style="clear: both"></div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>

<div style="clear: both"></div>

[[Image:romsjedi_logo.png|left]]
[[Image:romsjedi_schematic.png|1000px|left]]
<div style="clear: both"></div>

==Introduction==
The Joint Effort for Data Assimilation Integration ('''JEDI''') framework is a unified, high-level, state-of-the-art data assimilation ('''DA''') system for scientific exploration and operational forecasting. The diagram above shows that the '''JEDI''' building blocks are model agnostic, and any modern geophysical numerical kernels can be interfaced. Its separable generic design and abstraction allow the implementation of various '''DA''' methodologies, including variational, ensemble, hybrid, and coupled approaches that are common to any geophysical system, without the need to develop such algorithms separately for each model technically. The '''JEDI''' framework is centered around the Object-Oriented Prediction System ('''OOPS''') layer classes and operators implemented in '''C++''' using templates. The specific implementations like '''ROMS-JEDI''' are written in Fortran 2003 because of the need for class objects, polymorphism, inheritance, and interoperability between '''C++''' and '''Fortran'''. The interoperability binding allows '''Fortran''' to invoke '''C++''' functions and vice versa '''C++''' to invoke '''Fortran''' procedures.

The '''JEDI''' separation of concerns design allows the incorporation of diverse (current and future) observational data streams essential for any DA system. Observation operators can be efficiently coded and added to the Unified Forward Operator ('''UFO''') interface, which simulates, interpolates, and relates observable data to the model component state vector, '''H(x)'''. The '''UFO''' interface also implements quality control, data thinning, and bias correction. The Interface for Observation Data Access ('''IODA''') storage layer ingests observational data from providers into the '''UFO''' and '''OOPS''' building blocks. The immense amount of incoming data can be trimmed down, quality-controlled, and filtered by '''UFO''' operators before it is assimilated and during postprocessing when computing diagnostics, impacts, and sensitivities.

The '''ROMS-JEDI''' framework is intended for users interested in ocean prediction and data assimilation. Currently, we use the '''JEDI''' framework to incorporate physical observational data into regional '''ROMS''' applications using variational ('''3DVar'''/'''4DVar''', '''3D-FGAT'''/'''4D-FGAT'''), ensemble ('''EDA''', '''EnKF''', '''LETKF'''), and hybrid ('''3DEnVar'''/'''4DEnVar''') data assimilation methodologies. The '''ROMS''' Tangent Linear Model ('''TLM''') and Adjoint Model ('''ADM''') kernels are used in the '''4DVar''' family of algorithms.

<center>[[Image:ROMS-JEDI_circle_logo.png]]   [[Image:ROMS_circle_logo.png]]</center>

File:ROMS circle logo.png

2023-08-30T20:39:07Z

Robertson:

File:ROMS-JEDI circle logo.png

2023-08-30T20:38:52Z

Robertson:

ROMS-JEDI Intro

2023-08-30T20:24:40Z

Robertson:

ROMS-JEDI Unit Test Cases

2023-08-30T20:20:22Z

Robertson:

<div class="title">ROMS-JEDI Unit Test Cases</div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>

<div style="clear: both"></div>

There are several ways to run the '''ROMS-JEDI''' '''Unit Test Cases''':

* Run all the available Unit Tests using '''ctest''':<div class="box"><span class="red">cd build_wc13/roms-jedi</span> !> sub-directory of interface root location<br /><span class="red">ctest -N</span> !> lists all the Unit Tests available<br /><span class="red">ctest -E -V get_</span> !> runs all the Unit Tests</div>

* Run specific Unit Test using '''ctest''':<div class="box"><span class="red">cd build_wc13/roms-jedi</span> !> sub-directory of interface root location<br /><span class="red">ctest -VV -R test_romsjedi_geometry</span> !> run geometry Unit Test</div>

* Run specific Unit Test using '''mpirun''' to facilitate TotalView debugging:<div class="box"><span class="red">cd build_wc13/roms-jedi/test</span><br /><span class="red">mpirun -n 2 ../../bin/romsjedi_hofx.x testinput/hofx_4d.yaml > & log &</span><br /><span class="red">mpirun -n 2 -tv ../../bin/romsjedi_hofx.x testinput/hofx_4d.yaml &</span></div>

Currently, the '''ROMS-JEDI''' has the following '''Unit Test Cases''':

{| border="1" cellspacing="0" cellpadding="5" align="center"
! ctest -VV -R !! mpirun -n 2
|-
| '''romsjedi_coding_norms''' || NA
|-
| '''test_romsjedi_geometry''' || '''test_romsjedi_geometry''' ''testinput/geometry.yaml''
|-
| '''test_romsjedi_geometryiterator''' || '''test_romsjedi_geometryiterator''' ''testinput/geometryiterator.yaml''
|-
| '''test_romsjedi_state''' || '''test_romsjedi_state''' ''testinput/state.yaml''
|-
| '''test_romsjedi_increment''' || '''test_romsjedi_increment''' ''testinput/increment.yaml''
|-
| '''test_romsjedi_model''' || '''test_romsjedi_model''' ''testinput/model.yaml''
|-
| '''test_romsjedi_linearmodel''' || '''test_romsjedi_linearmodel''' ''testinput/linearmodel.yaml''
|-
| '''test_romsjedi_error_covariance''' || '''test_romsjedi_error_covariance''' ''testinput/error_covariance.yaml''
|-
| '''test_romsjedi_hofx_nomodel''' || '''../../bin/romsjedi_hofx_nomodel.x''' ''testinput/hofx_nomodel.yaml''
|-
| '''test_romsjedi_hofx_4d''' || '''../../bin/romsjedi_hofx.x''' ''testinput/hofx_4d.yaml''
|-
| '''test_romsjedi_makeobs_4d''' || '''../../bin/romsjedi_hofx.x''' ''testinput/makeobs_4d.yaml''
|-
| '''test_romsjedi_makeobs_4d_perturbed''' || '''../../bin/romsjedi_hofx.x''' ''testinput/makeobs_4d_perturbed.yaml''
|-
| '''test_romsjedi_diffstates''' || '''../../bin/romsjedi_diffstates.x''' ''testinput/diffstates.yaml''
|-
| '''test_romsjedi_forecast_roms''' || '''../../bin/romsjedi_forecast.x''' ''testinput/forecast_roms.yaml''
|-
| '''test_romsjedi_bump_parameters_cor_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/parameters_bump_cor_nicas.yaml''
|-
| '''test_romsjedi_bump_loc_parameters_cor_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/parameters_bump_loc_cor_nicas.yaml''
|-
| '''test_romsjedi_dirac_cov_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/dirac_cov_nicas.yaml''
|-
| '''test_romsjedi_ens_pert''' || '''../../bin/romsjedi_ens_pert.x''' ''testinput/ens_perturbation.yaml''
|-
| '''test_romsjedi_ens_variance''' || '''../../bin/romsjedi_ens_variance.x''' ''testinput/ens_variance.yaml''
|-
| '''test_romsjedi_ens_mean_variance''' || '''../../bin/romsjedi_ens_mean_variance.x''' ''testinput/ens_mean_variance.yaml''
|-
| '''test_romsjedi_ens_recenter''' || '''../../bin/romsjedi_ens_recenter.x''' ''testinput/ens_recenter.yaml''
|-
| '''test_romsjedi_ens_hofx''' || '''mpirun -n 6''' '''../../bin/romsjedi_ens_hofx.x''' ''testinput/ens_hofx.yaml''
|-
| '''test_romsjedi_dirac_ens_cov_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/dirac_ens_cov_nicas.yaml''
|-
| '''test_romsjedi_3dvar_zero_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_zero_obs.yaml''
|-
| '''test_romsjedi_3dvar_regular_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_single_obs.yaml''
|-
| '''test_romsjedi_3dfgat_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_single_obs.yaml''
|-
| '''test_romsjedi_4dfgat_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_single_obs.yaml''
|-
| '''test_romsjedi_3dvar_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_primal.yaml''
|-
| '''test_romsjedi_3dvar_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_dual.yaml''
|-
| '''test_romsjedi_3dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_primal.yaml''
|-
| '''test_romsjedi_3dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_dual.yaml''
|-
| '''test_romsjedi_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_primal.yaml''
|-
| '''test_romsjedi_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_dual.yaml''
|-
| '''test_romsjedi_3denvar_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_regular_primal.yaml''
|-
| '''test_romsjedi_3denvar_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_regular_dual.yaml''
|-
| '''test_romsjedi_3denvar_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_4dfgat_primal.yaml''
|-
| '''test_romsjedi_3denvar_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_4dfgat_dual.yaml''
|-
| '''test_romsjedi_3dhyb_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_regular_primal.yaml''
|-
| '''test_romsjedi_3dhyb_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_regular_dual.yaml''
|-
| '''test_romsjedi_3dhyb_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_4dfgat_primal.yaml''
|-
| '''test_romsjedi_3dhyb_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_4dfgat_dual.yaml''
|-
| '''test_romsjedi_letkf''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf.yaml''
|-
| '''test_romsjedi_letkf_split_observer''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf_split_observer.yaml''
|-
| '''test_romsjedi_letkf_split_solver''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf_split_solver.yaml''
|-
| '''test_romsjedi_letkf_3dhyb''' || '''../../bin/romsjedi_var.x''' ''testinput/letkf_3dhyb.yaml''
|}

ROMS-JEDI Observations

2023-08-30T20:19:53Z

Robertson:

<div class="title">ROMS-JEDI Observations</div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>
<div style="clear: both"></div>

The following diagram shows the observations used to test the '''ROMS-JEDI''' interface with the default '''WC13''' application:

[[Image:romsjedi_obs.png|1000px|left]]
<div style="clear: both"></div>

==IODA Observation files==
<br />

{| border="1" cellspacing="0" cellpadding="5" align="center"
!Observation Type !! NetCDF-4/HDF5 File
|-
| Absolute Dynamic Topography || wc13_adt_20040103.nc4
|-
| ADT single datum || wc13_adt_single_obs_20040105.nc4
|-
| Potential Temperature || wc13_ptemp_20040103.nc4
|-
| Salinity || wc13_salt_20040103.nc4
|-
| Sea Surface Temperature || wc13_sst_20040103.nc4
|-
| SST single datum || wc13_sst_single_obs_20040105.nc4
|-
| ''Insitu'' Temperature || wc13_temp_20040103.nc4
|-
| Temperature-Salinity Profile || wc13_TS_20040103.nc4
|-
| Surface U-velocity || wc13_usur_20040103.nc4
|-
| Surface V-velocity || wc13_vsur_20040103.nc4
|-
| Surface Currents || wc13_uv_sur_20040103.nc4
|}

ROMS-JEDI Implementation

2023-08-30T20:19:36Z

Robertson:

<div class="title">ROMS-JEDI Implementation</div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>

<div style="clear: both"></div>

Most modern geophysical models can be connected to the '''JEDI''' framework, provided that the following predefined abstract C++/Fortran '''classes''' or '''building blocks''' are coded:

{| border="1" cellspacing="0" cellpadding="5" align="center"
!CLASS!!Description
|-
|'''ErrorCovarinace''' || Background error covariance, training, and modeling ('''SABER''': BUMP/NICAS)
|-
| '''Field/Fields''' || Elemental operators to manipulate a field or a set of fields to the model state/increment vector and metadata
|-
| '''Geometry''' || Application grid definition, including coordinates, metrics, parallel decomposition, and operators
|-
| '''GeometryIterator''' || Methods to set/get state fields over specified grid points in '''LETKF''' applications
|-
| '''Increment''' || Procedures to operate on the increment vector that extends/inherits from the '''Fields''' class
|-
| '''LinearModel''' || Initializes, run, and finalizes Tangent Linear and Adjoint model dynamical/numerical kernels
|-
| '''LinearVariableChange''' || Tangent/adjoint increment vector variables transformation from one field to another
|-
| '''Localization''' || Model Ensemble Localization ('''SABER''': BUMP/NICAS)
|-
| '''Model''' || Initializes, run, and finalizes the Nonlinear model dynamical/numerical kernel
|-
| '''State''' || Procedures to operate on the state vector that extends/inherits from the '''Fields''' class
|-
| '''Trajectory''' || Methods to process the Nonlinear trajectory that linearizes the tangent linear and adjoint models
|-
| '''VariableChange''' || Nonlinear state vector variables transformation from one field to another
|}

==Code Design==

Below is a diagram showing the interoperability mechanism for the '''Geometry Class''' that allows '''Fortran''' to invoke '''C++''' function and vice versa for '''C++''' to invoke '''Fortran''' procedures.

[[Image:romsjedi_implementation_schematic.png|1000px|left]]

ROMS-JEDI Compiling

2023-08-30T20:18:53Z

Robertson:

<div class="title">Compiling ROMS-JEDI</div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>__TOC__

<div style="clear: both"></div>

==Configuring GIT==

Before downloading the '''ROMS-JEDI''' repository, ensure that your '''~/.gitconfig''' has the appropriate '''git-lfs''' configuration for correctly downloading the default application (U.S. West Coast, '''WC13''' option) input and observation NetCDF files. Otherwise, the '''Unit Test Cases''' requiring input NetCDF files will fail. The '''Git LFS''' is a command line extension and specification for managing large files with '''Git'''. A sample of the configuration file looks like this:<div class="box"><span class="red">more ~/.gitconf</span><br /><br />[User]<br /> name = GivenName MiddleName FamilyName<br /> email = your@email<br />[credential]<br /> helper = cache --timeout=7200<br /> helper = store --file ~/.my-credentials<br />[filter "lfs"]<br /> clean = git-lfs clean -- %f<br /> smudge = git-lfs smudge -- %f<br /> process = git-lfs filter-process<br /> required = true</div>

Alternatively, you may execute '''git lfs pull''' at your location of this repository to download a viable version of the '''Git LFS''' files from the remote repository in '''GitHub'''. Also, to automatically add the '''LFS''' filter to your existing '''~/.gitconfig''', you could use '''git lfs install''' from anywhere in your computer directories.

==Downloading ROMS-JEDI==

The '''ROMS-JEDI''' repository is private within the Joint Center for Satellite Data Assimilation ('''JCSDA''') community. Users need to be partners, contributors, or members of '''JCSDA''' to access all the '''JEDI''' components that are part of the abstract layer infrastructure centered around the Object Oriented Prediction System ('''OOPS''').

Use the following command to download the '''ROMS-JEDI''' interface:<div class="box"><span class="red">git clone <nowiki>https://github.com/JCSDA-internal/roms-jedi.git</nowiki></span></div>

==Requirements==

* Make sure you have the '''spack-stack''' software installed for either '''gfortran''' or '''ifort''' containing several packages need to run the '''ROMS-JEDI''' interface. For more information, please check https://github.com/JCSDA/spack-stack.<br /><br />In our computers at Rutgers University, the '''spack-stack''' can be loaded by executing either:<div class="box"><span class="red">module purge; module load stack-gcc; module list</span> (gfortran compiler)<br /><span class="red">module purge; module load stack-intel; module list</span> (ifort compiler)</div>Installing the '''spack-stack''' is not trivial and requires computer software expertise.

* We also like to define the '''MPIRUN''' environmental variable to specify the '''MPI''' executable in a particular computer (say, '''srun''', '''mpirun''', etc.). We need the full path for '''ctest''' to work. Notice that we specify this executable in the '''ecbuild''' command below.<br /><br />For example:<div class="box"><span class="twilightBlue">setenv MPIRUN /opt/slurm/bin/srun</span></div>

* Several environment variables can be activated for '''verbose debugging''' of the building blocks for the '''ROMS-JEDI''' interface: (use '''1''' or '''true''' to activate and '''0''' or '''false''' to deactivate)<div class="box"><span class="twilightBlue">setenv LdebugGeometry 1<br />setenv LdebugAnalyticInit 1<br />setenv LdebugField 1<br />setenv LdebugFields 1<br />setenv LdebugFieldsUtils 1<br />setenv LdebugGeometry 1<br />setenv LdebugLinearModel 1<br />setenv LdebugLinearModel2Geovals 1<br />setenv LdebugModel 1<br />setenv LdebugModel2Geovals 1<br />setenv LdebugTrajectory 1</span></div>

==Compiling ROMS-JEDI Default Application WC13==

By default, '''ROMS-JEDI''' is configured with a coarse (54x53x30 grid points) US West Coast ('''WC13''') application. See [[4DVar Tutorial Introduction]] for detailed information about the '''WC13''' configuration. However, any other ROMS application can be configured by including in the '''ecbuild''' command the directives '''`-DROMS_APP="MyAppCPP"`''' and '''-DROMS_APP_DIR="MyAppDirPath"'''.

The strategy is to create a '''Bundle''' and '''build''' subdirectory for each '''ROMS-JEDI''' application. For '''WC13''' (default), we create '''Bundle_wc13''' and '''build_wc13''' below. Please follow the following steps to configure '''ROMS-JEDI''' in your computer:<div class="box"><span class="red">git clone <nowiki>https://github.com/JCSDA-internal/roms-jedi.git</nowiki></span> !> If the first time on your computer<br /><br /><span class="red">cd roms-jedi</span> !> ROMS-JEDI interface directory, <interface_dir><br /><span class="red">mkdir Bundle_wc13</span><br /><span class="red">cp bundle/.gitignore bundle/CMakeLists.txt Bundle_wc13</span><br /><span class="red">mkdir build_wc13</span><br /><span class="red">cd build_wc13</span><br /><br /><span class="red">ecbuild -DMPIEXEC_EXECUTABLE=$MPIRUN -DMPIEXEC_NUMPROC_FLAG="-n" -DCMAKE_BUILD_TYPE=Release ../Bundle_wc13</span><br /><span class="red">make -j 10</span><br /><br /><span class="red">cd roms-jedi</span> !> sub-directory: <interface_dir>/build_wc13/roms-jedi<br /><span class="red">ctest -N</span> !> lists all the Unit Tests available<br /><span class="red">ctest -E -V get_</span> !> runs all the Unit Tests<br /><br /><span class="red">cd test/Data</span> !> sub-directory: <interface_dir>/build_wc13/roms-jedi/test/Data<br /> !> to check the results in various sub-directories</div>

The '''ecbuild''' command will clone the required '''JEDI''' components that are specified in '''Bundle_wc13/CMakeList.txt'''. For example:<div class="box"># Required repositories<br /><br /> ecbuild_bundle( PROJECT roms GIT "<nowiki>https://github.com/JCSDA-internal/roms_src.git</nowiki>" BRANCH master UPDATE )<br /> ecbuild_bundle( PROJECT oops GIT "<nowiki>https://github.com/jcsda-internal/oops.git</nowiki>" BRANCH develop UPDATE )<br /> ecbuild_bundle( PROJECT vader GIT "<nowiki>https://github.com/jcsda-internal/vader.git</nowiki>" BRANCH develop UPDATE )<br /> ecbuild_bundle( PROJECT saber GIT "<nowiki>https://github.com/jcsda-internal/saber.git</nowiki>" BRANCH develop UPDATE )<br /><br /> ecbuild_bundle( PROJECT gsw GIT "<nowiki>https://github.com/jcsda-internal/GSW-Fortran.git</nowiki>" BRANCH develop UPDATE )<br /> ecbuild_bundle( PROJECT ioda GIT "<nowiki>https://github.com/jcsda-internal/ioda.git</nowiki>" BRANCH develop UPDATE )<br /> ecbuild_bundle( PROJECT ufo GIT "<nowiki>https://github.com/jcsda-internal/ufo.git</nowiki>" BRANCH develop UPDATE )<br /><br /> ecbuild_bundle( PROJECT roms-jedi SOURCE ../ )</div>

The '''ecbuild''' tool builds by default the '''RelWithDebInfo''' ('''-O2 -g''' options) for optimized with debugging information version of '''JEDI''' and '''ROMS-JEDI''', rendering slower execution. However, for faster and optimized execution, we include the directive '''-DCMAKE_BUILD_TYPE=Release''' to the '''ecbuild''' command. For debugging with TotalView, we include instead the directive '''-DCMAKE_BUILD_TYPE=Debug''', which is much slower.

==Compiling other ROMS-JEDI Applications==

Any '''ROMS''' application can be run in '''ROMS-JEDI''' by specifying the appropriate CPP option and necessary input files. Only '''WC13''' is provided with the interface. However, we facilitate how to run such applications. Notice that the '''WC13''' configuration and input files are located in '''roms-jedi/test/Applications/wc13'''. The template input YAML files are located '''roms-jedi/test/templates'''. We use a '''Perl''' script to create the application input YAML files from the templates. Please check in '''roms-jedi/test/Applications/wc13/testinput'''.

For example, if running the U.S. East Coast '''DOPPIO''' application (240x104x40 grid points) or any other, please follow the following steps to configure '''ROMS-JEDI''':<div class="box"><span class="red">cd roms-jedi</span> !> ROMS-JEDI interface directory, <interface_dir><br /><span class="red">mkdir Bundle_doppio</span><br /><span class="red">cp bundle/.gitignore bundle/CMakeLists.txt Bundle_doppio</span><br /><span class="red">mkdir build_doppio</span><br /><span class="red">cd build_doppio</span><br /><br /><span class="red">ecbuild -DMPIEXEC_EXECUTABLE=$MPIRUN -DMPIEXEC_NUMPROC_FLAG="-n" -DROMS_APP="DOPPIO" \<br />-DROMS_APP_DIR="DOPPIOpath" -DCMAKE_BUILD_TYPE=Release ../Bundle_doppio</span><br /><br /><span class="red">make -j 10</span><br /><br /><span class="red">cd roms-jedi</span> !> sub-directory: <interface_dir>/build_doppio/roms-jedi<br /><span class="red">ctest -N</span> !> lists all the Unit Tests available<br /><span class="red">ctest -E -V get_</span> !> runs all the Unit Test<br /><span class="red">cd test/Data</span> !> sub-directory: <interface_dir>/build_doppio/roms-doppio/test/Data<br /> !> to check the results in various sub-directories</div>

ROMS-JEDI Intro

2023-08-30T20:18:31Z

Robertson:

<div class="title">ROMS-JEDI</div>

[[Image:romsjedi_logo.png|left]]

<div style="clear: both"></div>


<div style="float: left;margin: 0 20px 0 0;">{{ROMS-JEDI_TOC}}</div>

<div style="clear: both"></div>

[[Image:romsjedi_schematic.png|1000px|left]]
<div style="clear: both"></div>

==Introduction==
The Joint Effort for Data Assimilation Integration ('''JEDI''') framework is a unified, high-level, state-of-the-art data assimilation ('''DA''') system for scientific exploration and operational forecasting. The diagram above shows that the '''JEDI''' building blocks are model agnostic, and any modern geophysical numerical kernels can be interfaced. Its separable generic design and abstraction allow the implementation of various '''DA''' methodologies, including variational, ensemble, hybrid, and coupled approaches that are common to any geophysical system, without the need to develop such algorithms separately for each model technically. The '''JEDI''' framework is centered around the Object-Oriented Prediction System ('''OOPS''') layer classes and operators implemented in '''C++''' using templates. The specific implementations like '''ROMS-JEDI''' are written in Fortran 2003 because of the need for class objects, polymorphism, inheritance, and interoperability between '''C++''' and '''Fortran'''. The interoperability binding allows '''Fortran''' to invoke '''C++''' functions and vice versa '''C++''' to invoke '''Fortran''' procedures.

The '''JEDI''' separation of concerns design allows the incorporation of diverse (current and future) observational data streams essential for any DA system. Observation operators can be efficiently coded and added to the Unified Forward Operator ('''UFO''') interface, which simulates, interpolates, and relates observable data to the model component state vector, '''H(x)'''. The '''UFO''' interface also implements quality control, data thinning, and bias correction. The Interface for Observation Data Access ('''IODA''') storage layer ingests observational data from providers into the '''UFO''' and '''OOPS''' building blocks. The immense amount of incoming data can be trimmed down, quality-controlled, and filtered by '''UFO''' operators before it is assimilated and during postprocessing when computing diagnostics, impacts, and sensitivities.

The '''ROMS-JEDI''' framework is intended for users interested in ocean prediction and data assimilation. Currently, we use the '''JEDI''' framework to incorporate physical observational data into regional '''ROMS''' applications using variational ('''3DVar'''/'''4DVar''', '''3D-FGAT'''/'''4D-FGAT'''), ensemble ('''EDA''', '''EnKF''', '''LETKF'''), and hybrid ('''3DEnVar'''/'''4DEnVar''') data assimilation methodologies. The '''ROMS''' Tangent Linear Model ('''TLM''') and Adjoint Model ('''ADM''') kernels are used in the '''4DVar''' family of algorithms.

Template:ROMS-JEDI TOC

2023-08-30T20:15:47Z

Robertson: Created page with "{|class="toc" |- ! '''ROMS-JEDI Menu''' |- |   1. ''' Introduction  ''' |- |   2. ''' Compiling  ''' |- | &nb..."

{|class="toc"
|-
! '''ROMS-JEDI Menu'''
|-
|   1. '''[[ROMS-JEDI_Intro | Introduction]]  '''
|-
|   2. '''[[ROMS-JEDI_Compiling | Compiling]]  '''
|-
|   3. '''[[ROMS-JEDI_Implementation | Implementation]]  '''
|-
|   4. '''[[ROMS-JEDI_Observations | Observations]]  '''
|-
|   5. '''[[ROMS-JEDI_Unit_Test_Cases | Unit Test Cases]]  '''
|}

Documentation Portal

2023-08-30T20:04:43Z

Robertson:

<div class="title">Documentation Portal</div>

{|class="main" width="99%"
|style="vertical-align:top;width:33%" | 
'''Technical Documentation'''
* [[Regional_Ocean_Modeling_System_(ROMS) | Introduction]]
* [[Equations of Motion]]
* [[Vertical S-coordinate]]
* [[Terrain-Following Coordinate Transformation]]
* [[Curvilinear Coordinates Transformation]]
* [[Fractional Coordinate System (ξ - η space)]]
* [[Time-stepping Schemes Review]]
* [[Numerical Solution Technique]]
* [[Nested Grids]]
* [[Boundary Conditions]]
* [[Horizontal Mixing]]
* [[Vertical Mixing Parameterizations]]
* [[Atmospheric Boundary Layer]]
* [[Bottom Boundary Layer]]
* [[Lagrangian Drifters]]
* [[Balance Term Diagnostics]]
* [[Biogeochemical Models]]
* [[Sediment Model]]
* [[Wave Model]]
* [[Atmospheric Model]]
* [[Model Coupling]]
** [[NUOPC_Cap | ROMS Native ESMF/NUOPC Cap]]
** [[NUOPC_Cap_UFS | ROMS Stand-Alone ESMF/NUOPC Cap]]
* [[Model Diagnostics]]
* [[Sea-Ice Model]]
* [[Variational Data Assimilation]]
* [[ROMS-JEDI_Intro| ROMS-JEDI]]
** [[ROMS-JEDI_Compiling | Compiling]]
** [[ROMS-JEDI_Implementation | Implementation]]
** [[ROMS-JEDI_Observations | Observations]]
** [[ROMS-JEDI_Unit_Test_Cases | Unit Test Cases]]
* [[IO | ROMS I/O]]
* [[Bibliography]]
|style="vertical-align:top;width:33%;border-left:1px solid #aaa;padding-left:1em"| 
'''User Guide'''
* [[Getting Started]]
* [[Tutorials]]
* [[Nesting Examples]]
* [[Frequently Asked Questions]]
* [[Source Code]]
** [[Git]]
** [[Subversion]]
** [[Modules]]
** [[Drivers]]
** [[Subroutines]]
** [[Variables]]
** [[Functionals]]
** [[Lateral Boundary Conditions]]
* [[C Preprocessor]]
** [[Options]]
** [[cppdefs.h]]
** [[globaldefs.h]]
* [[Compiling and Linking]]
** [[makefile]]
** [[build_roms]]
** [[my_build_paths]]
** [[make]]
** [[gmake]]
** [[MakeDepend]]
** [[External Libraries]]
* [[Parallelization]]
* [[Benchmarking]]
* [[Matlab Scripts]]
* [[Visualization]]
|style="vertical-align:top;width:33%;border-left:1px solid #aaa;padding-left:1em"| 
'''Examples and Applications'''
* [[Test Cases]]
* [[Model Coupling ESMF | Coupling Examples]]
* [[Lake_Jersey_Refinement_Example_Introduction | Nesting Examples]]
* [[Application Set-up]]
* [[Metadata Design]]
** [[YAML_Parser | YAML Parser]]
** [[coupling_esmf_atm.yaml]]
** [[roms_cmeps.yaml]]
** [[varinfo.yaml]]
* [[Grid Generation]]
* [[Initialization]]
* [[Forcing]]
** [[Atmospheric Forcing]]
** [[Tidal Forcing]]
** [[River Runoff]]
* [[Input Parameter Files]]
** [[roms.in]]
** [[s4dvar.in]]
** [[biology.in]]
** [[sediment.in]]
** [[floats.in]]
** [[stations.in]]
** [[ice.in]]
** [[swan.in]]
* [[Standard Output Files]]
|}
----

ROMS-JEDI Unit Test Cases

2023-08-30T19:53:10Z

Robertson: Created page with "<div class="title">ROMS-JEDI Unit Test Cases</div> There are several ways to run the '''ROMS-JEDI''' '''Unit Test Cases''': * Run all the available Unit Tests using '''ctest..."

<div class="title">ROMS-JEDI Unit Test Cases</div>

There are several ways to run the '''ROMS-JEDI''' '''Unit Test Cases''':

* Run all the available Unit Tests using '''ctest''':<div class="box"><span class="red">cd build_wc13/roms-jedi</span> !> sub-directory of interface root location<br /><span class="red">ctest -N</span> !> lists all the Unit Tests available<br /><span class="red">ctest -E -V get_</span> !> runs all the Unit Tests</div>

* Run specific Unit Test using '''ctest''':<div class="box"><span class="red">cd build_wc13/roms-jedi</span> !> sub-directory of interface root location<br /><span class="red">ctest -VV -R test_romsjedi_geometry</span> !> run geometry Unit Test</div>

* Run specific Unit Test using '''mpirun''' to facilitate TotalView debugging:<div class="box"><span class="red">cd build_wc13/roms-jedi/test</span><br /><span class="red">mpirun -n 2 ../../bin/romsjedi_hofx.x testinput/hofx_4d.yaml > & log &</span><br /><span class="red">mpirun -n 2 -tv ../../bin/romsjedi_hofx.x testinput/hofx_4d.yaml &</span></div>

Currently, the '''ROMS-JEDI''' has the following '''Unit Test Cases''':

{| border="1" cellspacing="0" cellpadding="5" align="center"
! ctest -VV -R !! mpirun -n 2
|-
| '''romsjedi_coding_norms''' || NA
|-
| '''test_romsjedi_geometry''' || '''test_romsjedi_geometry''' ''testinput/geometry.yaml''
|-
| '''test_romsjedi_geometryiterator''' || '''test_romsjedi_geometryiterator''' ''testinput/geometryiterator.yaml''
|-
| '''test_romsjedi_state''' || '''test_romsjedi_state''' ''testinput/state.yaml''
|-
| '''test_romsjedi_increment''' || '''test_romsjedi_increment''' ''testinput/increment.yaml''
|-
| '''test_romsjedi_model''' || '''test_romsjedi_model''' ''testinput/model.yaml''
|-
| '''test_romsjedi_linearmodel''' || '''test_romsjedi_linearmodel''' ''testinput/linearmodel.yaml''
|-
| '''test_romsjedi_error_covariance''' || '''test_romsjedi_error_covariance''' ''testinput/error_covariance.yaml''
|-
| '''test_romsjedi_hofx_nomodel''' || '''../../bin/romsjedi_hofx_nomodel.x''' ''testinput/hofx_nomodel.yaml''
|-
| '''test_romsjedi_hofx_4d''' || '''../../bin/romsjedi_hofx.x''' ''testinput/hofx_4d.yaml''
|-
| '''test_romsjedi_makeobs_4d''' || '''../../bin/romsjedi_hofx.x''' ''testinput/makeobs_4d.yaml''
|-
| '''test_romsjedi_makeobs_4d_perturbed''' || '''../../bin/romsjedi_hofx.x''' ''testinput/makeobs_4d_perturbed.yaml''
|-
| '''test_romsjedi_diffstates''' || '''../../bin/romsjedi_diffstates.x''' ''testinput/diffstates.yaml''
|-
| '''test_romsjedi_forecast_roms''' || '''../../bin/romsjedi_forecast.x''' ''testinput/forecast_roms.yaml''
|-
| '''test_romsjedi_bump_parameters_cor_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/parameters_bump_cor_nicas.yaml''
|-
| '''test_romsjedi_bump_loc_parameters_cor_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/parameters_bump_loc_cor_nicas.yaml''
|-
| '''test_romsjedi_dirac_cov_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/dirac_cov_nicas.yaml''
|-
| '''test_romsjedi_ens_pert''' || '''../../bin/romsjedi_ens_pert.x''' ''testinput/ens_perturbation.yaml''
|-
| '''test_romsjedi_ens_variance''' || '''../../bin/romsjedi_ens_variance.x''' ''testinput/ens_variance.yaml''
|-
| '''test_romsjedi_ens_mean_variance''' || '''../../bin/romsjedi_ens_mean_variance.x''' ''testinput/ens_mean_variance.yaml''
|-
| '''test_romsjedi_ens_recenter''' || '''../../bin/romsjedi_ens_recenter.x''' ''testinput/ens_recenter.yaml''
|-
| '''test_romsjedi_ens_hofx''' || '''mpirun -n 6''' '''../../bin/romsjedi_ens_hofx.x''' ''testinput/ens_hofx.yaml''
|-
| '''test_romsjedi_dirac_ens_cov_nicas''' || '''../../bin/romsjedi_error_covariance_toolbox.x''' ''testinput/dirac_ens_cov_nicas.yaml''
|-
| '''test_romsjedi_3dvar_zero_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_zero_obs.yaml''
|-
| '''test_romsjedi_3dvar_regular_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_single_obs.yaml''
|-
| '''test_romsjedi_3dfgat_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_single_obs.yaml''
|-
| '''test_romsjedi_4dfgat_single_obs''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_single_obs.yaml''
|-
| '''test_romsjedi_3dvar_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_primal.yaml''
|-
| '''test_romsjedi_3dvar_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dvar_regular_dual.yaml''
|-
| '''test_romsjedi_3dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_primal.yaml''
|-
| '''test_romsjedi_3dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dfgat_dual.yaml''
|-
| '''test_romsjedi_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_primal.yaml''
|-
| '''test_romsjedi_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/4dfgat_dual.yaml''
|-
| '''test_romsjedi_3denvar_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_regular_primal.yaml''
|-
| '''test_romsjedi_3denvar_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_regular_dual.yaml''
|-
| '''test_romsjedi_3denvar_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_4dfgat_primal.yaml''
|-
| '''test_romsjedi_3denvar_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3denvar_4dfgat_dual.yaml''
|-
| '''test_romsjedi_3dhyb_regular_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_regular_primal.yaml''
|-
| '''test_romsjedi_3dhyb_regular_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_regular_dual.yaml''
|-
| '''test_romsjedi_3dhyb_4dfgat_primal''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_4dfgat_primal.yaml''
|-
| '''test_romsjedi_3dhyb_4dfgat_dual''' || '''../../bin/romsjedi_var.x''' ''testinput/3dhyb_4dfgat_dual.yaml''
|-
| '''test_romsjedi_letkf''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf.yaml''
|-
| '''test_romsjedi_letkf_split_observer''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf_split_observer.yaml''
|-
| '''test_romsjedi_letkf_split_solver''' || '''../../bin/romsjedi_letkf.x''' ''testinput/letkf_split_solver.yaml''
|-
| '''test_romsjedi_letkf_3dhyb''' || '''../../bin/romsjedi_var.x''' ''testinput/letkf_3dhyb.yaml''
|}

ROMS-JEDI Observations

2023-08-30T19:32:43Z

Robertson: Created page with "<div class="title">ROMS-JEDI Observations</div> The following diagram shows the observations used to test the '''ROMS-JEDI''' interface with the default '''WC13''' applicatio..."

<div class="title">ROMS-JEDI Observations</div>

The following diagram shows the observations used to test the '''ROMS-JEDI''' interface with the default '''WC13''' application:

[[Image:romsjedi_obs.png|1000px|left]]
<div style="clear: both"></div>

==IODA Observation files==
<br />

{| border="1" cellspacing="0" cellpadding="5" align="center"
!Observation Type !! NetCDF-4/HDF5 File
|-
| Absolute Dynamic Topography || wc13_adt_20040103.nc4
|-
| ADT single datum || wc13_adt_single_obs_20040105.nc4
|-
| Potential Temperature || wc13_ptemp_20040103.nc4
|-
| Salinity || wc13_salt_20040103.nc4
|-
| Sea Surface Temperature || wc13_sst_20040103.nc4
|-
| SST single datum || wc13_sst_single_obs_20040105.nc4
|-
| ''Insitu'' Temperature || wc13_temp_20040103.nc4
|-
| Temperature-Salinity Profile || wc13_TS_20040103.nc4
|-
| Surface U-velocity || wc13_usur_20040103.nc4
|-
| Surface V-velocity || wc13_vsur_20040103.nc4
|-
| Surface Currents || wc13_uv_sur_20040103.nc4
|}

File:romsjedi obs.png

2023-08-30T19:25:13Z

Robertson: