NUOPC Cap
Overview
This document describes the ROMS ESMF/NUOPC cap module Master/esmf_roms.F used in the ROMS native coupling infrastructure shown below. In this case, ROMS is driving the coupled system (Master/esmf_driver.h), which creates, register, initialize, run, and finalize the coupling of each Earth System Model (ESM) component (data, atmosphere, sea-ice, wave, and ocean). It also sets the coupler connectors and the RunSequence policy (Master/esmf_esm.F). The framework provides the NUOPC cap modules for each coupled component. Detailed information about ESMF/NUOPC and how it is implemented in ROMS can be found on the Earth System Modeling Framework WikiROMS page.
A NUOPC cap is a Fortran module that serves as the interface to a model when used in a NUOPC-based coupled system. The term cap is used because it is a lightweight software layer that sits on top of the model kernel, making calls to its initialize, run, and finalize phases and exposing the model data structures to the field exchange (import and export states) between coupled components.

Implementation
The Master/mod_esmf_esm.F module declares several derived-type structures to facilitate the management of all internal objects and variables in each ESM coupled component:
- ESM coupling time managing variables and ESMF objects. The ClockInfo object is allocated as (0:Nmodels), where Nmodels is the number of coupled ESM components. Here, index zero corresponds to the driver.TYPE :: ESM_Clock
logical :: Restarted
integer (i8b) :: AdvanceCount ! advance counter
real (dp) :: Current_Time ! seconds
real (dp) :: Time_Reference ! seconds
real (dp) :: Time_Restart ! seconds
real (dp) :: Time_Start ! seconds
real (dp) :: Time_Stop ! seconds
real (dp) :: Time_Step ! seconds
character (len=22) :: Name
character (len=22) :: CalendarString ! 360_day, gregorian
character (len=22) :: Time_ReferenceString
character (len=22) :: Time_RestartString
character (len=22) :: Time_StartString
character (len=22) :: Time_StopString
TYPE (ESMF_Calendar) :: Calendar
TYPE (ESMF_Clock) :: Clock
TYPE (ESMF_Direction_flag) :: Direction
TYPE (ESMF_Time) :: CurrentTime
TYPE (ESMF_Time) :: ReferenceTime
TYPE (ESMF_Time) :: RestartTime
TYPE (ESMF_Time) :: StartTime
TYPE (ESMF_Time) :: StopTime
TYPE (ESMF_TimeInterval) :: TimeStep
END TYPE ESM_Clock
. . .
TYPE (ESM_Clock), allocatable, target :: ClockInfo(:)
- ESM coupled state sets. If appropriate, it includes the logic for connecting nested grids. The COUPLED object is of size Nmodels.TYPE :: ESM_CplSet
logical, allocatable :: LinkedGrid(:,:) ! connected grid
logical, allocatable :: DataCoupledSets(:,:) ! DATA linked sets
character (len=100), allocatable :: SetLabel(:) ! set label
character (len=100), allocatable :: ExpLabel(:) ! export label
character (len=100), allocatable :: ImpLabel(:) ! import label
TYPE (ESMF_State), allocatable :: ExportState(:,:) ! export set
END TYPE ESM_CplSet
TYPE (ESM_CplSet), allocatable, target :: COUPLED(:)
- DATA model field processing information. The DATA component is often necessary for applications with incongruent grids to provide complete field melding capabilities or methods for reading and interpolating stream data in space and time from archived NetCDF files.TYPE ESM_Data
logical :: Lcycle ! cycling time coordinate
logical :: Lcoord ! coordinates attribute
logical :: Lmask ! land/sea mask
logical :: Lmulti ! field across multi-files
logical :: LastRec ! processed last record
integer :: Icomp ! target component index
integer :: ncid ! NetCDF file ID
integer :: Tid ! time variable ID
integer :: Vid ! field variable ID
integer :: Nvdim ! number spatial dimensions
integer :: Zlevel ! level index to process
integer :: Nrec ! number of time records
integer :: Trec ! latest read time record
integer :: Tindex ! rolling two-time indices
integer :: LandValue ! Masking land value
integer :: SeaValue ! Masking sea value
integer, allocatable :: Vsize(:) ! dimensions size
real(dp) :: add_offset ! add_offset attribute
real(dp) :: FillValue ! _FillValue attribute
real(dp) :: scale_factor ! scale_factor attribute
real(dp) :: Clength ! time cycling length
real(r8) :: LonMin ! grid minimum longitude
real(r8) :: LonMax ! grid maximum longitude
real(r8) :: LatMin ! grid minimum latitude
real(r8) :: LatMax ! grid maximum latitude
real(dp) :: Tscale ! time scale to day
real(dp) :: Tmono ! monotonic time (days)
real(dp) :: Tmin ! time minimum value
real(dp) :: Tmax ! time maximum value
real(dp) :: Tstr ! lower time-snapshot
real(dp) :: Tend ! upper time-snapshot
real(r8) :: Vmin ! variable minimum value
real(r8) :: Vmax ! variable maximum value
real(dp) :: Tintrp(2) ! interpolation time (days)
real(dp) :: Vtime(2) ! latest two-time values
real(dp) :: Date(6,2) ! time-snapshots dates YYYY,MM,DD hh:mm:ss.ss
character(len=30), allocatable :: Dname(:) ! variable dimensions names
character(len=20), allocatable :: Vcoord(:) ! variable coordinates names
character(len=22 ) :: DateString(2) ! date-snapshots string
character(len=30 ) :: SpecialAction ! special processing
character(len=100) :: Tname ! time variable name
character(len=100) :: Tunits ! time variable units
character(len=100) :: Vname ! variable name
character(len=100) :: Vunits ! variable units
character(len=256) :: Vdescriptor ! Variable descriptive name
character(len=256) :: Vlongname ! long_name attribute
character(len=256) :: ncfile ! NetCDF filename
real(r8), allocatable :: lon(:,:) ! field longitude
real(r8), allocatable :: lat(:,:) ! field latitude
real(r8), allocatable :: mask(:,:) ! field land/sea mask
real(r8), allocatable :: A2d(:,:) ! time interpolated 2D data
real(r8), allocatable :: A3d(:,:,:) ! time interpolated 3D data
real(r8), allocatable :: A2dG(:,:,:) ! latest 2D data snapsnots
real(r8), allocatable :: A3dG(:,:,:,:)! latest 3D data snapsnots
TYPE (ESMF_Field) :: field ! field object
TYPE (ESMF_Grid) :: grid ! field grid object
TYPE (ESMF_RouteHandle) :: rhandle ! field RouteHandle
END TYPE ESM_Data
- DATA model high-level structure. TYPE ESM_DataSet
integer :: Nfields ! number of fields
integer :: Nfiles ! number of input files
character(len=20), allocatable :: Ctarget(:) ! component target
character(len=20), allocatable :: Field(:) ! field short-name
TYPE (ESM_Data), allocatable :: Export(:) ! Export field structure
TYPE (T_IO), allocatable :: IFS(:) ! Input Files structure
END TYPE ESM_DataSet
- Import and export field metadata information between the source (SRC) and destination (DST) components. TYPE :: ESM_Field
logical :: connected ! connected to coupler
logical :: debug_write ! write exchanged field
logical :: enable_integral_adj ! area integral adjusted
integer :: fid ! internal field ID
integer :: gtype ! field grid mesh type
integer :: etype ! field extrapolation flag
integer :: itype ! field interpolation flag
integer :: Tindex ! rolling two-time indices
character (len=20) :: Ctarget ! component destination
character (len=22) :: DateString(2) ! date-snapshots string
character (len=:), allocatable :: short_name ! short name
character (len=:), allocatable :: standard_name ! standard name
character (len=:), allocatable :: long_name ! long name
character (len=:), allocatable :: dst_gtype ! DST grid type
character (len=:), allocatable :: dst_units ! DST units
character (len=:), allocatable :: src_gtype ! SRC grid type
character (len=:), allocatable :: src_units ! SRC units
character (len=:), allocatable :: nc_vname ! DATA Vname
character (len=:), allocatable :: nc_tname ! DATA Tname
character (len=:), allocatable :: RegridMethod ! regrid method
character (len=:), allocatable :: ExtrapMethod ! extrapolate
real (dp) :: scale_factor ! field scale factor
real (dp) :: add_offset ! field add offset value
real (dp) :: Tmin ! DATA time minimum value
real (dp) :: Tmax ! DATA time maximum value
real (dp) :: Tstr ! DATA lower time-snapshot
real (dp) :: Tend ! DATA upper time-snapshot
real (dp) :: Tintrp(2) ! interpolation time (day)
real (dp) :: Vtime(2) ! latest two-time values
TYPE (ESMF_RouteHandle) :: rhandle ! field RouteHandle
END TYPE ESM_Field
- Import and export fields mesh data. TYPE :: ESM_Mesh
integer :: gid ! grid ID
integer :: gtype ! grid mesh type
integer (i4b), allocatable :: mask(:,:) ! grid land/sea mask
real (r8), allocatable :: lon(:,:) ! grid longitude
real (r8), allocatable :: lat(:,:) ! grid latitude
real (r8), allocatable :: area(:,:) ! grid area
END TYPE ESM_Mesh
- Melding coefficients used to combine fields from DATA and ESM components. The weight factors are read from the input NetCDF specified in the WeightsFile(atmos) keyword. The user has full control of how the merging is done. It is recommended to provide a gradual transition between the two components.
Recall that the DATA component supplies needed data to a particular ESM component. For example, it may export data to the atmosphere model at locations not covered by the other ESM components because of smaller grid coverage. If the atmosphere and ocean model grids are incongruent, the atmosphere component needs to import sea surface temperature (SST) on those grid points not covered by the ocean component. Thus, the weighting coefficients are used to merge the SST data:SST_atm(:,:) = Cesm(:,:) * SST_esm(:,:) + Cdat(:,:) * SST_dat(:,:)where Cesm(:,:) + Cdat(:,:) = 1.TYPE ESM_Meld
integer :: NestedGrid ! grid needing merged field
character(len=100) :: VnameDATA ! DATA weights variable name
character(len=100) :: VnameESM ! ESM weights variable name
character(len=256) :: ncfile ! Weights NetCDF filename
real (r8), allocatable :: Cdat(:,:) ! coefficients for DATA
real (r8), allocatable :: Cesm(:,:) ! coefficients for ESM
END TYPE ESM_Meld
TYPE (ESM_Meld), allocatable, target :: WEIGHTS(:)
real (dp) :: WeightDAT = 0.0_dp ! DATA component weight
real (dp) :: WeightESM = 1.0_dp ! ESM component weight
- Coupled models high-level object, [Nmodels=5]. TYPE :: ESM_Model
logical :: IsActive ! active for coupling
integer (i4b) :: LandValue ! land mask value
integer (i4b) :: SeaValue ! sea mask value
integer :: Ngrids ! number nested grids
integer :: ExportCalls ! export CALL counter
integer :: ImportCalls ! import CALL counter
integer :: nPETs ! number model PETs
integer, allocatable :: PETlist(:) ! component PETs list
integer, allocatable :: TimeFrac(:,:) ! driver time fraction
character (len=:), allocatable :: name ! component name
TYPE (ESMF_Grid), allocatable :: grid(:) ! grid object
TYPE (ESM_Mesh), allocatable :: mesh(:) ! mesh
TYPE (ESM_Field), allocatable :: ImportField(:) ! import fields
TYPE (ESM_Field), allocatable :: ExportField(:) ! export fields
TYPE (ESMF_State), allocatable :: ImportState(:) ! import state
TYPE (ESMF_State), allocatable :: ExportState(:) ! export state
END TYPE ESM_Model
TYPE (ESM_Model), allocatable, target :: MODELS(:)
- Coupling models connector used for the interpolation/extrapolaton between source and destination fields, [Nmodels, Nmodels]. TYPE ESM_Conn
logical :: IsActive ! active connector
integer :: divDT
integer :: MaskInteraction ! connector mask interaction
integer :: nPETs ! number of connector PETs
integer, allocatable :: PETlist(:) ! connector PETs list
character (len=100) :: name ! connector name
END TYPE ESM_Conn
TYPE (ESM_Conn), allocatable, target :: CONNECTORS(:,:)
- Define DATA Model field processing information from input data files, [1:Nmodels]. Currently, the DATA Model only export fields and it only supports input NetCDF files. TYPE (ESM_DataSet), allocatable, target :: DataSet(:)
- ESM clock for driver (zeroth element) and coupled components, [0:Nmodels]. TYPE (ESM_Clock), allocatable, target :: ClockInfo(:)
- Internal module parameters and variables:
- Number of coupled ESM gridded components. Currently, five ESM components are supported (ROMS, DATA, ATMOSPHERE, SEA-ICE, and WAVE model types).
All supported components are accounted here even if we are running an application with less number of models. The IsActive switches are use to operate only on the desired coupled components. This is done to have complete infornation in the above structures. The gridded arrays are never allocated if a particular component is not active.integer, parameter :: Nmodels = 5 - Number of nested grids. All are initialized to just one grid; its values are overwritten during processing. An additional variable NgridsR is created to avoid using ROMS module mod_param in the generic interface. Both Ngrids and NgridsR will have the same value.integer, parameter :: NgridsA = 1 ! Atmosphere Model
integer, parameter :: NgridsD = 1 ! DATA Model
integer, parameter :: NgridsI = 1 ! Sea-ice Model
integer, parameter :: NgridsR = 1 ! ROMS
integer, parameter :: NgridsW = 1 ! Wave Model - Coupled ESM components identification indices.integer, parameter :: Idriver = 0
integer, parameter :: Iroms = 1
integer, parameter :: Iatmos = 2
integer, parameter :: Idata = 3
integer, parameter :: Iseaice = 4
integer, parameter :: Iwave = 5 - Generic ESM component labels used in the CASE constructs. We cannot use the identification indices because the vector Iroms(:) cannot be defined as a parameter and a non constant expression is illegal: CASE ( Iroms(1) : Iroms(NgridsR) )character (len= 3), allocatable :: Clabel(:)
character (len=10), allocatable :: Cmodel(:) - Number of ROMS export and import fields per component. integer, allocatable :: Nexport(:)
integer, allocatable :: Nimport(:) - Model coupling type: [1] Explicit, [otherwise] Semi-Implicit. In explicit coupling, exchange fields at the next time-step are defined using known values from the time-step before it. Explicit methods require less computational effort and are accurate for small coupling time-steps. In implicit coupling, exchange fields at the next time-step are defined by including values at the next time-step. Implicit methods are stable and allow longer coupling time-steps but are more computationally expensive. In semi-implicit coupling, ROMS -> ATM is explicit, ATM -> ROMS is implicit. integer :: CouplingType = 1
- Distributed-memory communicator handle for each component, Total number of PETs needed in concurrent PET layout, rank of each PET, and PET layout (sequential or concurrent). integer, allocatable :: ESMcomm(:)
integer :: sumPETs
integer :: PETrank
character (len=10) :: PETlayoutOption - Driver clock parameters specified in configuration script. A integer vector with six elements: (1) year including century, like 2017; (2) month of the year, 1 to 12; (3) day of the month; (4) hour of the day, 0 to 23; (5) minutes of the hour, 0 to 59; and (6) seconds of the minute, 0 to 59 integer :: ReferenceDate(6) ! reference date
integer :: RestartDate(6) ! restarting date
integer :: StartDate(6) ! starting date
integer :: StopDate(6) ! stopping date
integer :: TimeStep(6) ! coupling interval - Today date string. character (len=44) :: TodayDateString
- ESM coupling simulation reference date number, element: (1) seconds, and (2) fractional daysreal (dp) :: ReferenceDateNumber(2)
- DATA component parallel distributed-memory domain partions in the I- and J-directions (lon,lat). integer :: ItileD
integer :: JtileD - Coupling debugging flag: [0] no debugging, [1] reports informative messages, or [2] '1' plus writes grid information in VTK format, [3] '2' plus writes exchage fields into NetCDF files. integer :: DebugLevel = 0
- Execution tracing level flag: [0] no tracing, [1] reports sequence of coupling subroutine calls, or [2] <1> plus writes voluminous ESMF library tracing information which slowdown performace, and creates large log file. integer :: TraceLevel = 0
- Switch to trace/track run sequence during debugging. All information is written to Fortan unit trac. For now, use standard output unit. logical :: ESM_track = .FALSE. ! trace/track CALL sequence switch
integer :: trac = 6 ! trace/track CALL sequence unit - Coupled model staggered grid-cell type indices. integer, parameter :: Inan = 0 ! unstaggered, cell center
integer, parameter :: Icenter = 1 ! cell center
integer, parameter :: Icorner = 2 ! cell corners
integer, parameter :: Iupoint = 3 ! right and left cell faces
integer, parameter :: Ivpoint = 4 ! upper and lower cell faces
character (len=6), dimension(0:4) :: GridType = &
& (/ 'N/A ', &
& 'Center', &
& 'Corner', &
& 'U ', &
& 'V ' /) - REGRID interpolation method between source and destination fields. integer, parameter :: Inone = 0 ! none
integer, parameter :: Ibilin = 1 ! bilinear
integer, parameter :: Ipatch = 2 ! high-order patch recovery
integer, parameter :: Iconsv1 = 3 ! first-order conservative
integer, parameter :: Iconsv2 = 4 ! second-order conservative
integer, parameter :: InStoD = 5 ! nearest neighbor Src to Dst
integer, parameter :: InDtoS = 6 ! nearest neighbor Dst to Src
character (len=4), dimension(0:6) :: IntrpType = &
& (/ 'NONE', &
& 'BLIN', &
& 'PTCH', &
& 'CNS1', &
& 'CNS2', &
& 'NS2D', &
& 'ND2S' /) - Extrapolation method for unmapped destination points. integer, parameter :: Enone = 0 ! none
integer, parameter :: ExStoD = 1 ! nearear neighbor Src to Dst
integer, parameter :: Eidavg = 2 ! inverse distance average
integer, parameter :: Ecreep = 3 ! creep fill
integer, parameter :: E2steps = 4 ! Turuncoglu two steps
character (len=4), dimension(0:4) :: ExtrpType = &
& (/ 'NONE', &
& 'NS2D', &
& 'IDAV', &
& 'CREE', &
& '2STP' /) - The number of levels to output for the extrapolation methods that fill levels, like creep fill (ESMF_EXTRAPMETHOD_CREEP). Unmapped destination points are supplied by repeatedly moving data from mapped locations to neighboring unmapped locations for a user-specified number of levels. For each creeped point, its value is the average of the values of the immediate neighbors from the mapped points from regridding (ESMF Reference Manual, v 8.0.0). integer :: extrapNumLevels = 1
- Interpolation connectors mask interaction flags. integer, parameter :: OverLand = 1
integer, parameter :: OverOcean = 2
integer, parameter :: OverAll = 3
character (len=3), dimension(3) :: MaskType = &
& (/ 'LND', &
& 'OCN', &
& 'ALL'/) - Coupling run mode: sequential or concurrent. integer, parameter :: Iseq = 1
integer, parameter :: Ipar = 2
character (len=10), dimension(2) :: RunMode = &
& (/ 'SEQUENTIAL', &
& 'CONCURRENT' /) - Compling standard input parameters filename. character (len=256) :: CinpName
- ESM free-format run sequence configuration filename. character (len=256) :: CONFname
- ROMS coupling YAML configuration filename. character (len=256), allocatable :: CPLname
- Standard input filename for each coupled model, [Nmodels]. character (len=256), allocatable :: INPname(:)
- Standard output units and log messages filename for coupler and ESMF library. integer :: cplout = 77 ! coupling driver
integer :: dataout = 77 ! data component
character (len= 8), parameter :: ESMnameLog = 'log.esmf'
character (len= 8), parameter :: CouplerLog = 'log.coupler' - Output NetCDF file used to store field snapshot attributes needed for time interpolation by the ESM component kernel during concurrent coupling. character (len=17), parameter :: AttFileName = 'time_intrp_att.nc'
- ESM single and double precision constants. integer (i4b), parameter :: MAPPED_MASK = 99_i4b
integer (i4b), parameter :: UNMAPPED_MASK = 98_i4b
real (dp), parameter :: MISSING_dp = 1.0E20_dp
real (r4), parameter :: MISSING_r4 = 1.0E20_r4
real (r8), parameter :: MISSING_r8 = 1.0E20_r8
real (dp), parameter :: TOL_dp = 0.5E20_dp
real (r4), parameter :: TOL_r4 = 0.5E20_r4
real (r8), parameter :: TOL_r8 = 0.5E20_r8
- Number of coupled ESM gridded components. Currently, five ESM components are supported (ROMS, DATA, ATMOSPHERE, SEA-ICE, and WAVE model types).
Capabilities
The ROMS cap module contains a set of subroutines that are required by NUOPC. According to a predefined calling sequence, these subroutines are called by the NUOPC infrastructure. Some subroutines are called during the initialization of the coupled system, some during the run of the coupled system, and some during the finalization of the coupled system.
The initialization sequence is the most complex and is governed by the NUOPC technical rules. Details about the initialization sequence can be found in the NUOPC Reference Manual. The ROMS cap requires ESMF version 8 or higher.
ROMS_SetServices Entry point to the ROMS cap and the only public routine. It sets the ROMS component chared-object entry points for using NUOPC generic methods for initialize, run, and finalize. ROMS_SetInitializeP1 ROMS component phase 1 initialization which sets import and export fields long and short names into its respective state. ROMS_SetInitializeP2 ROMS component phase 2 initialization which initializes the ROMS component (ROMS_initialize), sets component grid (ROMS_SetGridArrays), and adds fields into import and export into respective states. ROMS_DataInit Exports ROMS component fields during initialization or restart. ROMS_SetClock Sets ROMS component date calendar, start and stop times, and coupling interval. ROMS_SetRunClock Sets ROMS run clock manually to avoid getting zero time stamps at the first regridding call. ROMS_CheckImport Checks if ROMS component import field is at the correct time. ROMS_SetGridArrays Sets ROMS component staggered, horizontal grid arrays, grid area, and land/sea mask if any. ROMS_SetStates Adds ROMS component export and import fields into its respective state. ROMS_ModelAdvance Advances ROMS component for a coupling interval. It calls ROMS_Import and ROMS_Export routines. ROMS_SetFinalize Finalizes ROMS component execution. ROMS_Import Loads import fields into ROMS internal kernel arrays. ROMS_Export Exports ROMS fields to other gridded components. ROMS_Rotate Rotates vector components from computational grid to geographical EAST and NORTH directions or vice versa.
Other NUOPC cap modules are available within the ROMS native framework:
ESMF/NUOPC Modules Tested Description Master/esmf_driver.h yes ESMF/NUOPC Master Driver Master/esmf_esm.F yes Sets ESMF/NUOPC Services and RunSequence for each coupled component Master/esmf_coupler.h yes Computes, Execute, and Release the Connectors between source and destination fields Master/mod_esmf_esm.h yes Defines framework object structures and variables and includes support routines Master/esmf_atm_coamps.h yes Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS) Master/esmf_atm_coamps.h yes Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS) Master/esmf_atm_regcm.h no ICTP Regional Climate Model (RegCM, Version 4.5) Master/esmf_atm_wrf.h yes Weather Research and Forecasting model (WRF) Master/esmf_data.F yes Generic DATA component via NetCDF files Master/esmf_ice_cice.h no Los Alamos Sea Ice Model (CICE) Master/esmf_roms.F yes Regional Ocean Modeling System (ROMS) Master/esmf_wav_wam.h no ECMWF's Wave Model (WAM) Master/esmf_wav_ww3.h no WaveWatch III (WW3), under development
Import and Export Fields
The export and import fields are configured and specified in the YAML file coupling_esmf_atm.yaml. Check the following wikiROMS page for detailed information. The user has full control of the coupling exchange fields metadata.
The ESMF/NUOPC library version 8.0.0 or higher supports native nesting capabilities. The NUOPC layer now allows coupling sets in the import and export states of the ESM components. A nested grid is represented as another ESMF_Grid object so that a coupled model can have a set of telescoping meshes of decreasing spatial resolution. The NUOPC connector can recognize the different grids and exchange fields to a connected ESM component. Various types of connections are possible, like nest-to-nest, fine-to-coarse, or coarse-to-fine. The user decides which nested grids to connect. The NUOPC_AddNestedState routine is used to advertise the nested couple sets.
For example, the ROMS NOUPC cap module routine ROMS_SetInitializeP1 illustrates how easily the coupled sets is added to the ImportState and ExportState as shown below in the green blocks of code.
![]() |
![]() |