NUOPC Cap UFS

ROMS ESMF/NUOPC Cap for the UFS

Overview

This document describes the stand-alone ROMS ESMF/NUOPC cap module to be used by third-party coupling frameworks, like the Unified Forecast System (UFS). It is a lightweight software layer on top of ROMS that can be used by NUOPC-based packages (CMEPS/CDEPS, NEMS, and others) to couple to other Earth System Models (ESMs). Detailed information about ESMF/NUOPC and its implementation in ROMS can be found on the Earth System Modeling Framework WikiROMS page.

A NUOPC cap is a Fortran module that serves as the interface to a model when it's used in a NUOPC-based coupled system. The term cap is used because it is a lightweight software layer that sits on top of model code, making calls to it and exposing model data structures in a standard way.

Implementation

The cmeps_roms.F module declare several derive-type structure to facilitate the management of all internal objects and variables:

• ESM coupling time managing variables and ESMF objects.
  TYPE, PRIVATE :: ESM_Clock
  
  logical :: Restarted
  
  integer (i8b) :: AdvanceCount ! advance counter
  
  real (dp) :: Current_Time ! seconds
  real (dp) :: Time_Reference ! seconds
  real (dp) :: Time_Restart ! seconds
  real (dp) :: Time_Start ! seconds
  real (dp) :: Time_Stop ! seconds
  real (dp) :: Time_Step ! seconds
  
  character (len=22) :: Name
  character (len=22) :: CalendarString ! 360_day, gregorian
  character (len=22) :: Time_ReferenceString
  character (len=22) :: Time_RestartString
  character (len=22) :: Time_StartString
  character (len=22) :: Time_StopString
  
  TYPE (ESMF_Calendar)  :: Calendar
  TYPE (ESMF_Clock)  :: Clock
  TYPE (ESMF_Direction_flag) :: Direction
  TYPE (ESMF_Time)  :: CurrentTime
  TYPE (ESMF_Time)  :: ReferenceTime
  TYPE (ESMF_Time)  :: RestartTime
  TYPE (ESMF_Time)  :: StartTime
  TYPE (ESMF_Time)  :: StopTime
  TYPE (ESMF_TimeInterval)  :: TimeStep
  
  END TYPE ESM_Clock
  
  TYPE (ESM_Clock), allocatable, target :: ClockInfo(:)

• ESM coupled state sets. If appropriate, it includes the logic for connecting nested grids.
  TYPE, PRIVATE :: ESM_CplSet
  
  logical, allocatable :: LinkedGrid(:,:) ! connected grid
  
  character (len=100), allocatable :: SetLabel(:) ! set label
  character (len=100), allocatable :: ExpLabel(:) ! export label
  character (len=100), allocatable :: ImpLabel(:) ! import label
  
  END TYPE ESM_CplSet
  
  TYPE (ESM_CplSet), allocatable, target :: COUPLED(:)

• Import and export fields metadata information.
  TYPE, PRIVATE :: ESM_Field
  
  logical :: connected ! connected to coupler
  logical :: debug_write ! write exchanged field
  
  integer :: gtype  ! field grid mesh type
  integer :: Tindex ! rolling two-time indices
  
  character (len=:), allocatable :: short_name ! short name
  character (len=:), allocatable :: standard_name ! standard name
  character (len=:), allocatable :: long_name ! long name
  character (len=:), allocatable :: dst_gtype ! DST grid type
  character (len=:), allocatable :: dst_units ! DST units
  character (len=:), allocatable :: src_gtype ! SRC grid type
  character (len=:), allocatable :: src_units ! SRC units
  character (len=:), allocatable :: nc_vname ! DATA Vname
  character (len=:), allocatable :: nc_tname ! DATA Tname
  character (len=:), allocatable :: map_norm ! mapping norm
  character (len=:), allocatable :: map_type ! regrid method
  
  character (len=22)  :: DateString(2) ! snapshots date
  
  real (dp) :: scale_factor ! field scale factor
  real (dp) :: add_offset ! field add offset value
  real (dp) :: Tmin ! DATA time minimum value
  real (dp) :: Tmax ! DATA time maximum value
  real (dp) :: Tstr ! DATA lower time-snapshot
  real (dp) :: Tend ! DATA upper time-snapshot
  real (dp) :: Tintrp(2) ! interpolation time (day)
  real (dp) :: Vtime(2) ! latest two-time values
  
  TYPE (ESMF_RouteHandle) :: rhandle  ! field RouteHandle
  
  END TYPE ESM_Field

• Import and export fields mesh data.
  TYPE, PRIVATE :: ESM_Mesh
  
  integer :: gid ! grid ID
  integer :: gtype ! grid mesh type
  
  integer (i4b), allocatable :: mask(:,:) ! grid land/sea mask
  
  real (r8), allocatable :: lon(:,:) ! grid longitude
  real (r8), allocatable :: lat(:,:) ! grid latitude
  real (r8), allocatable :: area(:,:) ! grid area
  
  END TYPE ESM_Mesh

• Coupled models high-level object, [Nmodels=1].
  TYPE, PRIVATE :: ESM_Model
  
  logical :: IsActive ! active for coupling
  
  integer (i4b) :: LandValue ! land mask value
  integer (i4b) :: SeaValue ! sea mask value
  
  integer :: Ngrids ! number nested grids
  
  integer :: ExportCalls ! export CALL counter
  integer :: ImportCalls ! import CALL counter
  
  integer :: nPETs ! number model PETs
  integer, allocatable :: PETlist(:) ! component PETs list
  
  integer, allocatable :: TimeFrac(:,:) ! driver time fraction
  
  character (len=:), allocatable :: name ! component name
  
  TYPE (ESMF_Grid), allocatable :: grid(:) ! grid object
  TYPE (ESM_Mesh), allocatable :: mesh(:) ! mesh
  TYPE (ESM_Field), allocatable :: ImportField(:) ! import fields
  TYPE (ESM_Field), allocatable :: ExportField(:) ! export fields
  TYPE (ESMF_State), allocatable :: ImportState(:) ! import state
  TYPE (ESMF_State), allocatable :: ExportState(:) ! export state
  
  END TYPE ESM_Model
  
  TYPE (ESM_Model), allocatable, target :: MODELS(:)

• Internal module parameters and variables:
  
  • Switch to trace/track run sequence during debugging. All information is written to Fortran unit trac. For now, use standard output unit.
    logical :: ESM_track = .TRUE. ! trace/track CALL sequence switch
  • Number of coupled ESM gridded components and identification index.
    integer, parameter :: Nmodels = 1
    integer, parameter :: Idriver = 0
    integer, parameter :: Iroms = 1
  • Number of ROMS export and import fields per component.
    integer, allocatable :: Nexport(:)
    integer, allocatable :: Nimport(:)
  • Model coupling type: [1] Explicit, [otherwise] Semi-Implicit. In explicit coupling, exchange fields at the next time-step are defined using known values from the time-step before it. Explicit methods require less computational effort and are accurate for small coupling time-steps. In implicit coupling, exchange fields at the next time-step are defined by including values at the next time-step. Implicit methods are stable and allow longer coupling time-steps but are expensier. In semi-implicit coupling, ROMS -> ATM is explicit, ATM -> ROMS is implicit.
    integer :: CouplingType = 1
  • Linked/coupled ROMS nested grid number.
    integer :: linked_grid
  • Distributed-memory communicator handle for each component, rank of each PET, and PET layout (sequential or concurrent).
    integer, allocatable :: ESMcomm(:)
    integer :: PETrank
    character (len=10), allocatable :: PETlayoutOption(:)
  • Coupling debugging flag: [0] no debugging, [1] reports informative messages, or [2] '1' plus writes grid information in VTK format, [3] '2' plus writes exchage fields into NetCDF files.
    integer :: DebugLevel = 0
  • Execution tracing level flag: [0] no tracing, [1] reports sequence of coupling subroutine calls, or [2] <1> plus writes voluminous ESMF library tracing information which slowdown performace, and creates large log file.
    integer :: TraceLevel = 0
  • Standard output units.
    integer :: cplout = 77 ! coupling driver
    integer :: trac = 6 ! trace/track CALL sequence unit
  • Coupled model staggered grid-cell type indices.
    integer, parameter :: Inan = 0 ! unstaggered, cell center
    integer, parameter :: Icenter = 1 ! cell center
    integer, parameter :: Icorner = 2 ! cell corners
    integer, parameter :: Iupoint = 3 ! right and left cell faces
    integer, parameter :: Ivpoint = 4 ! upper and lower cell faces
    
    character (len=6), dimension(0:4) :: GridType = &
    & (/ 'N/A ', &
    & 'Center', &
    & 'Corner', &
    & 'U ', &
    & 'V ' /)
  • Standard input filename for each coupled model, [Nmodels].
    character (len=256), allocatable :: INPname(:)
  • ROMS coupling YAML configuration filename.
    character (len=:), allocatable :: CPLname
  • ESM strings.
    character (len=:), allocatable :: CoupledSet
    character (len=:), allocatable :: ExportStateName
    character (len=:), allocatable :: ImportStateName
  • ESM constants.
    integer (i4b), parameter :: MAPPED_MASK = 99_i4b
    integer (i4b), parameter :: UNMAPPED_MASK = 98_i4b
    
    real (dp), parameter :: MISSING_dp = 1.0E20_dp
    real (r4), parameter :: MISSING_r4 = 1.0E20_r4
    real (r8), parameter :: MISSING_r8 = 1.0E20_r8
    
    real (dp), parameter :: TOL_dp = 0.5E20_dP
    real (r4), parameter :: TOL_r4 = 0.5E20_r4
    real (r8), parameter :: TOL_r8 = 0.5E20_r8

UFS Interface

The NUOPC capColored text here file is connected to the UFS.F90 and EARTH_GRID_COMP.F90 with C-preprocessing option FRONT_ROMS as follows:

MODULE module_EARTH_GRID_COMP

...

#ifdef FRONT_ROMS
USE cmeps_roms_mod, ONLY : ROMS_SS => ROMS_SetServices
#endif

...

END MODULE module_EARTH_GRID_COMP

Module Subroutines

The ROMS cap module contains a set of subroutines that are required by NUOPC. These subroutines are called by the NUOPC infrastructure according to a predefined calling sequence. Some subroutines are called during the initialization of the coupled system, some during the run of the coupled system, and some during the finalization of the coupled system.

The initialization sequence is the most complex and is governed by the NUOPC technical rules. Details about the initialization sequence can be found in the NUOPC Reference Manual. The ROMS cap requires ESMF version 8 or higher.

ROMS_SetServices	Entry point to the ROMS cap and the only public routine. It sets the ROMS component shared-object entry points for using NUOPC generic methods for initialize, run, and finalize.
ROMS_Create	Allocates module internal structures and process configuration from YAML file: roms_cmeps.yaml.
ROMS_SetInitializeP1	ROMS component phase 1 initialization which sets import and export fields long and short names into its respective state.
ROMS_SetInitializeP2	ROMS component phase 2 initialization which initializes the ROMS component (ROMS_Initialize), sets component grid (ROMS_SetGridArrays), and adds fields into import and export into respective states.
ROMS_DataInit	Exports ROMS component fields during initialization or restart.
ROMS_SetClock	Sets ROMS component date calendar, start and stop times, and coupling interval.
ROMS_SetRunClock	Sets ROMS run clock manually to avoid getting zero time stamps at the first regridding call.
ROMS_CheckImport	Checks if ROMS component import field is at the correct time.
ROMS_SetGridArrays	Sets ROMS component staggered, horizontal grid arrays, grid area, and land/sea mask if any.
ROMS_SetStates	Adds ROMS component export and import fields into its respective state.
ROMS_ModelAdvance	Advances ROMS component for a coupling interval. It calls ROMS_Import and ROMS_Export routines.
ROMS_SetFinalize	Finalizes ROMS component execution.
ROMS_Import	Loads import fields into ROMS internal kernel arrays.
ROMS_Export	Exports ROMS fields to other gridded components.
Report_TimeStamp	Reports coupling time-stamp.