From WikiROMS
Jump to navigationJump to search
ROMS Stand-Alone ESMF/NUOPC Cap for the UFS


This document describes the stand-alone ROMS ESMF/NUOPC cap module to be used by third-party coupling frameworks, like the Unified Forecast System (UFS). It is a lightweight software layer on top of ROMS that can be used by NUOPC-based packages (CMEPS/CDEPS, NEMS, and others) to couple to other Earth System Models (ESMs). Detailed information about ESMF/NUOPC and its implementation in ROMS can be found on the Earth System Modeling Framework WikiROMS page.

A NUOPC cap is a Fortran module that serves as the interface to a model when it's used in a NUOPC-based coupled system. The term cap is used because it is a lightweight software layer that sits on top of model code, making calls to it and exposing model data structures in a standard way.


The cmeps_roms.F module declares several derived-type structures to facilitate the management of all internal objects and variables:

  • ESM coupling time managing variables and ESMF objects.
    TYPE, PRIVATE :: ESM_Clock

    logical :: Restarted

    integer (i8b) :: AdvanceCount ! advance counter

    real (dp) :: Current_Time ! seconds
    real (dp) :: Time_Reference ! seconds
    real (dp) :: Time_Restart ! seconds
    real (dp) :: Time_Start ! seconds
    real (dp) :: Time_Stop ! seconds
    real (dp) :: Time_Step ! seconds

    character (len=22) :: Name
    character (len=22) :: CalendarString ! 360_day, gregorian
    character (len=22) :: Time_ReferenceString
    character (len=22) :: Time_RestartString
    character (len=22) :: Time_StartString
    character (len=22) :: Time_StopString

    TYPE (ESMF_Calendar)  :: Calendar
    TYPE (ESMF_Clock)  :: Clock
    TYPE (ESMF_Direction_flag) :: Direction
    TYPE (ESMF_Time)  :: CurrentTime
    TYPE (ESMF_Time)  :: ReferenceTime
    TYPE (ESMF_Time)  :: RestartTime
    TYPE (ESMF_Time)  :: StartTime
    TYPE (ESMF_Time)  :: StopTime
    TYPE (ESMF_TimeInterval)  :: TimeStep

    END TYPE ESM_Clock

    TYPE (ESM_Clock), allocatable, target :: ClockInfo(:)
  • ESM coupled state sets. If appropriate, it includes the logic for connecting nested grids.

    logical, allocatable :: LinkedGrid(:,:) ! connected grid

    character (len=100), allocatable :: SetLabel(:) ! set label
    character (len=100), allocatable :: ExpLabel(:) ! export label
    character (len=100), allocatable :: ImpLabel(:) ! import label


    TYPE (ESM_CplSet), allocatable, target :: COUPLED(:)
  • Import and export fields metadata information.
    TYPE, PRIVATE :: ESM_Field

    logical :: connected ! connected to coupler
    logical :: debug_write ! write exchanged field

    integer :: gtype  ! field grid mesh type
    integer :: Tindex ! rolling two-time indices

    character (len=:), allocatable :: short_name ! short name
    character (len=:), allocatable :: standard_name ! standard name
    character (len=:), allocatable :: long_name ! long name
    character (len=:), allocatable :: dst_gtype ! DST grid type
    character (len=:), allocatable :: dst_units ! DST units
    character (len=:), allocatable :: src_gtype ! SRC grid type
    character (len=:), allocatable :: src_units ! SRC units
    character (len=:), allocatable :: nc_vname ! DATA Vname
    character (len=:), allocatable :: nc_tname ! DATA Tname
    character (len=:), allocatable :: map_norm ! mapping norm
    character (len=:), allocatable :: map_type ! regrid method

    character (len=22)  :: DateString(2) ! snapshots date

    real (dp) :: scale_factor ! field scale factor
    real (dp) :: add_offset ! field add offset value
    real (dp) :: Tmin ! DATA time minimum value
    real (dp) :: Tmax ! DATA time maximum value
    real (dp) :: Tstr ! DATA lower time-snapshot
    real (dp) :: Tend ! DATA upper time-snapshot
    real (dp) :: Tintrp(2) ! interpolation time (day)
    real (dp) :: Vtime(2) ! latest two-time values

    TYPE (ESMF_RouteHandle) :: rhandle  ! field RouteHandle

    END TYPE ESM_Field
  • Import and export fields mesh data.

    integer :: gid ! grid ID
    integer :: gtype ! grid mesh type

    integer (i4b), allocatable :: mask(:,:) ! grid land/sea mask

    real (r8), allocatable :: lon(:,:) ! grid longitude
    real (r8), allocatable :: lat(:,:) ! grid latitude
    real (r8), allocatable :: area(:,:) ! grid area

  • Coupled models high-level object, [Nmodels=1].
    TYPE, PRIVATE :: ESM_Model

    logical :: IsActive ! active for coupling

    integer (i4b) :: LandValue ! land mask value
    integer (i4b) :: SeaValue ! sea mask value

    integer :: Ngrids ! number nested grids

    integer :: ExportCalls ! export CALL counter
    integer :: ImportCalls ! import CALL counter

    integer :: nPETs ! number model PETs
    integer, allocatable :: PETlist(:) ! component PETs list

    integer, allocatable :: TimeFrac(:,:) ! driver time fraction

    character (len=:), allocatable :: name ! component name

    TYPE (ESMF_Grid), allocatable :: grid(:) ! grid object
    TYPE (ESM_Mesh), allocatable :: mesh(:) ! mesh
    TYPE (ESM_Field), allocatable :: ImportField(:) ! import fields
    TYPE (ESM_Field), allocatable :: ExportField(:) ! export fields
    TYPE (ESMF_State), allocatable :: ImportState(:) ! import state
    TYPE (ESMF_State), allocatable :: ExportState(:) ! export state

    END TYPE ESM_Model

    TYPE (ESM_Model), allocatable, target :: MODELS(:)
  • Internal module parameters and variables:
    • Switch to trace/track run sequence during debugging. All information is written to Fortran unit trac. For now, it uses the standard output unit.
      logical :: ESM_track = .TRUE. ! trace/track CALL sequence switch
    • Number of coupled ESM gridded components and identification index.
      integer, parameter :: Nmodels = 1
      integer, parameter :: Idriver = 0
      integer, parameter :: Iroms = 1
    • Number of ROMS export and import fields per component.
      integer, allocatable :: Nexport(:)
      integer, allocatable :: Nimport(:)
    • Model coupling type: [1] Explicit, [otherwise] Semi-Implicit. In explicit coupling, exchange fields at the next time-step are defined using known values from the time-step before it. Explicit methods require less computational effort and are accurate for small coupling time-steps. In implicit coupling, exchange fields at the next time-step are defined by including values at the next time-step. Implicit methods are stable and allow longer coupling time-steps but are more computationally expensive. In semi-implicit coupling, ROMS -> ATM is explicit, ATM -> ROMS is implicit.
      integer :: CouplingType = 1
    • Linked/coupled ROMS nested grid number.
      integer :: linked_grid
    • Distributed-memory communicator handle for each component, rank of each PET, and PET layout (sequential or concurrent).
      integer, allocatable :: ESMcomm(:)
      integer :: PETrank
      character (len=10), allocatable :: PETlayoutOption(:)
    • Coupling debugging flag: [0] no debugging, [1] reports informative messages, or [2] '1' plus writes grid information in VTK format, [3] '2' plus writes exchage fields into NetCDF files.
      integer :: DebugLevel = 0
    • Execution tracing level flag: [0] no tracing, [1] reports sequence of coupling subroutine calls, or [2] <1> plus writes voluminous ESMF library tracing information which slowdown performace, and creates large log file.
      integer :: TraceLevel = 0
    • Standard output units.
      integer :: cplout = 77 ! coupling driver
      integer :: trac = 6 ! trace/track CALL sequence unit
    • Coupled model staggered grid-cell type indices.
      integer, parameter :: Inan = 0 ! unstaggered, cell center
      integer, parameter :: Icenter = 1 ! cell center
      integer, parameter :: Icorner = 2 ! cell corners
      integer, parameter :: Iupoint = 3 ! right and left cell faces
      integer, parameter :: Ivpoint = 4 ! upper and lower cell faces

      character (len=6), dimension(0:4) :: GridType = &
      & (/ 'N/A ', &
      & 'Center', &
      & 'Corner', &
      & 'U ', &
      & 'V ' /)
    • Standard input filename for each coupled model, [Nmodels].
      character (len=256), allocatable :: INPname(:)
    • ROMS coupling YAML configuration filename.
      character (len=:), allocatable :: CPLname
    • ESM strings.
      character (len=:), allocatable :: CoupledSet
      character (len=:), allocatable :: ExportStateName
      character (len=:), allocatable :: ImportStateName
    • ESM single and double precision constants.
      integer (i4b), parameter :: MAPPED_MASK = 99_i4b
      integer (i4b), parameter :: UNMAPPED_MASK = 98_i4b

      real (dp), parameter :: MISSING_dp = 1.0E20_dp
      real (r4), parameter :: MISSING_r4 = 1.0E20_r4
      real (r8), parameter :: MISSING_r8 = 1.0E20_r8

      real (dp), parameter :: TOL_dp = 0.5E20_dp
      real (r4), parameter :: TOL_r4 = 0.5E20_r4
      real (r8), parameter :: TOL_r8 = 0.5E20_r8

UFS Interface

The NUOPC cap file is connected to the UFS.F90 and EARTH_GRID_COMP.F90 with C-preprocessing option FRONT_ROMS as follows:



USE cmeps_roms_mod, ONLY : ROMS_SS => ROMS_SetServices



Module Subroutines

The ROMS cap module contains a set of subroutines that are required by NUOPC. These subroutines are called by the NUOPC infrastructure according to a predefined calling sequence. Some subroutines are called during the initialization of the coupled system, some during the run of the coupled system, and some during the finalization of the coupled system.

The initialization sequence is the most complex and is governed by the NUOPC technical rules. Details about the initialization sequence can be found in the NUOPC Reference Manual. The ROMS cap requires ESMF version 8 or higher.

ROMS_SetServices Entry point to the ROMS cap and the only public routine. It sets the ROMS component shared-object entry points for using NUOPC generic methods for initialize, run, and finalize.
ROMS_Create Allocates module internal structures and processes configuration from YAML file: roms_cmeps.yaml.
ROMS_SetInitializeP1 ROMS component phase 1 initialization which sets import and export fields long and short names into its respective state.
ROMS_SetInitializeP2 ROMS component phase 2 initialization which initializes the ROMS component (ROMS_Initialize), sets component grid (ROMS_SetGridArrays), and adds fields into import and export into respective states.
ROMS_DataInit Exports ROMS component fields during initialization or restart.
ROMS_SetClock Sets ROMS component date calendar, start and stop times, and coupling interval.
ROMS_SetRunClock Sets ROMS run clock manually to avoid getting zero time stamps at the first regridding call.
ROMS_CheckImport Checks if ROMS component import field is at the correct time.
ROMS_SetGridArrays Sets ROMS component staggered, horizontal grid arrays, grid area, and land/sea mask if any.
ROMS_SetStates Adds ROMS component export and import fields into its respective state.
ROMS_ModelAdvance Advances ROMS component for a coupling interval. It calls ROMS_Import and ROMS_Export routines.
ROMS_SetFinalize Finalizes ROMS component execution.
ROMS_Import Loads import fields into ROMS internal kernel arrays.
ROMS_Export Exports ROMS fields to other gridded components.
ROMS_Rotate Rotates vector components from computational grid to geographical EAST and NORTH directions or vice versa.
Report_TimeStamp Reports coupling time-stamp.

Import and Export Fields

The export and import fields are configured and specified in the YAML file roms_cmeps.yaml. Check the following wikiROMS page for detailed information. The user has full control of the coupling exchange fields metadata.

Currently, CMEPS does not support coupling of nested grids. The ESMF/NUOPC library version 8.0.0 or higher supports native nesting capabilities. The NUOPC layer now allows coupling sets in the import and export states of the ESM components. A nested grid is represented as another ESMF_Grid object so that a coupled model can have a set of telescoping meshes of decreasing spatial resolution. The NUOPC connector can recognize the different grids and exchange fields to a connected ESM component. Various types of connections are possible, like nest-to-nest, fine-to-coarse, or coarse-to-fine. The user decides which nested grids to connect. The NUOPC_AddNestedState routine is used to advertise the nested couple sets.

For example, the NOUPC cap module routines ROMS_SetInitializeP1 can be modified easily to add the coupled sets to the ImportState and ExportState as shown below in the green blocks of code.

ImportCoupledStates.png ExportCoupledStates.png