id summary reporter owner description type status priority milestone component version resolution keywords cc 888 VERY IMPORTANT: Model Coupling with the ESMF/NUOPC Library arango "I have been developing the coupling with the Earth System Modeling Framework ('''ESMF''') library with the National Unified Operational Prediction Capability ('''NUOPC''') layer for a while. It was challenging to code the coupling infrastructure from scratch. It took lots of debugging. For more information about this development, check [https://www.myroms.org/wiki/Model_Coupling_ESMF WikiROMS]. We will continue expanding the online documentation. Currently, the '''ROMS''' coupling '''ESMF'''/'''NUOPC''' interface can be illustrated with the following diagram. [[Image(http://www.myroms.org/trac/ROMS_Coupling.png, center, 600)]] It includes four distinct compartments: '''Driver''', '''Model Components''', '''Coupler''', and '''Connectors'''. Righ now, we are considering five classes of coupled Earth System Model ('''ESM''') components: '''ROMS''', '''ATM''', '''DATA''', '''SEA-ICE''', and '''WAVE''' models. There is a Fortran module that sits on of each '''ESM''' component and referred as the [[span(style=color: #FF0000, '''NUOPC cap''')]] file. It is a Fortran code layer making calls to the '''ESM''' numerical kernel via the '''initialize''', '''run''', and '''finalize''' phases There is a separate [[span(style=color: #FF0000, '''cap''')]] file for each coupled '''ESM''' component, which, from the '''ROMS''' perspective, corresponds to the atmosphere, sea-ice, wave, data models. Therefore, it is an abstract block that allows '''ROMS''' to communicate and exchange data seamlessly within the '''ESMF'''/'''NUOPC''' framework. The '''ROMS''' and other '''ESM''' components grids may (and usually do) have a different geographical extent and horizontal resolution. So if, for example, the atmosphere component domain is larger than the '''ROMS''' domain, it is necessary to provide data from another source to the atmosphere model at all surface grid points outside the '''ROMS''' domain. Therefore, a '''DATA''' component and its [[span(style=color: #FF0000, '''cap''')]] file are also required for most applications. In this case, SST is exported from '''ROMS''' and '''DATA''' to the atmosphere component, which imports the regridded values and then melds both SST fields. The melding is computed as: [[br]] [[br]] [[span(style=color: #FF0000,'''SST^ATM^(:,:) = W^ROMS^(:,:) * SST^ROMS^(:,:) + W^DATA^(:,:) * SST^DATA^(:,:)''')]] [[br]] [[br]] where [[span(style=color: #FF0000, '''W^ROMS^(:,:)+ W^DATA^(:,:) = 1''')]] are the weight coefficients that warantee a smooth transition between the '''ROMS''' and '''DATA''' values, as illustrated below for the Hurricane Irene application. [[Image(http://www.myroms.org/trac/melding_weights.png, center, 600)]] These weights are computed with the script '''coupling/wrf_weights.m''' of the ROMS Matlab repository. The design of the '''ROMS''' coupling interface with the '''ESMF'''/'''NUOPC''' library allows both '''driver''' and '''component''' modes of operation: In the '''driver''' mode, '''esmf_driver.h''' is the main program that provides all the interfaces and logistics to couple to other '''ESM''' components. In addition, it provides '''NUOPC'''-based generic '''ESM''' component services, interaction between gridded components in terms of '''NUOPC''' cap files, connectors between components for the regridding of source and destination fields ('''esmf_coupler.h'''), input scrips, and coupling metadata management. In the '''component''' mode, the '''esmf_roms.F''' [[span(style=color: #FF0000, '''cap''')]] module is provided. It can be adapted and incorporated into other '''NUOPC'''-based coupling systems, like the NOAA Environmental Modeling System ('''NEMS''') or the Community Mediator for Earth Prediction Systems ('''CMEPS'''). All the '''ESMF'''/'''NUOCP''' source codes are located in the '''Master''' sub-directory: * [[span(style=color: #4000FF, '''mod_esmf_esm.F''')]]: It defines all the structures and variables for '''ESM''' coupling. It also includes several support routines. It is fully tested. * [[span(style=color: #4000FF, '''esmf_driver.h''')]]: Main program for the '''driver''' mode of operation. It initializes '''ESM''' coupling, creates '''ESM''' components, process '''ESM''' configuration, register '''ESM''' components, initialize '''ESM''' components, run '''ESM''' components, finalize '''ESM''' components, and finalize '''ESMF''' coupling. It is fully tested. * [[span(style=color: #4000FF, '''esmf_coupler.h''')]]: Computes, executes, and releases '''!RouteHandle''' connectors used for the regridding of source and destination fields. It is fully tested. * [[span(style=color: #4000FF, '''esmf_esm.F''')]]: Sets '''NUOPC''' generic methods, entry points for each coupled components connectors, sets system internal clock, and sets coupling '''!RunSequence'''. It is fully tested. * [[span(style=color: #4000FF, '''esmf_roms.F''')]]: '''ROMS''' [[span(style=color: #FF0000, '''cap''')]] file. Sets '''NUOPC''' generic methods for '''initialize''', '''run''', and '''finalize'''. Checks import fields, sets grid component object, sets import and export states, advances '''ROMS''' for a couple interval, and sets import and export routines for the exchange of fields with other '''ESM''' components. It is fully tested. * [[span(style=color: #4000FF, '''esmf_data.F''')]]: '''DATA''' [[span(style=color: #FF0000, '''cap''')]] file. It is used to provide data to '''ESM''' components uncoupled fields or to provide values at locations not available because of noncoincident grids. The data comes from specified input files, usually NetCDF files. It is fully tested. * [[span(style=color: #4000FF, '''esmf_atm.F''')]]: Generic module for the atmosphere model component. [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_atm_coamps.h''')]]: Coupled Ocean-Atmosphere Mesoscale Prediction System ('''COAMPS''', Hodur, 1997; Chen ''et al.'', 2014) [[span(style=color: #FF0000, '''cap''')]] file. It is fully tested in applications for the US West Coast and Indian Ocean. It can be used for both free and restricted NRL versions. [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_atm_regcm.h''')]]: ICTP Regional Climate Model ('''RegCM''', version 4.5 and up, Elguindi ''et al.'', 2014 ) [[span(style=color: #FF0000, '''cap''')]] file. '''RegCM''' is maintained by the International Centre for Theoretical Physics (ICTP), Trieste, Italy. It was built upon the NCAR-PSU Mesoscale Model version 4 (MM4; Dickinson ''et al.'', 1989; Giorgi, 1989). This [[span(style=color: #FF0000, '''cap''')]] file is based on '''RegESM''' (Turuncoglu and Sannino, 2016). [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_atm_wrf.h''')]]: Weather Research and Forecasting ('''WRF''', Skamarock ''et al.'', 2005) model [[span(style=color: #FF0000, '''cap''')]] file. It is fully tested in regional applications for the US West and East Coasts. '''WRF''' is maintained by a consortium of US agencies and institutions (NCAR, NCEP, FSL, AFWA, NRL, FAA, and University of Oklahoma). * [[span(style=color: #4000FF, '''esmf_ice.F''')]]: Generic module for the seaice model component. [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_ice_cice.h''')]]: Los Alamos Sea Ice ('''CICE''') model [[span(style=color: #FF0000, '''cap''')]] file. It is not fully tested. We need experts with applications to help us test this coupling component. * [[span(style=color: #4000FF, '''esmf_wav.F''')]]: Generic module for the wave model component. [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_wav_wam.h''')]]: ECMWF Wave Model ('''WAM''', WAMDI Group, 1998) [[span(style=color: #FF0000, '''cap''')]] file. This [[span(style=color: #FF0000, '''cap''')]] file is adapted from '''RegESM''' (Turuncoglu and Sannino, 2016). [[br]] [[br]] * [[span(style=color: #8000FF, '''esmf_wav_www3.h''')]]: Wavewatch III Model ('''WW3''', WW3DG 2019) [[span(style=color: #FF0000, '''cap''')]] file. It will be available in the near future. [[br]] [[br]] The '''ROMS''' [[span(style=color: #FF0000, '''cap''')]] file has the following routines: {{{ MODULE esmf_roms_mod ROMS_SetServices Sets entry points using NUPOC generic methods for initialize, run, and finalize kernel routines. ROMS_SetInitializeP1 Phase 1 initialization: advertise import and export fields long- and short names. ROMS_SetInitializeP2 Phase 2 initialization: Initializes solution, sets application grid arrays, and registers fields into import state and export state. ROMS_DataInit Exports fields during initialization or restart. ROMS_SetClock Sets calendar, start and stop times, and coupling interval. ROMS_SetRunClock Sets ROMS run clock manually. ROMS_CheckImport Checks if the import field is at the correct time. ROMS_SetGridArrays Sets staggered, horizontal grid arrays, grid area, and land/sea masks. ROMS_SetStates Realize fields into import and export states by allocating and initializing pointers. ROMS_ModelAdvance Advances kernel for a coupling interval and calls import and export routines. ROMS_SetFinalize Finalizes execution. ROMS_Import Imports fields from other connected components. ROMS_Export Exports fields to other connected components MODULE esmf_roms_mod }}} The '''ESMF''' [[span(style=color: #FF0000, '''!RunSequence''')]] configuration file sets how the '''ESM''' components are connected and coupled. All the components interact with the same or different coupling time step. Usually, the connector from '''ROMS''' to '''ATM''' is explicit, whereas the connector from '''ATM''' to '''ROMS''' is semi-implicit. Often, the timestep of the atmosphere kernel is smaller than that for the ocean. Therefore, it advantageous for the '''ATM''' model to export time-averaged fields over the coupling interval, which is the same as the '''ROMS''' timestep. It is semi-implicit because '''ROMS''' right-hand-side terms are forced with n+1/2 '''ATM''' fields because of the time-averaging. The following diagram shows the explicit and semi-implicit coupling for a '''DATA'''-'''ATM'''-'''ROMS''' system: [[Image(http://www.myroms.org/trac/DATA-ATM-ROMS_Semi-Implicit_Avg_Coupling.png, center, 600)]] For example, in the Hurricane Irene application, we have: {{{ runSeq:: @* # timeStep = wildcard (*), single time loop DATA -> WRF # DATA to WRF connector, step (1) DATA ROMS -> WRF # ROMS to WRF connector, step (2) WRF # step (3) WRF -> ROMS # WRF to ROMS connector, step (4) ROMS # step (5) @ :: }}} ---- The test repository is updated to include the '''Hurricane Irene''' Test Case. We have the following directory structure for that test: {{{ test/IRENE /Data /FRC /HyCOM /NRM /OBS /ROMS /STD /WPS /WRF /Coupling /Coupled_RBL4DVAR /Forward }}} [[span(style=color: #FF0000, '''WARNING:''')]] This is quite a complex test using several advanced features of '''ROMS'''. It is intended for expert users of '''ROMS'''. If you a '''ROMS''' beginner, I highly recommend staying away from this test. It would be best if you had lots of expertise to run and understand this system. There are '''Readme''' files with instructions to run the several flavors for this test case. ---- Many thanks to my collaborators John Wilkin, Julia Levin, Travis Miles, David Robertson, and Joseph Brodie at Rutgers for helping me with several aspects of the configuration and testing of this complex test case. Special thanks to David Robertson for his herculean task of fixing and modernizing '''COAMPS''' to compile with aggressive compiler flags for debugging. Also, for his software management skills to compile, link and maintain all these libraries. It requires patience, expertise, and experience! Many thanks to our colleagues at '''NRL'''-Monterrey for their help with '''COAMPS'''. Finally, many thanks to my colleague Ufuk Turuncoglu for his help with the '''ESMF'''/'''NUOPC''' coding strategies. This system is inspired by his '''RegESM''' system. This work was funded by the Office of Naval Research ('''ONR''') and the National Oceanic and Atmospheric Administration ('''NOAA''')." upgrade new major Release ROMS/TOMS 4.0 Nonlinear 3.9