id summary reporter owner description type status priority milestone component version resolution keywords cc 769 VERY IMPORTANT Update: Everyone needs to read provide information arango "This update contains critical information, and I recommend everybody to consider updating their version: * While coding and debugging the tangent linear and adjoint of the nesting algorithms, we discovered a two-way bug in the nonlinear model code in telescoping nested applications. Recall that in ROMS, a '''telescoping''' grid is a refined grid containing another refined grid inside. For Example: [[Image(https://www.myroms.org/trac/multi-refinement_nd.png, 600)]] In the above diagram, grid '''2''' is the only telescoping grid in this configuration. We need to have a two-way transfer of information from '''4''' to '''2''' and '''2''' to '''1''' for grid '''2''' to be considered a telescoping type. The coaser grid '''1''' is not considered a telescoping grid by definition in ROMS. In a three-grid refinenemt application for the US east coast, we noticed unnecessary two-way exchanges between telescoping grid '''2''' and coarser grid '''1'''. See symbol '''>>>>''' print in the standard output below: {{{ NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 000000000001 - 000000007200) NL ROMS/TOMS: started time-stepping: (Grid: 02 TimeSteps: 000000000001 - 000000021600) NL ROMS/TOMS: started time-stepping: (Grid: 03 TimeSteps: 000000000001 - 000000043200) TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME Grid C => (i,j,k) Cu Cv Cw Max Speed 0 2014-01-01 00:00:00.00 2.479705E-02 1.895958E+04 1.895961E+04 2.057379E+15 01 (128,010,40) 6.586887E-02 8.830755E-02 0.000000E+00 2.146392E+00 0 2014-01-01 00:00:00.00 9.826892E-03 1.355795E+04 1.355796E+04 2.279106E+14 02 (067,001,40) 6.153040E-02 3.379189E-02 0.000000E+00 1.549213E+00 0 2014-01-01 00:00:00.00 4.571369E-03 1.116663E+04 1.116663E+04 4.813844E+13 03 (170,008,29) 2.109439E-02 2.133386E-02 0.000000E+00 3.891363E-01 1 2014-01-01 00:01:00.00 4.569719E-03 1.116663E+04 1.116663E+04 4.813848E+13 03 (224,073,01) 1.516512E-02 1.260101E-02 8.206179E-02 3.891982E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 1 2014-01-01 00:02:00.00 9.831911E-03 1.355797E+04 1.355798E+04 2.279109E+14 02 (135,136,01) 5.762565E-03 1.577325E-04 2.528599E-01 1.547482E+00 2 2014-01-01 00:02:00.00 4.574353E-03 1.116663E+04 1.116664E+04 4.813852E+13 03 (222,074,37) 2.382373E-02 1.525561E-02 1.121914E-01 3.892563E-01 3 2014-01-01 00:03:00.00 4.576449E-03 1.116665E+04 1.116665E+04 4.813859E+13 03 (233,065,40) 1.694378E-02 6.053802E-03 1.822713E-01 3.886253E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 2 2014-01-01 00:04:00.00 9.848035E-03 1.355799E+04 1.355800E+04 2.279112E+14 02 (134,137,40) 2.929985E-03 7.659153E-04 6.330313E-01 1.548821E+00 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 4 2014-01-01 00:04:00.00 4.576183E-03 1.116666E+04 1.116667E+04 4.813865E+13 03 (237,066,40) 1.547985E-02 5.251661E-03 2.077931E-01 3.886372E-01 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 5 2014-01-01 00:05:00.00 4.578296E-03 1.116668E+04 1.116669E+04 4.813873E+13 03 (240,066,40) 1.518760E-02 4.659222E-03 1.958341E-01 3.885507E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 1 2014-01-01 00:06:00.00 2.472506E-02 1.895960E+04 1.895962E+04 2.057381E+15 01 (031,036,01) 2.730114E-02 1.131899E-02 2.988898E-01 2.164003E+00 3 2014-01-01 00:06:00.00 9.864558E-03 1.355800E+04 1.355801E+04 2.279115E+14 02 (136,140,40) 2.016993E-03 3.375310E-03 5.526394E-01 1.550218E+00 6 2014-01-01 00:06:00.00 4.583192E-03 1.116670E+04 1.116670E+04 4.813881E+13 03 (242,068,40) 1.469385E-02 4.240571E-03 1.770060E-01 3.886908E-01 7 2014-01-01 00:07:00.00 4.589692E-03 1.116671E+04 1.116672E+04 4.813888E+13 03 (214,082,40) 1.952714E-02 3.661358E-03 1.516865E-01 3.892088E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 4 2014-01-01 00:08:00.00 9.875476E-03 1.355809E+04 1.355810E+04 2.279128E+14 02 (136,140,40) 2.392311E-03 2.585103E-03 5.107434E-01 1.552007E+00 8 2014-01-01 00:08:00.00 4.596425E-03 1.116673E+04 1.116673E+04 4.813895E+13 03 (216,085,40) 1.761177E-02 3.653321E-03 1.553607E-01 3.887313E-01 9 2014-01-01 00:09:00.00 4.603149E-03 1.116675E+04 1.116676E+04 4.813905E+13 03 (250,064,40) 1.535948E-02 1.897569E-03 1.583301E-01 3.885450E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 5 2014-01-01 00:10:00.00 9.884353E-03 1.355818E+04 1.355819E+04 2.279141E+14 02 (136,141,40) 3.595284E-03 2.194098E-03 4.981900E-01 1.553226E+00 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 10 2014-01-01 00:10:00.00 4.605400E-03 1.116677E+04 1.116678E+04 4.813915E+13 03 (253,064,40) 1.487150E-02 1.964936E-03 1.542588E-01 3.883007E-01 >>>> FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 11 2014-01-01 00:11:00.00 4.605182E-03 1.116681E+04 1.116681E+04 4.813930E+13 03 (255,065,40) 1.409242E-02 1.498900E-03 1.489003E-01 3.873421E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 2 2014-01-01 00:12:00.00 2.470726E-02 1.895960E+04 1.895962E+04 2.057385E+15 01 (026,035,40) 3.741860E-02 1.691325E-03 5.639773E-01 2.167463E+00 }}} Therefore, for each timestep of grid '''1''' there are three two-way exchanges from grid '''2''' to '''1''' in the contact region '''cr = 2'''. These redundant exchanges are not a problem in the nonlinear model because grid '''1''' is waiting for grids '''2''' and '''3''' to to reach its current time. They are fatal in the adjoint model. The two-way exchanges are expensive in distributed-memory parallel condigurations because it involves MPI communication that slow the solution affecting performace. However, the bug is that the two-way exchange between grid '''2''' and '''1''' is missing the last piece from grid '''3'''. It needs to happen after the exchange between '''3''' and '''2''', so grid '''1''' can advance another timestep (see '''++++'''). This bug does not seem to change the solution that much but it still is wrong. We need to have instead: {{{ NL ROMS/TOMS: started time-stepping: (Grid: 01 TimeSteps: 000000000001 - 000000007200) NL ROMS/TOMS: started time-stepping: (Grid: 02 TimeSteps: 000000000001 - 000000021600) NL ROMS/TOMS: started time-stepping: (Grid: 03 TimeSteps: 000000000001 - 000000043200) TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME Grid C => (i,j,k) Cu Cv Cw Max Speed 0 2014-01-01 00:00:00.00 2.479705E-02 1.895958E+04 1.895961E+04 2.057379E+15 01 (128,010,40) 6.586887E-02 8.830755E-02 0.000000E+00 2.146392E+00 0 2014-01-01 00:00:00.00 9.826892E-03 1.355795E+04 1.355796E+04 2.279106E+14 02 (067,001,40) 6.153040E-02 3.379189E-02 0.000000E+00 1.549213E+00 0 2014-01-01 00:00:00.00 4.571369E-03 1.116663E+04 1.116663E+04 4.813844E+13 03 (170,008,29) 2.109439E-02 2.133386E-02 0.000000E+00 3.891363E-01 1 2014-01-01 00:01:00.00 4.569719E-03 1.116663E+04 1.116663E+04 4.813848E+13 03 (224,073,01) 1.516512E-02 1.260101E-02 8.206179E-02 3.891982E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 1 2014-01-01 00:02:00.00 9.831911E-03 1.355797E+04 1.355798E+04 2.279109E+14 02 (135,136,01) 5.762565E-03 1.577325E-04 2.528599E-01 1.547482E+00 2 2014-01-01 00:02:00.00 4.574353E-03 1.116663E+04 1.116664E+04 4.813852E+13 03 (222,074,37) 2.382373E-02 1.525561E-02 1.121914E-01 3.892563E-01 3 2014-01-01 00:03:00.00 4.576449E-03 1.116665E+04 1.116665E+04 4.813859E+13 03 (233,065,40) 1.694378E-02 6.053802E-03 1.822713E-01 3.886253E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 2 2014-01-01 00:04:00.00 9.848035E-03 1.355799E+04 1.355800E+04 2.279112E+14 02 (134,137,40) 2.929985E-03 7.659153E-04 6.330313E-01 1.548821E+00 4 2014-01-01 00:04:00.00 4.576183E-03 1.116666E+04 1.116667E+04 4.813865E+13 03 (237,066,40) 1.547985E-02 5.251661E-03 2.077931E-01 3.886372E-01 5 2014-01-01 00:05:00.00 4.578296E-03 1.116668E+04 1.116669E+04 4.813873E+13 03 (240,066,40) 1.518760E-02 4.659222E-03 1.958341E-01 3.885507E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 ++++ FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 1 2014-01-01 00:06:00.00 2.472503E-02 1.895960E+04 1.895962E+04 2.057381E+15 01 (031,036,01) 2.730114E-02 1.131899E-02 2.988898E-01 2.164003E+00 3 2014-01-01 00:06:00.00 9.864558E-03 1.355800E+04 1.355801E+04 2.279115E+14 02 (136,140,40) 2.016993E-03 3.375310E-03 5.526394E-01 1.550218E+00 6 2014-01-01 00:06:00.00 4.583192E-03 1.116670E+04 1.116670E+04 4.813881E+13 03 (242,068,40) 1.469385E-02 4.240571E-03 1.770060E-01 3.886908E-01 7 2014-01-01 00:07:00.00 4.589692E-03 1.116671E+04 1.116672E+04 4.813888E+13 03 (214,082,40) 1.952714E-02 3.661358E-03 1.516865E-01 3.892088E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 4 2014-01-01 00:08:00.00 9.875476E-03 1.355809E+04 1.355810E+04 2.279128E+14 02 (136,140,40) 2.392311E-03 2.585103E-03 5.107434E-01 1.552007E+00 8 2014-01-01 00:08:00.00 4.596425E-03 1.116673E+04 1.116673E+04 4.813895E+13 03 (216,085,40) 1.761177E-02 3.653321E-03 1.553607E-01 3.887313E-01 9 2014-01-01 00:09:00.00 4.603149E-03 1.116675E+04 1.116676E+04 4.813905E+13 03 (250,064,40) 1.535948E-02 1.897569E-03 1.583301E-01 3.885450E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 5 2014-01-01 00:10:00.00 9.884353E-03 1.355818E+04 1.355819E+04 2.279141E+14 02 (136,141,40) 3.595284E-03 2.194098E-03 4.981900E-01 1.553226E+00 10 2014-01-01 00:10:00.00 4.605400E-03 1.116677E+04 1.116678E+04 4.813915E+13 03 (253,064,40) 1.487150E-02 1.964936E-03 1.542588E-01 3.883007E-01 11 2014-01-01 00:11:00.00 4.605182E-03 1.116681E+04 1.116681E+04 4.813930E+13 03 (255,065,40) 1.409242E-02 1.498900E-03 1.489003E-01 3.873421E-01 FINE2COARSE - exchanging data between grids: dg = 03 and rg = 02 at cr = 04 ++++ FINE2COARSE - exchanging data between grids: dg = 02 and rg = 01 at cr = 02 2 2014-01-01 00:12:00.00 2.470724E-02 1.895960E+04 1.895962E+04 2.057385E+15 01 }}} Once that this bug is corrected, telescoping applications run faster. The gain in efficiency depends on the applicatuion and the number of processors used. We have observed improvements between 6-10 percent. '''Therefore, this is a critical update for users running nested applications.''' * We also found that additional improvements (around 8 percent) in ROMS solutions compiled with '''ifort''' when the option '''-heap_arrays''' is removed. However, we need to set the '''stacksize''' option to a large value in some computers: {{{ FFLAGS += -Wl,-stack_size,0x64000000 }}} or set the environmental variable '''stacksize''' to unlimited in the login script. For example, I have the following command in my '''.tcshrc''': {{{ limit stacksize unlimited }}} If I type the '''limit''' UNIX command on a Linux cluster, I get: {{{ % limit cputime unlimited filesize unlimited datasize unlimited stacksize unlimited coredumpsize 0 kbytes memoryuse unlimited vmemoryuse unlimited descriptors 1024 memorylocked unlimited maxproc 1024 }}} ROMS has lots of authomatic arrays, so one has the option to allocate those arrays on '''heap''' or '''stack'''. Usually, the '''stack''' option is faster but we need to have enough of it. Otherwise, ROMS will blow-up because memory corruption. * Corrected the reporting of the longitude and latitude ranges at RHO-points in '''get_grid.F'''. Many thanks to John Warner for bringing this to our attention. * Added MPI broadcasting of '''2D''' and '''3D''' string arrays in '''distribute.F'''. Now, we have the following interface for '''mp_bcasts''': {{{ INTERFACE mp_bcasts MODULE PROCEDURE mp_bcasts_0d MODULE PROCEDURE mp_bcasts_1d MODULE PROCEDURE mp_bcasts_2d MODULE PROCEDURE mp_bcasts_3d END INTERFACE mp_bcasts }}} * Added the reading and writting of generic 2D and 3D strigs to NetCDF files. The mod_netcdf.F now have the following updated intefaces for '''netcdf_get_svar''' and '''netcdf_put_svar''': {{{ INTERFACE netcdf_get_svar MODULE PROCEDURE netcdf_get_svar_0d MODULE PROCEDURE netcdf_get_svar_1d MODULE PROCEDURE netcdf_get_svar_2d MODULE PROCEDURE netcdf_get_svar_3d END INTERFACE netcdf_get_svar INTERFACE netcdf_put_fvar INTERFACE netcdf_put_svar MODULE PROCEDURE netcdf_put_svar_0d MODULE PROCEDURE netcdf_put_svar_1d MODULE PROCEDURE netcdf_put_svar_2d MODULE PROCEDURE netcdf_put_svar_3d END INTERFACE netcdf_put_svar }}} The reasons for this update will be obvious in the future. * Added new C-preprocessing option '''IMPLICIT_NUDGING''' to the momentum radiation boundary conditions in '''u2dbc_im.F''', '''v2dbc_im.F''', '''u3dbc_im.F''', and '''v3dbc_im.F'''. The implicit treatment of the nudging term in the radiation equation is more stable but one need to be sure that the land/sea masking does not have one-point bays. Many thanks to Alistar and Kate for suggesting this option." upgrade closed major Release ROMS/TOMS 3.7 Nonlinear 3.7 Done