id summary reporter owner description type status priority milestone component version resolution keywords cc 825 VERY IMPORTANT: Added multi-file capability to adjoint-based drivers arango "'''There is a lot of information in this update, please read carefully and understand if you use any of the algorithms mentioned below.''' ---- This update is crucial because it includes the capability allows the forward nonlinear solution to be time split into multiple files to avoid creating huge NetCDF files in applications with large grids. Recall that the nonlinear trajectory is used to linearize the tangent linear and adjoint models used in 4D-Var data assimilation, observation impacts, observation sensitivities, adjoint sensitivities, generalized stability analysis, and other adjoint-based algorithms. * Two subroutines ('''edit_multifile''' and '''edit_file_struct''') are included in '''edit_multifile.F''' to manipulate the basic state forward trajectory used in various adjoint-based drivers. The '''TYPE(T_IO)''' structure can be deallocated and allocate with multiple file capabilities if so desired. For example in '''edit_multifile''', we have: {{{ DO ng=1,Ngrids ! SELECT CASE (TRIM(ADJUSTL(task))) ! ! Load HIS information into the FWD structure so it can be used to ! process the NLM background trajectory by the ADM and TLM kernels. ! CASE ('HIS2FWD') IF (ndefHIS(ng).gt.0) THEN IF (HIS(ng)%ncid.ne.-1) THEN CALL netcdf_close (ng, iNLM, HIS(ng)%ncid) END IF Nfiles=ntimes(ng)/ndefHIS(ng) IF (nHIS(ng).eq.ndefHIS(ng)) Nfiles=Nfiles+1 CALL edit_file_struct (ng, Nfiles, FWD) DO ifile=1,Nfiles FWD(ng)%files(ifile)=TRIM(HIS(ng)%files(ifile)) END DO FWD(ng)%name=TRIM(FWD(ng)%files(1)) ELSE FWD(ng)%ncid=HIS(ng)%ncid FWD(ng)%name=TRIM(HIS(ng)%name) FWD(ng)%files(1)=TRIM(HIS(ng)%name) END IF CASE ('HIS2BLK') ... CASE ('FWD2BLK') ... CASE ('FWD2HIS') ... CASE ('FCTA2FWD') ... CASE ('FCTA2BLK') ... CASE ('FCTB2FWD') ... CASE ('FCTB2BLK') ... CASE ('TLM2FWD') ... END SELECT END DO }}} As you can see, there are several possibilities in the manipulations between file structures. In the calling driver, we need to add the specific instructions for such handling. For example in '''w4dpsas_ocean.h''', we have: {{{ ! ! Set structure for the nonlinear forward trajectory to be processed ! by the tangent linear and adjoint models. Also, set switches to ! process the FWD structure in routine ""check_multifile"". Notice that ! it is possible to split solution into multiple NetCDF files to reduce ! their size. ! CALL edit_multifile ('HIS2FWD') IF (FoundError(exit_flag, NoError, __LINE__, & & __FILE__)) RETURN DO ng=1,Ngrids LreadFWD(ng)=.TRUE. END DO #if defined BULK_FLUXES && defined NL_BULK_FLUXES ! ! Set structure for the nonlinear surface fluxes to be processed by ! by the tangent linear and adjoint models. Also, set switches to ! process the BLK structure in routine ""check_multifile"". Notice that ! it is possible to split solution into multiple NetCDF files to reduce ! their size. ! CALL edit_multifile ('HIS2BLK') IF (FoundError(exit_flag, NoError, __LINE__, & & __FILE__)) RETURN DO ng=1,Ngrids LreadBLK(ng)=.TRUE. END DO #endif }}} * The routine '''check_multifile''' was updated to include more '''TYPE(T_IO)''' structures to process for information about multiple files or not. Two new generic subroutines '''multifile_info_s1d''' and '''multifile_info_s2d''' where written to inquire about 1D and 2D''' TYPE(T_IO)''' structures, repectively. They are called in a compact way in '''check_multifile''': {{{ #ifdef FORWARD_READ ! !----------------------------------------------------------------------- ! Nonlinear model forward trajectory data. !----------------------------------------------------------------------- ! IF (LreadFWD(ng)) THEN CALL multifile_info_s1d (ng, model, 'Forward', FWD) IF (FoundError(exit_flag, NoError, __LINE__, & & __FILE__)) RETURN END IF #endif ... #ifdef FRC_FILE ! !----------------------------------------------------------------------- ! Input forcing data. !----------------------------------------------------------------------- ! file_type='Forcing' max_files=MAXVAL(nFfiles) CALL multifile_info_s2d (ng, model, file_type, nFfiles, & & max_files, FRC) IF (FoundError(exit_flag, NoError, __LINE__, & & __FILE__)) RETURN #endif }}} * Several switches ('''LreadADM''', '''LreadBLK''', '''LreadFWD''', and '''LreadTLM''') are introduced in '''mod_scalar.F''' to activate the processing of information about file structures at the right time in the algorithms. * The '''TYPE(T_IO)''' file structure definition in '''mod_param.F''' now includes the '''head''' field, which remains unchanged after initialization and has the I/O heading prefix filename. It is used as follows: {{{ WRITE (HIS(ng)%name,10) TRIM(FWD(ng)%head), outer lstr=LEN_TRIM(HIS(ng)%name) HIS(ng)%base=HIS(ng)%name(1:lstr-3) }}} The '''HIS(ng)%base''' is used in output.F to create the multile files. For example, in 4D-Var we can have: {{{ wc13_fwd_outer0_0001.nc wc13_fwd_outer0_0002.nc wc13_fwd_outer0_0003.nc wc13_fwd_outer0_0004.nc wc13_fwd_outer1_0001.nc wc13_fwd_outer1_0002.nc wc13_fwd_outer1_0003.nc wc13_fwd_outer1_0004.nc }}} Here, the basic state forward trajectory is split in daily files having data records every two-hours in a 4D-Var 4-days data assimilation cycle for each outer loop. '''WARNING''': If not using output history multi-files, '''NDEFHIS=0''' in '''roms.in''', we will have {{{ wc13_fwd_outer0.nc instead of wc13_fwd_0000.nc wc13_fwd_outer1.nc instead of wc13_fwd_0001.nc }}} as in previous versions of the code. So you need to update your post-processing scripts. * Several files were updated to prevent having '''Too many open files''' error. In the '''UNIX''' environment, there is a limit to the number of open files during program execution. Use the commands: {{{ % ulimit -a % ulimit -S -n }}} to check the limit in your system. Usually, 256 files can be opened by default. A new C-preprocessing option ('''CHECK_OPEN_FILES''') is introduced to report the number of files created, opened, and closed for an application. The report is written to Fortran file '''fort.1000'''. One can easily check that report by typing: {{{ % grep CREATE fort.1000 | wc -l % grep OPEN fort.1000 | wc -l % grep CLOSE fort.1000 | wc -l }}} or using provided scripts '''check_openfiles.bash''' or '''check_openfiles.sh''' located in '''ROMS/Bin'''. * Use the multiple filename entries in roms.in and end with the verical bar (|) symbol. For example: {{{ FWDNAME == wc13_fwd_outer0_0001.nc | wc13_fwd_outer0_0002.nc | wc13_fwd_outer0_0003.nc | wc13_fwd_outer0_0004.nc FWDNAME == wc13_his_0001.nc | wc13_his_0002.nc | wc13_his_0003.nc | wc13_his_0004.nc FCTnameA == wc13_fct_A_0001.nc | wc13_fct_A_0002.nc | wc13_fct_A_0003.nc | wc13_fct_A_0004.nc | wc13_fct_A_0005.nc | wc13_fct_A_0006.nc | wc13_fct_A_0007.nc FCTnameB == wc13_fct_B_0001.nc | wc13_fct_B_0002.nc | wc13_fct_B_0003.nc | wc13_fct_B_0004.nc | wc13_fct_B_0005.nc | wc13_fct_B_0006.nc | wc13_fct_B_0007.nc }}} if appropriate." upgrade new major Release ROMS/TOMS 3.8 Adjoint 3.8