I4DVAR hang with "<<<< Conjugate Gradient Algorithm >>>>"

Discussion about tangent linear and adjoint models, variational data assimilation, and other related issues.

Moderators: arango, robertson

Post Reply
Message
Author
briskbamboo
Posts: 20
Joined: Thu Aug 20, 2020 1:24 pm
Location: University of Tsukuba

I4DVAR hang with "<<<< Conjugate Gradient Algorithm >>>>"

#1 Unread post by briskbamboo »

I compiled and run the I4DVAR with all default cpp.h , input files ... etc. It run then hang with <<<< Conjugate Gradient Algorithm >>>>, the cpu usage still show up but no thing happened.
This is the last log

Code: Select all

 ADM: GET_STATE - Reading previous adjoint state solution,                2004-01-03 00:00:00.00
                   (Grid 01, Outer=0001, t = 13008.0000, File: wc13_adj_outer1.nc, Rec=0001, Index=1)
                - free-surface
                   (Min = -9.36331275E-05 Max =  3.75458121E-05)
                - free-surface, open boundaries
                   (Min = -2.97985417E-02 Max =  4.50654422E-03)
                - vertically integrated u-momentum component
                   (Min = -3.19639299E+02 Max =  5.51285904E+02)
                - vertically integrated u-momentum component, open boundaries
                   (Min =  0.00000000E+00 Max =  0.00000000E+00)
                - vertically integrated v-momentum component
                   (Min = -4.74124122E+02 Max =  2.47583478E+02)
                - vertically integrated v-momentum component, open boundaries
                   (Min =  0.00000000E+00 Max =  0.00000000E+00)
                - surface u-momentum stress, adjusted ad_ustr
                   (Min = -2.68776972E-03 Max =  2.77484868E-03)
                - surface v-momentum stress, adjusted ad_vstr
                   (Min = -4.69903417E-03 Max =  4.84276342E-03)
                - u-momentum component
                   (Min = -3.40267567E-03 Max =  3.47397634E-03)
                - u-momentum component, open boundaries
                   (Min = -6.00707799E-04 Max =  3.93827981E-03)
                - v-momentum component
                   (Min = -2.59196451E-03 Max =  3.39703444E-03)
                - v-momentum component, open boundaries
                   (Min = -1.05940062E-03 Max =  2.28738160E-03)
                - potential temperature
                   (Min = -2.41010813E-02 Max =  5.90115782E-02)
                - salinity
                   (Min = -8.62845187E-02 Max =  1.25436613E-01)
                - potential temperature, open boundaries
                   (Min = -7.20358057E-05 Max =  2.83838639E-04)
                - salinity, open boundaries
                   (Min = -2.29848571E-04 Max =  6.27136769E-05)
                - surface net heat flux, adjusted ad_tflux
                   (Min = -4.32641798E-03 Max =  1.57185732E-03)
                - surface net salt flux, (E-P)*SALT, adjusted ad_tflux
                   (Min = -1.26399719E-03 Max =  6.11255552E-03)

 <<<< Conjugate Gradient Algorithm >>>>
starting log

Code: Select all

--------------------------------------------------------------------------------
 Model Input Parameters:  ROMS/TOMS version 3.9  
                          Friday - August 28, 2020 -  8:45:29 AM
--------------------------------------------------------------------------------

 California Current System, 1/3 degree resolution

 Operating system  : Linux
 CPU/hardware      : x86_64
 Compiler system   : gfortran
 Compiler command  : /usr/bin/mpif90
 Compiler flags    : -frepack-arrays -O3 -ffast-math -
 OCN Communicator  : 0,  PET size = 3

 Input Script      : roms_wc13_2hours.in

 SVN Root URL     : https://www.myroms.org/svn/src/trunk
 SVN Revision     : 1037

 Local Root       : /home/bangdt/200820/trunk
 Header Dir       : /home/bangdt/200820/test/WC13/I4DVAR
 Header file      : wc13.h
 Analytical Dir   : /home/bangdt/200820/test/WC13/Functionals

 Resolution, Grid 01: 54x53x30,  Parallel Nodes: 3,  Tiling: 1x3


 Physical Parameters, Grid: 01
 =============================

        192  ntimes            Number of timesteps for 3-D equations.
   1800.000  dt                Timestep size (s) for 3-D equations.
         72  ndtfast           Number of timesteps for 2-D equations between
                                 each 3D timestep.
          1  ERstr             Starting ensemble/perturbation run number.
          1  ERend             Ending ensemble/perturbation run number.
          1  Nouter            Maximun number of 4DVAR outer loop iterations.
         25  Ninner            Maximum number of 4D-Var inner loop iterations.
          0  nrrec             Number of restart records to read from disk.
          T  LcycleRST         Switch to recycle time-records in restart file.
        192  nRST              Number of timesteps between the writing of data
                                 into restart fields.
          1  ninfo             Number of timesteps between print of information
                                 to standard output.
          T  ldefout           Switch to create a new output NetCDF file(s).
          4  nHIS              Number of timesteps between the writing fields
                                 into history file.
          0  nQCK              Number of timesteps between the writing fields
                                 into quicksave file.
          F  LcycleTLM         Switch to recycle time-records in tangent file.
         48  nTLM              Number of timesteps between the writing of
                                 data into tangent file.
          F  LcycleADJ         Switch to recycle time-records in adjoint file.
        192  nADJ              Number of timesteps between the writing of
                                 data into adjoint file.
         48  nOBC              Number of timesteps between 4D-Var adjustment of
                                 open boundaries, Nbrec = 5
         48  nSFF              Number of timesteps between 4D-Var adjustment of
                                 surface forcing fields, Nfrec = 5
 4.0000E+00  nl_tnu2(01)       NLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 01: temp
 4.0000E+00  ad_tnu2(01)       ADM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 01: temp
 4.0000E+00  tl_tnu2(01)       TLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 01: temp
 4.0000E+00  nl_tnu2(02)       NLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 02: salt
 4.0000E+00  ad_tnu2(02)       ADM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 02: salt
 4.0000E+00  tl_tnu2(02)       TLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for tracer 02: salt
 4.0000E+00  nl_visc2          NLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for momentum.
 4.0000E+00  ad_visc2          ADM Horizontal, harmonic mixing coefficient
                                 (m2/s) for momentum.
 4.0000E+00  tl_visc2          TLM Horizontal, harmonic mixing coefficient
                                 (m2/s) for momentum.
          T  LuvSponge         Turning ON  sponge on horizontal momentum.
          T  LtracerSponge(01) Turning ON  sponge on tracer 01: temp
          T  LtracerSponge(02) Turning ON  sponge on tracer 02: salt
 1.0000E-05  Akt_bak(01)       Background vertical mixing coefficient (m2/s)
                                 for tracer 01: temp
 1.0000E-05  Akt_bak(02)       Background vertical mixing coefficient (m2/s)
                                 for tracer 02: salt
 1.0000E-05  Akv_bak           Background vertical mixing coefficient (m2/s)
                                 for momentum.
 5.0000E-06  Akk_bak           Background vertical mixing coefficient (m2/s)
                                 for turbulent energy.
 5.0000E-06  Akp_bak           Background vertical mixing coefficient (m2/s)
                                 for turbulent generic statistical field.
     -1.000  gls_p             GLS stability exponent.
      0.500  gls_m             GLS turbulent kinetic energy exponent.
     -1.000  gls_n             GLS turbulent length scale exponent.
 7.6000E-06  gls_Kmin          GLS minimum value of turbulent kinetic energy.
 1.0000E-12  gls_Pmin          GLS minimum value of dissipation.
 5.4770E-01  gls_cmu0          GLS stability coefficient.
 5.5500E-01  gls_c1            GLS shear production coefficient.
 8.3300E-01  gls_c2            GLS dissipation coefficient.
-6.0000E-01  gls_c3m           GLS stable buoyancy production coefficient.
 1.0000E+00  gls_c3p           GLS unstable buoyancy production coefficient.
 2.0000E+00  gls_sigk          GLS constant Schmidt number for TKE.
 2.0000E+00  gls_sigp          GLS constant Schmidt number for PSI.
   1400.000  charnok_alpha     Charnok factor for Zos calculation.
      0.500  zos_hsig_alpha    Factor for Zos calculation using Hsig(Awave).
      0.250  sz_alpha          Factor for Wave dissipation surface tke flux .
    100.000  crgban_cw         Factor for Craig/Banner surface tke flux.
 1.0000E+00  ad_Akt_fac(01)    ADM basic state vertical mixing scale factor
                                 for tracer 01: temp
 1.0000E+00  tl_Akt_fac(01)    TLM basic state vertical mixing scale factor
                                 for tracer 01: temp
 1.0000E+00  ad_Akt_fac(02)    ADM basic state vertical mixing scale factor
                                 for tracer 02: salt
 1.0000E+00  tl_Akt_fac(02)    TLM basic state vertical mixing scale factor
                                 for tracer 02: salt
 1.0000E+00  ad_Akv_fac        ADM basic state vertical mixing scale factor
                                 for momentum.
 1.0000E+00  tl_Akv_fac        TLM basic state vertical mixing scale factor
                                 for momentum.
 3.0000E-04  rdrg              Linear bottom drag coefficient (m/s).
 2.5000E-03  rdrg2             Quadratic bottom drag coefficient.
 2.0000E-02  Zob               Bottom roughness (m).
 2.0000E-02  Zos               Surface roughness (m).
 1.0000E+01  blk_ZQ            Height (m) of surface air humidity measurement.
 2.0000E+00  blk_ZT            Height (m) of surface air temperature measurement.
 1.0000E+01  blk_ZW            Height (m) of surface winds measurement.
          1  lmd_Jwt           Jerlov water type.
          1  Vtransform        S-coordinate transformation equation.
          1  Vstretching       S-coordinate stretching function.
 5.0000E+00  theta_s           S-coordinate surface control parameter.
 4.0000E-01  theta_b           S-coordinate bottom  control parameter.
     10.000  Tcline            S-coordinate surface/bottom layer width (m) used
                                 in vertical coordinate stretching.
   1025.000  rho0              Mean density (kg/m3) for Boussinesq approximation.
 1.0000E-05  bvf_bak           Background Brunt-Vaisala frequency squared (1/s2).
  13008.000  dstart            Time-stamp assigned to model initialization (days).
      -2.00  time_ref          Reference time for units attribute (yyyymmdd.dd)
 3.6000E+02  Tnudg(01)         Nudging/relaxation time scale (days)
                                 for tracer 01: temp
 3.6000E+02  Tnudg(02)         Nudging/relaxation time scale (days)
                                 for tracer 02: salt
 3.6000E+02  Znudg             Nudging/relaxation time scale (days)
                                 for free-surface.
 3.6000E+02  M2nudg            Nudging/relaxation time scale (days)
                                 for 2D momentum.
 3.6000E+02  M3nudg            Nudging/relaxation time scale (days)
                                 for 3D momentum.
 3.6000E+02  obcfac            Factor between passive and active
                                 open boundary conditions.
          F  VolCons(1)        NLM western  edge boundary volume conservation.
          F  VolCons(2)        NLM southern edge boundary volume conservation.
          F  VolCons(3)        NLM eastern  edge boundary volume conservation.
          F  VolCons(4)        NLM northern edge boundary volume conservation.
          F  ad_VolCons(1)     ADM western  edge boundary volume conservation.
          F  ad_VolCons(2)     ADM southern edge boundary volume conservation.
          F  ad_VolCons(3)     ADM eastern  edge boundary volume conservation.
          F  ad_VolCons(4)     ADM northern edge boundary volume conservation.
          F  tl_VolCons(1)     TLM western  edge boundary volume conservation.
          F  tl_VolCons(2)     TLM southern edge boundary volume conservation.
          F  tl_VolCons(3)     TLM eastern  edge boundary volume conservation.
          F  tl_VolCons(4)     TLM northern edge boundary volume conservation.
      5.000  T0                Background potential temperature (C) constant.
     35.000  S0                Background salinity (PSU) constant.
 1.0000E-04  Tcoef             Thermal expansion coefficient (1/Celsius).
 7.6000E-04  Scoef             Saline contraction coefficient (1/PSU).
      1.000  gamma2            Slipperiness variable: free-slip (1.0) or 
                                                      no-slip (-1.0).
          F  LuvSrc            Turning OFF momentum point Sources/Sinks.
          F  LwSrc             Turning OFF volume influx point Sources/Sinks.
          F  LtracerSrc(01)    Turning OFF point Sources/Sinks on tracer 01: temp
          F  LtracerSrc(02)    Turning OFF point Sources/Sinks on tracer 02: salt
          F  LsshCLM           Turning OFF processing of SSH climatology.
          F  Lm2CLM            Turning OFF processing of 2D momentum climatology.
          F  Lm3CLM            Turning OFF processing of 3D momentum climatology.
          F  LtracerCLM(01)    Turning OFF processing of climatology tracer 01: temp
          F  LtracerCLM(02)    Turning OFF processing of climatology tracer 02: salt
          F  LnudgeM2CLM       Turning OFF nudging of 2D momentum climatology.
          F  LnudgeM3CLM       Turning OFF nudging of 3D momentum climatology.
          F  LnudgeTCLM(01)    Turning OFF nudging of climatology tracer 01: temp
          F  LnudgeTCLM(02)    Turning OFF nudging of climatology tracer 02: salt

          T  Hout(idFsur)      Write out free-surface.
          T  Hout(idUbar)      Write out 2D U-momentum component.
          T  Hout(idVbar)      Write out 2D V-momentum component.
          T  Hout(idUvel)      Write out 3D U-momentum component.
          T  Hout(idVvel)      Write out 3D V-momentum component.
          T  Hout(idWvel)      Write out W-momentum component.
          T  Hout(idOvel)      Write out omega vertical velocity.
          T  Hout(idTvar)      Write out tracer 01: temp
          T  Hout(idTvar)      Write out tracer 02: salt
          T  Hout(idUsms)      Write out surface U-momentum stress.
          T  Hout(idVsms)      Write out surface V-momentum stress.
          T  Hout(idTsur)      Write out surface net heat flux.
          T  Hout(idTsur)      Write out surface net salt flux.
          T  Hout(idSrad)      Write out shortwave radiation flux.
          T  Hout(idLrad)      Write out longwave radiation flux.
          T  Hout(idLhea)      Write out latent heat flux.
          T  Hout(idShea)      Write out sensible heat flux.
          T  Hout(idEmPf)      Write out E-P flux.
          T  Hout(idevap)      Write out evaporation rate.
          T  Hout(idrain)      Write out rain rate.
          T  Hout(idVvis)      Write out vertical viscosity: AKv.
          T  Hout(idTdif)      Write out vertical diffusion: AKt(itemp).
          T  Hout(idSdif)      Write out vertical diffusion: AKt(isalt).

 Output/Input Files:

    Output DA Initial/Restart File:  wc13_dai.nc
               Output Restart File:  wc13_rst.nc
               Output History File:  wc13_fwd.nc
               Output Tangent File:  wc13_tlm.nc
               Output Adjoint File:  wc13_adj.nc
                   Input Grid File:  ../Data/wc13_grd.nc
      Input Nonlinear Initial File:  wc13_ini.nc
             Input Forcing File 01:  ../Data/coamps_wc13_lwrad_down.nc
             Input Forcing File 02:  ../Data/coamps_wc13_Pair.nc
             Input Forcing File 03:  ../Data/coamps_wc13_Qair.nc
             Input Forcing File 04:  ../Data/coamps_wc13_rain.nc
             Input Forcing File 05:  ../Data/coamps_wc13_swrad.nc
             Input Forcing File 06:  ../Data/coamps_wc13_Tair.nc
             Input Forcing File 07:  ../Data/coamps_wc13_wind.nc
    Input Lateral Boundary File 01:  ../Data/wc13_ecco_bry.nc
      Assimilation Parameters File:  i4dvar.in
  ROMS I/O variables Metadata File:  /home/bangdt/200820/trunk/ROMS/External/varinfo.dat
            Input/Output USER File:  MyFile.dat

 Generic User Parameters:

 5.0000E+00  user(01)          User parameter 01.

 Tile partition information for Grid 01:  54x53x30  tiling: 1x3

     tile     Istr     Iend     Jstr     Jend     Npts

        0        1       54        1       18    29160
        1        1       54       19       36    29160
        2        1       54       37       53    27540

 Tile minimum and maximum fractional coordinates for Grid 01:
   (interior points only)

     tile     Xmin     Xmax     Ymin     Ymax     grid

        0     0.50    54.50     0.50    18.50  RHO-points
        1     0.50    54.50    18.50    36.50  RHO-points
        2     0.50    54.50    36.50    53.50  RHO-points

        0     1.00    54.00     0.50    18.50    U-points
        1     1.00    54.00    18.50    36.50    U-points
        2     1.00    54.00    36.50    53.50    U-points

        0     0.50    54.50     1.00    18.50    V-points
        1     0.50    54.50    18.50    36.50    V-points
        2     0.50    54.50    36.50    53.00    V-points

 Maximum halo size in XI and ETA directions:

               HaloSizeI(1) =     130
               HaloSizeJ(1) =      60
                TileSide(1) =      59
                TileSize(1) =    1416



 Assimilation Parameters, Grid: 01
 =================================

Post Reply