Can't run my IS4DVar case Gnorm = NaN
Can't run my IS4DVar case Gnorm = NaN
Hi, everyone!
When I run my IS4DVar model, it blows up at the beginning of the first inner loop. The log file shows:
…
<< Descent Algorithm >
(001,000): Initial gradient norm, Gnorm = NaN
…
< 4D Variational Data Assimilation, Outer = 001, Inner = 001 >
STEP KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 NaN NaN NaN 1.287561E+16
The initial gradient norm is NaN and I think this is why the model blows up. But I don’t know why the model outputs initial gradient norms of NaN.
My ocean.in and s4dvar.in file are like those in WC13 case. At first I ran the forward model for one year. Then I computed the standard deviations using this forward model output data and wrote them into the initial STD files(use the ROMS matlab script d_std.m). The normalization coefficients were computed using the exact method. The observation data are AVISO SLA data and NOAA OISST data. I added the mean sea surface, which are calculated from the one year forward model, to the AVISO SLA data, then wrote them into the observation file(use ROMS matlab script d_ssh_obs.m, d_sst_obs.m and obs_merge.m).
Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.
I attached my log file for your convenience.
Thank you for your help.
When I run my IS4DVar model, it blows up at the beginning of the first inner loop. The log file shows:
…
<< Descent Algorithm >
(001,000): Initial gradient norm, Gnorm = NaN
…
< 4D Variational Data Assimilation, Outer = 001, Inner = 001 >
STEP KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 NaN NaN NaN 1.287561E+16
The initial gradient norm is NaN and I think this is why the model blows up. But I don’t know why the model outputs initial gradient norms of NaN.
My ocean.in and s4dvar.in file are like those in WC13 case. At first I ran the forward model for one year. Then I computed the standard deviations using this forward model output data and wrote them into the initial STD files(use the ROMS matlab script d_std.m). The normalization coefficients were computed using the exact method. The observation data are AVISO SLA data and NOAA OISST data. I added the mean sea surface, which are calculated from the one year forward model, to the AVISO SLA data, then wrote them into the observation file(use ROMS matlab script d_ssh_obs.m, d_sst_obs.m and obs_merge.m).
Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.
I attached my log file for your convenience.
Thank you for your help.
 Attachments

 ocean_nscs_i4dvar.log
 (644.04 KiB) Downloaded 135 times
Re: Can't run my IS4DVar case Gnorm = NaN
Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.
JH
Does your domain include equator?
If it's ok for you, I want to see your model domain.
JH
Re: Can't run my IS4DVar case Gnorm = NaN
Hi, JHsusonic wrote:Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.
JH
Thanks.
My domain doesn't include equator.It's 16N  24N, 105E  132E, in north South China Sea.
And I have another question while I compile with SANITY_CHECK, the error goes:
cd /mnt/volume04/yinpeng/Projects/nSCS/Build; /usr/local/mpich21.2.1p1/bin/mpif90 c frepackarrays O3 ffastmath ocean_control.f90
ocean_control.f90:394.58:
CALL mp_collect (ng, iTLM, 4, IniVal, val(1,ng))
1
Error: There is no specific subroutine for the generic 'mp_collect' at (1)
make: *** [/mnt/volume04/yinpeng/Projects/nSCS/Build/ocean_control.o] Error 1
In fact I'm not quite clear ahout what deos SANITY_CHECK do.But have you ever met this error?
Thank you again for your help.
Re: Can't run my IS4DVar case Gnorm = NaN
I've never done SANITY_CHECK before. But I compiled with SANITY_CHECK and faced with same error message as you noted above(with gfortran).
Maybe code developer can helped this out to solve the problem.
I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.
Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?
Maybe code developer can helped this out to solve the problem.
I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.
Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?
Re: Can't run my IS4DVar case Gnorm = NaN
For above situation, try to use super observation file instead.Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.
Re: Can't run my IS4DVar case Gnorm = NaN
Many thanks.
Yes, it works!For above situation, try to use super observation file instead.
I tried NHIS=NOBC=NSFF=240(120), but the results remained the same.I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.
Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?
Re: Can't run my IS4DVar case Gnorm = NaN
I checked out the log file you attached several times but it is difficult to diagnose the causes of NaN value.
By the way, your Haney(rfactor) and Beckman & Haidvogel number is quiet high compare with known value(say, rfactor < 0.2) which means you did not smooth the bottom topography much.
By the way, your Haney(rfactor) and Beckman & Haidvogel number is quiet high compare with known value(say, rfactor < 0.2) which means you did not smooth the bottom topography much.
Re: Can't run my IS4DVar case Gnorm = NaN
Hi,
I wonder if you have solved this problem?
Now I have the similar error like you in my IS4DVar run, my initial gradient norm (Gnorm) is NaN.
<<<< Descent Algorithm >>>>
(001,000): Initial gradient norm, Gnorm = NaN
……
TLM: GET_STATE  Read state initial conditions, t = 1 00:00:00
(Grid 01, File: wc13_itl.nc, Rec=0001, Index=1)
 freesurface
(Min = NaN Max = NaN )
 freesurface, open boundaries
(Min = NaN Max = NaN )
 vertically integrated umomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component, open boundaries
(Min = NaN Max = NaN )
 vertically integrated vmomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component, open boundaries
(Min = NaN Max = NaN )
 surface umomentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
……
STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 1 00:00:00 NaN NaN NaN 1.159856E+16
Could you tell me more about this error and how you solved it?
Thank you very much.
I wonder if you have solved this problem?
Now I have the similar error like you in my IS4DVar run, my initial gradient norm (Gnorm) is NaN.
<<<< Descent Algorithm >>>>
(001,000): Initial gradient norm, Gnorm = NaN
……
TLM: GET_STATE  Read state initial conditions, t = 1 00:00:00
(Grid 01, File: wc13_itl.nc, Rec=0001, Index=1)
 freesurface
(Min = NaN Max = NaN )
 freesurface, open boundaries
(Min = NaN Max = NaN )
 vertically integrated umomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component, open boundaries
(Min = NaN Max = NaN )
 vertically integrated vmomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component, open boundaries
(Min = NaN Max = NaN )
 surface umomentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
……
STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 1 00:00:00 NaN NaN NaN 1.159856E+16
Could you tell me more about this error and how you solved it?
Thank you very much.
Re: Can't run my IS4DVar case Gnorm = NaN
Hi,
I'm also wondering if you solved this problem and I am having the same problem.
<<<< Descent Algorithm >>>>
(001,000): Initial gradient norm, Gnorm = NaN
.....
TLM: GET_STATE  Read state initial conditions, t = 4367 00:00:00
(File: ocean_itl.nc, Rec=0001, Index=1)
 freesurface
(Min = NaN Max = NaN )
 freesurface, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 surface umomentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
 surface vmomentum stress, adjusted tl_vstr
(Min = NaN Max = NaN )
 umomentum component
(Min = NaN Max = NaN )
 umomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vmomentum component
(Min = NaN Max = NaN )
 vmomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 potential temperature
(Min = NaN Max = NaN )
 salinity
(Min = NaN Max = NaN )
 potential temperature, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 salinity, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 surface net heat flux, adjusted tl_tflux
(Min = NaN Max = NaN )
 surface net salt flux, (EP)*SALT, adjusted tl_tflux
(Min = NaN Max = NaN )
....
STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 4367 00:00:00 NaN NaN NaN 1.188685E+14
Blowingup: Saving latest model state into RESTART file
I'm also wondering if you solved this problem and I am having the same problem.
<<<< Descent Algorithm >>>>
(001,000): Initial gradient norm, Gnorm = NaN
.....
TLM: GET_STATE  Read state initial conditions, t = 4367 00:00:00
(File: ocean_itl.nc, Rec=0001, Index=1)
 freesurface
(Min = NaN Max = NaN )
 freesurface, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated umomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vertically integrated vmomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 surface umomentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
 surface vmomentum stress, adjusted tl_vstr
(Min = NaN Max = NaN )
 umomentum component
(Min = NaN Max = NaN )
 umomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 vmomentum component
(Min = NaN Max = NaN )
 vmomentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 potential temperature
(Min = NaN Max = NaN )
 salinity
(Min = NaN Max = NaN )
 potential temperature, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 salinity, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
 surface net heat flux, adjusted tl_tflux
(Min = NaN Max = NaN )
 surface net salt flux, (EP)*SALT, adjusted tl_tflux
(Min = NaN Max = NaN )
....
STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 4367 00:00:00 NaN NaN NaN 1.188685E+14
Blowingup: Saving latest model state into RESTART file