Can't run my IS4D-Var case Gnorm = NaN

Discussion about tangent linear and adjoint models, variational data assimilation, and other related issues.

Moderators: arango, robertson

Post Reply
Message
Author
Arvin_Yin
Posts: 15
Joined: Wed Apr 23, 2014 12:05 pm
Location: Tsinghua University

Can't run my IS4D-Var case Gnorm = NaN

#1 Post by Arvin_Yin » Sat Sep 26, 2015 7:52 am

Hi, everyone!

When I run my IS4D-Var model, it blows up at the beginning of the first inner loop. The log file shows:

<< Descent Algorithm >
(001,000): Initial gradient norm, Gnorm = NaN

< 4D Variational Data Assimilation, Outer = 001, Inner = 001 >
STEP KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 NaN NaN NaN 1.287561E+16

The initial gradient norm is NaN and I think this is why the model blows up. But I don’t know why the model outputs initial gradient norms of NaN.

My ocean.in and s4dvar.in file are like those in WC13 case. At first I ran the forward model for one year. Then I computed the standard deviations using this forward model output data and wrote them into the initial STD files(use the ROMS matlab script d_std.m). The normalization coefficients were computed using the exact method. The observation data are AVISO SLA data and NOAA OISST data. I added the mean sea surface, which are calculated from the one year forward model, to the AVISO SLA data, then wrote them into the observation file(use ROMS matlab script d_ssh_obs.m, d_sst_obs.m and obs_merge.m).

Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.

I attached my log file for your convenience.

Thank you for your help.
Attachments
ocean_nscs_i4dvar.log
(644.04 KiB) Downloaded 120 times

User avatar
susonic
Posts: 159
Joined: Tue Aug 21, 2007 5:44 pm
Location: Jeju National University
Contact:

Re: Can't run my IS4D-Var case Gnorm = NaN

#2 Post by susonic » Sun Sep 27, 2015 7:45 pm

Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.

-JH

Arvin_Yin
Posts: 15
Joined: Wed Apr 23, 2014 12:05 pm
Location: Tsinghua University

Re: Can't run my IS4D-Var case Gnorm = NaN

#3 Post by Arvin_Yin » Mon Sep 28, 2015 3:15 am

susonic wrote:Hi,
Does your domain include equator?
If it's ok for you, I want to see your model domain.

-JH
Hi, JH
Thanks.
My domain doesn't include equator.It's 16N - 24N, 105E - 132E, in north South China Sea.

And I have another question while I compile with SANITY_CHECK, the error goes:

cd /mnt/volume04/yinpeng/Projects/nSCS/Build; /usr/local/mpich2-1.2.1p1/bin/mpif90 -c -frepack-arrays -O3 -ffast-math ocean_control.f90
ocean_control.f90:394.58:

CALL mp_collect (ng, iTLM, 4, IniVal, val(1,ng))
1
Error: There is no specific subroutine for the generic 'mp_collect' at (1)
make: *** [/mnt/volume04/yinpeng/Projects/nSCS/Build/ocean_control.o] Error 1

In fact I'm not quite clear ahout what deos SANITY_CHECK do.But have you ever met this error?

Thank you again for your help.
Attachments
my_domain.png

User avatar
susonic
Posts: 159
Joined: Tue Aug 21, 2007 5:44 pm
Location: Jeju National University
Contact:

Re: Can't run my IS4D-Var case Gnorm = NaN

#4 Post by susonic » Mon Sep 28, 2015 3:58 am

I've never done SANITY_CHECK before. But I compiled with SANITY_CHECK and faced with same error message as you noted above(with gfortran).

Maybe code developer can helped this out to solve the problem.


I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.

Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?

User avatar
susonic
Posts: 159
Joined: Tue Aug 21, 2007 5:44 pm
Location: Jeju National University
Contact:

Re: Can't run my IS4D-Var case Gnorm = NaN

#5 Post by susonic » Mon Sep 28, 2015 4:08 am

Also, I noticed that, when the model only includes the SLA data or the SST data in the observation file, the model rejected most of them, but if the model includes both the SLA data and the SST data, all the observation data can be accepted by the model.
For above situation, try to use super observation file instead.

Arvin_Yin
Posts: 15
Joined: Wed Apr 23, 2014 12:05 pm
Location: Tsinghua University

Re: Can't run my IS4D-Var case Gnorm = NaN

#6 Post by Arvin_Yin » Mon Sep 28, 2015 1:35 pm

Many thanks.
For above situation, try to use super observation file instead.
Yes, it works!
I saw the log file you attached and found TLM increments are zero even though ROMS continuing the intergration. I was wondering why this value is appearing.

Would you give same value to nHIS and nOBC, nSFF(say nHIS=240, nOBC=240, NSFF=240) and run the model again?
I tried NHIS=NOBC=NSFF=240(120), but the results remained the same.

User avatar
susonic
Posts: 159
Joined: Tue Aug 21, 2007 5:44 pm
Location: Jeju National University
Contact:

Re: Can't run my IS4D-Var case Gnorm = NaN

#7 Post by susonic » Tue Sep 29, 2015 4:52 am

I checked out the log file you attached several times but it is difficult to diagnose the causes of NaN value.

By the way, your Haney(rfactor) and Beckman & Haidvogel number is quiet high compare with known value(say, rfactor < 0.2) which means you did not smooth the bottom topography much.

wavewatch
Posts: 7
Joined: Thu Nov 14, 2013 1:51 am
Location: institute of oceanology

Re: Can't run my IS4D-Var case Gnorm = NaN

#8 Post by wavewatch » Mon Mar 26, 2018 7:03 am

Hi,

I wonder if you have solved this problem?
Now I have the similar error like you in my IS4D-Var run, my initial gradient norm (Gnorm) is NaN.

<<<< Descent Algorithm >>>>

(001,000): Initial gradient norm, Gnorm = NaN
……
TLM: GET_STATE - Read state initial conditions, t = 1 00:00:00
(Grid 01, File: wc13_itl.nc, Rec=0001, Index=1)
- free-surface
(Min = NaN Max = NaN )
- free-surface, open boundaries
(Min = NaN Max = NaN )
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component, open boundaries
(Min = NaN Max = NaN )
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component, open boundaries
(Min = NaN Max = NaN )
- surface u-momentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )

……

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
0 1 00:00:00 NaN NaN NaN 1.159856E+16

Could you tell me more about this error and how you solved it?

Thank you very much.

ckerry
Posts: 10
Joined: Wed Sep 15, 2010 1:59 pm
Location: UNSW Australia

Re: Can't run my IS4D-Var case Gnorm = NaN

#9 Post by ckerry » Sun Jul 22, 2018 11:55 pm

Hi,
I'm also wondering if you solved this problem and I am having the same problem.

<<<< Descent Algorithm >>>>

(001,000): Initial gradient norm, Gnorm = NaN

.....

TLM: GET_STATE - Read state initial conditions, t = 4367 00:00:00
(File: ocean_itl.nc, Rec=0001, Index=1)
- free-surface
(Min = NaN Max = NaN )
- free-surface, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated u-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- vertically integrated v-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- surface u-momentum stress, adjusted tl_ustr
(Min = NaN Max = NaN )
- surface v-momentum stress, adjusted tl_vstr
(Min = NaN Max = NaN )
- u-momentum component
(Min = NaN Max = NaN )
- u-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- v-momentum component
(Min = NaN Max = NaN )
- v-momentum component, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- potential temperature
(Min = NaN Max = NaN )
- salinity
(Min = NaN Max = NaN )
- potential temperature, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- salinity, open boundaries
(Min = 0.00000000E+00 Max = 0.00000000E+00)
- surface net heat flux, adjusted tl_tflux
(Min = NaN Max = NaN )
- surface net salt flux, (E-P)*SALT, adjusted tl_tflux
(Min = NaN Max = NaN )

....

STEP Day HH:MM:SS KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME

0 4367 00:00:00 NaN NaN NaN 1.188685E+14

Blowing-up: Saving latest model state into RESTART file

Post Reply