observation data rejected!

Discussion about tangent linear and adjoint models, variational data assimilation, and other related issues.

Moderators: arango, robertson

Post Reply
Message
Author
simion1232006
Posts: 60
Joined: Tue Sep 29, 2009 3:50 pm
Location: School of Environment System Engineering,UWA

observation data rejected!

#1 Unread post by simion1232006 »

Number of State Observations Processed: ObsTime = 40124.0000

Variable IstrObs IendObs Count Rejected

temp 1 33 33 32

Total 33 32
Obs Tally 33 32

Wrote NLM state at observation locations, datum = 0000001 - 0000033


Almost all of my data were rejected.Does anyone have the similar problems?

marciath
Posts: 2
Joined: Thu Jan 14, 2010 7:28 pm
Location: University of Hawaii Manoa

Re: observation data rejected!

#2 Unread post by marciath »

Have you checked to make sure that your obs file variables are all in the correct format? The model is very particular in how it reads in the data.

User avatar
arango
Site Admin
Posts: 1347
Joined: Wed Feb 26, 2003 4:41 pm
Location: DMCS, Rutgers University
Contact:

Re: observation data rejected!

#3 Unread post by arango »

The 4D-Var algorithms reject data if the input observations locations are not in inside the application grid or inside the assimilation time window. Notice the the spatial locations (obs_Xgrid, obs_Ygrid, obs_Zgrid) are in in grid fractional coordinates (0:L, 0:M, 1:N) and the time coordinate (survey_time, obs_time) is in days with the same calendar as the model time and forcing (for example, days since 1968-05-23 00:00:00 GMT).

Users need to check the observations before assimilating. There is not quality control on the observations inside ROMS. This needs to be done during pre-processing.

wangzc
Posts: 28
Joined: Fri Dec 28, 2012 5:44 am
Location: National Marine Environmental Forecasting Center

Re: observation data rejected!

#4 Unread post by wangzc »

hi,
i come with the same question. Gauge data is assimilated to the model and i check every point to make sure
the spatial locations (obs_Xgrid, obs_Ygrid, obs_Zgrid) are in in grid fractional coordinates (0:L, 0:M, 1:N) and the time coordinate (survey_time, obs_time) is in days with the same calendar as the model time and forcing
. The cycle runs 1 day and data per hour has prepared for the poried. Only one time level data is accepted。
Anyone know why?

wangzc
Posts: 28
Joined: Fri Dec 28, 2012 5:44 am
Location: National Marine Environmental Forecasting Center

Re: observation data rejected!

#5 Unread post by wangzc »

And also,
i prepare 24 gauge data along the coast, and restore it in form of grid.Then 'd_ssh_obs.m' is changed to process the grid data. The result is, only 7 diffetent gauge data writes to the obs.nc.Why?

Rafael_Santana
Posts: 12
Joined: Wed Jun 04, 2014 1:45 pm
Location: UFBA and REMO

Re: observation data rejected!

#6 Unread post by Rafael_Santana »

I'm also having SST gridded satellite data getting rejected.

About 195 out of 204 SST data points are being rejected. And only 4 out of 204 SSH data points were rejected.

I'm setting the SST (temp) depth to 2 cm and the minimum error to 0.2oC.

Please let me know if you have any ideas.

Fragments of the log file below:

Code: Select all

" Number of State Observations Processed:  ObsTime =    9585.0000,  2016-03-30 00:00:00.00

          Variable          IstrObs    IendObs      Count   Rejected

          zeta                    1        204        204          4

          Total                                       204          4
          Obs Tally                                   204          4
...

 Number of State Observations Processed:  ObsTime =    9585.5000,  2016-03-30 12:00:00.00

          Variable          IstrObs    IendObs      Count   Rejected

          temp                  205        408        204        195

          Total                                       204        195
          Obs Tally                                   408        199
...

 Number of State Observations Processed:  ObsTime =    9586.0000,  2016-03-31 00:00:00.00

          Variable          IstrObs    IendObs      Count   Rejected

          zeta                  409        612        204          4

          Total                                       204          4
          Obs Tally                                   612        203

...

 Number of State Observations Processed:  ObsTime =    9586.5000,  2016-03-31 12:00:00.00

          Variable          IstrObs    IendObs      Count   Rejected

          temp                  613        816        204        189

          Total                                       204        189
          Obs Tally                                   816        392

"

jsouza
Posts: 31
Joined: Tue Apr 10, 2012 10:55 pm
Location: Metocean Solutions

Re: observation data rejected!

#7 Unread post by jsouza »

Hi Raf.

Did you check your obs_Zgrid?
Another thing to check is if you have BGQC activated. This will reject data based on the parameters set in your s4dvar.in

Cheers

Joao

User avatar
wilkin
Posts: 872
Joined: Mon Apr 28, 2003 5:44 pm
Location: Rutgers University
Contact:

Re: observation data rejected!

#8 Unread post by wilkin »

Rafael,

Look in the mod file. It reports all the variables that ROMS uses to make a decision as to which observations are rejected.
There you will see reported the innovations, obs and background errors. If you have BGQC active you can determine whether it is configured as you intended for your specified error hypotheses. Be careful to note whether values are standard deviations or variances.

Data outside the domain will be rejected, so make sure your obs coordinates are correct. Mishandling of the vertical grid is a common mistake.

John.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu

Rafael_Santana
Posts: 12
Joined: Wed Jun 04, 2014 1:45 pm
Location: UFBA and REMO

Re: observation data rejected!

#9 Unread post by Rafael_Santana »

Thanks Joao and John for your replies.

The problem was in obs_Zgrid. I've changed the depth of temp from 0.02 to 0.5 m in the scripts I'm using and I got better values for obs_Zgrid (below).

Before:
obs_Zgrid = -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, 29.9982020984498, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, 29.9931272988072, 29.9914863532605,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0, -0,
-0, -0, -0, -0, -0, -0, -0, -0, -0, -0, 0, 0, 0, 0, -0, -0, -0, -0

After:
obs_Zgrid = 29.7888158381515, 29.5984096106621, 29.6626000352451,
29.756141155388, 29.6407162246107, 29.8618950435596, 29.7500118677251,
29.7599251774095, 29.6392872744959, 29.7943965092529, 29.762106521323,
29.8524734031373, 29.7938844741204, 29.809034937104, 29.7090891536804,
29.644895458076, 29.5965694954022, 29.627895167986, 29.6355442815496 ...

The BGQC is on and (bgqc_type == 1) and applied to tracers: S_bgqc(isTvar) == 4.0d0 4.0d0 ! 1:NT tracers

I've started to check the mod.nc files to have a summary of each DA cycle.

Thanks,
Rafael

ckerry
Posts: 18
Joined: Wed Sep 15, 2010 1:59 pm
Location: UNSW Australia

Re: observation data rejected!

#10 Unread post by ckerry »

Hello,
I am seeing some unusual behaviour with Zgrid when running VERIFICATION.
In my input obs.nc file I have Zgrid=50 and depth=-2 for SST obs (my model has 50 layers).
When I run the VERIFICATION, the obs file is written to and altered. For the first day (of 6 daily times for which I have SST data) the Zgrid is altered from 50 to zero in some places and to range from 30-50 in others. At all following days Zgrid is left at 50.
The SST values that are output to the mod file are incorrect when the newly written Zgrid=50 and Zgrid=0, and only make ‘some’ sense when Zgrid is between 30-50 i.e. on day 1.

I tried writing Zgrid as 49.8 rather than 50 but I still get the same error. I also tried leaving Zgrid at 50 and changed depth to 0 for SST, same error. I also tried swapping Zgrid and depth in the obs file, as I have done in 4D-Var applications in the past, but still get the same error.

For SSH I have Zgrid=50 and depth=0 and the data is output correctly. Likewise I have some subsurface obs that are output correctly.
Attachments
Screen Shot 2022-09-23 at 11.59.23 am.png
Screen Shot 2022-09-23 at 11.59.30 am.png
Screen Shot 2022-09-23 at 12.02.07 pm.png

Post Reply