Problem with scaling of non-gridded data

Bug reports, work arounds and fixes

Moderators: arango, robertson

Post Reply
Message
Author
User avatar
m.hadfield
Posts: 521
Joined: Tue Jul 01, 2003 4:12 am
Location: NIWA

Problem with scaling of non-gridded data

#1 Post by m.hadfield » Tue Feb 09, 2010 2:35 am

There is a problem with the scaling of non-gridded data from netCDF files using the scale_factor and add_offset attributes. The scale factor is being applied twice. :cry:

For example I have a netCDF file with ubar data on the western boundary as follows:

Code: Select all

netcdf roms_bry {
dimensions:
        xi_u = 79 ;
        eta_u = 89 ;
        time_fix = 1 ;
variables:
        float time_fix(time_fix) ;
        short ubar_west(time_fix, eta_u) ;
                ubar_west:time = "time_fix" ;
                ubar_west:add_offset = -0.000152594875864068 ;
                ubar_west:scale_factor = 0.000305189751728137 ;

// global attributes:
                :type = "Boundary forcing file" ;
data:

 ubar_west =
  20, 11, -14, -55, -20, 43, -3, -122, -64, 254, 442, 301, 176, 182, 260, 
  299, 257, 200, 139, 82, 22, -10, 6, 34, 50, 41, 40, 42, 37, 24, 5, -13, 
  -24, -24, -18, -9, -1, 4, 10, 15, 16, 11, 1, -6, -6, 1, 5, 6, 5, 3, 3, 
  2, 4, 6, 7, 6, 2, -1, -1, 1, 3, 5, 2, -3, -6, 1, 6, 8, 7, 5, 3, 1, 3, 
  8, 13, 18, 19, 18, 14, 12, 12, 8, 6, 12, 20, 17, -3, -27, -38 ;
}
 
The maximum and minimum integer values are -122 and 442, respectively. With the scaling attributes in effect, these numbers scale to physical values of -0.0374 and 0.1347.

This file is being read by GET_NGFLD, which reports

Code: Select all

    GET_NGFLD   - 2D u-momentum western boundary condition,  t =     0 00:00:00
                   (Rec=0001, Index=2, File: roms_bry.nc)
                   (Tmin=          0.0000 Tmax=          0.0000)
                   (Min = -1.14097461E-05 Max =  4.11216564E-05)
The minimum and maximum reported by GET_NGFLD are much too small, by a factor equal to the scale_factor attribute, 0.0003051.

With a bit of debugging in get_ngfld.F, I see the data in question are being read from the netCDF file at lines 400-404 by function NETCDF_GET_FVAR, then the minimum and maximum values are calculated at lines 410-421 and written to stdout at lines 432-434. The scale_factor attribute was previously read from the file (line 147) and stored in Finfo(10,ifield,ng). This scale factor is applied in the calculation of the minimum and maximum at lines 410-421, but the data returned by NETCDF_GET_FVAR have already been scaled inside that function and are between -0.0374 and 0.1347, so the extra scaling causes the numbers to be much too small.

I'm not 100% sure but I think the scale factor in Finfo(10,ifield,ng) is also passed out of GET_NGFLD and used to scale the values used in the hydrodynamic simulation, leading to unrealistic results.

I think the same problem applies to point data processed by GET_2DFLD, see lines 448-456 in get_2dfld.F.

User avatar
jivica
Posts: 135
Joined: Mon May 05, 2003 2:41 pm
Location: The University of Western Australia, Perth, Australia

Re: Problem with scaling of non-gridded data

#2 Post by jivica » Wed Feb 10, 2010 12:42 am

Are you sure that you are using the last version of ROMS?
This was corrected recently, we had the problem with atmo forcing scaled to short int (to save diskspace)!
Think there are still some issues if trying to read boundary files using scales (short int), but those files are small anyway and we are not using that feature.

Cheers,
Ivica

User avatar
m.hadfield
Posts: 521
Joined: Tue Jul 01, 2003 4:12 am
Location: NIWA

Re: Problem with scaling of non-gridded data

#3 Post by m.hadfield » Wed Feb 10, 2010 2:19 am

I was using code from my own branch on the developers's SVN site, which I recently brought up to date with Hernan's branch through to revision 1155 (implementing Trac ticket 407). To the best of my knowledge the latter is the same as the master version

I just checked the master copy of get_ngfld.F. It has the double-scaling problem that I reported.

I agree that boundary files are generally small, so there is not a lot to gain by packing them. Still the present behaviour is clearly wrong and should be fixed. I'm happy to have a go myself if no-one else is willing/able.

User avatar
arango
Site Admin
Posts: 1117
Joined: Wed Feb 26, 2003 4:41 pm
Location: IMCS, Rutgers University
Contact:

Re: Problem with scaling of non-gridded data

#4 Post by arango » Thu Feb 11, 2010 9:19 pm

Yes, this is new bug. The generic routines in mod_netcdf.F take care of the scale_factor and add_offset attributes. The Fscale factor is a different issue in ROMS and doesn't have nothing to do with the compression of data but field units manipulation from varinfo.dat. Usually, Fscale=1.0 for mostly all input fields.

I fixed this problem and :arrow: updated the repository. Thank you for reporting this problem.

User avatar
m.hadfield
Posts: 521
Joined: Tue Jul 01, 2003 4:12 am
Location: NIWA

Re: Problem with scaling of non-gridded data

#5 Post by m.hadfield » Fri Feb 12, 2010 3:41 am

I believe the same erroneous scaling occurs for point data in get_2dfld.F (line 453), get_2dlfr.F (line 463), get_3dfld.F (line 434) and get_3dfldr.F (line 443).

User avatar
arango
Site Admin
Posts: 1117
Joined: Wed Feb 26, 2003 4:41 pm
Location: IMCS, Rutgers University
Contact:

Re: Problem with scaling of non-gridded data

#6 Post by arango » Fri Feb 12, 2010 4:26 am

Yeah, the same correction is also needed for get_2dfld.F, get_2dfldr.F, get_3dfld.F, and get_3dfldr.F when processing point-data. However, it very unlikely to have an input NetCDF file with point-data that needs to be compressed as byte and scaled with the scale_factor and add_offset attributes.

Thank you, I re-opened the :arrow: ticket 408 and updated the repository.

Post Reply