id	summary	reporter	owner	description	type	status	priority	milestone	component	version	resolution	keywords	cc
531	Updated Matlab processing scripts for NetCDF	arango	arango	"I updated several Matlab scripts to allow processing of NetCDF file(s) on '''OpenDAP''' servers.  Previously, all the Matlab scripts in the ROMS repository (https://www.myroms.org/svn/src/matlab) used the '''MEXNC''' interface to Matlab. Currently, the processing of '''URL''' files (single or aggregated) on an '''OpenDAP''' server can be only processed with the Java interface of '''SNCTOOLS'''.

The following files were updated:

 * '''utility/nc_dim.m''': Added the capability to inquire about the dimensions in input '''URL''' NetCDF file(s).

 * '''utility/nc_getatt.m''': Added the capability to inquire about a specified global or variable attribute in input '''URL''' NetCDF file(s).

 * '''utility/nc_inq.m''': Added the capability to print the dimensions and variables available in input '''URL''' NetCDF file(s).

 * '''utility/nc_read.m''': Added the capability to read data from input '''URL''' NetCDF file(s).  The function was updated to read correctly any type of numerical data (double, single, interger, short, and byte). It uses the new '''SNCTOOLS''' function '''nc_vargetr''' which processes the input data in its native (raw) data type. This is extremely important for post-processing the data correctly with '''_FillValue''', '''missing_value''', '''scale_factor''', and '''add_offset''' attributes.  The default '''SNCTOOLS''' function '''nc_varget''' converts all numeric data types to double precession before the post-processing is done. In my opinion this is incorrect when unpacking data.  Sometimes, it also introduces round-off errors with surprising results.  A more robust post-processing of the input data is introduced.  For example, to find all fill or missing values, stored in '''spval''', we compute:
{{{
   ind=find(abs(f-spval) < 4*eps(double(spval));
}}}
  Then, the scaling of byte/short data and subsequent fill/missing value replacement is, for example:
{{{
   f=double(f)*scale+offset;
   f(ind)=ReplaceValue;         % usually zero or NaN for ploting
}}}
  Therefore, the steps are as follows: '''i''') read requested data in its native (raw) numerical data type, '''ii''') search for fill or missing values if any, '''iii''') unpack the data if the attributes '''scale_factor''' and/or '''add_offset''' are present, '''iv''') replace fill/missing values with the provided replace value.  Notice that the function call to '''nc_read''' is now:
{{{
   f=nc_read(Fname,Vname,Tindex,ReplaceValue,PreserveType);
}}}
  That is, a new optional fifth argument is added to '''nc_read.m''' to allow the user to preserve input numerical data type.  The user may set '''!PreseveType=true''', so the native data type is returned. If this argument is omitted, the default is to convert numerical data to double precision.  If the data is packed to store the most significant bits with '''scale_factor''' and '''add_offset''', the returned data is in double precision.

 * '''utility/nc_url.m''': New function to inquire if the input NetCDF file name is a '''URL''' from an '''OpenDAP''' server.

 * '''utility/nc_varinfo.m''': New script returning a structure containing the variable information. This function was adapted and have similar capabilities to the '''SNCTOOLS''' '''nc_getvarinfo''' function, except that all numerical values are returned in its native data type.  The original function '''nc_getvarinfo''' unwisely converted all the numerical values to double precision.  This is problematic when dealing with the variable numerical attributes. Specially, when using the NetCDF attributes: '''_FillValue''', '''missing_value''', '''scale_factor''', and '''add_offset'''.  And to lesser extend when using the attributes: '''valid_min''', '''valid_max''', and '''valid_range'''.

 * '''utility/nc_vinfo.m''': added the capability of inquiring the variable dimensions and the C-grid type in input '''URL''' NetCDF file(s).

 * '''utility/nc_vname.m''': added the capability of inquiring all the variable names in input '''URL''' NetCDF file(s).

 * '''utility/nc_write.m''': Added the capability of writing byte and short type data. Updated the processing of the '''_FillValue''' attribute.

Notice that we can only read data from '''URL''' NetCDF file(s).  We cannot write neither modify '''URL''' NetCDF file(s).  We don't own them, have the privileges to modify them, and the Java interface does not allow it.

In order for '''nc_read.m''' to work correctly, the user need to update the '''SNCTOOL''' scripts.  The new function '''nc_vargetr''' was added on Dec 18, 2011. Many thanks to '''John Evans''' for providing this function to return the data in its native (raw) data type.

Recall that the '''SNCTOOLS''' can be downloaded/updated from the '''svn''' repository at '''sourceforge''':
{{{
svn checkout https://mexcdf.svn.sourceforge.net/svnroot/mexcdf/snctools/trunk ~/matlab/snctools
}}}

Similarly, the '''MEXNC''' interface can be downloaded/updated from the '''svn''' repository at '''sourceforge''':
{{{
svn checkout https://mexcdf.svn.sourceforge.net/svnroot/mexcdf/mexnc/trunk ~/matlab/mexnc
}}}

Happy New Year 2012...
"	upgrade	closed	major	Matlab Processing Scripts	Matlab	3.6	Done