Opened 9 years ago

Closed 9 years ago

#248 closed upgrade (Done)

Updated output to NetCDF-4/HDF5 format file

Reported by: arango Owned by: arango
Priority: major Milestone: Parallel Input/Output
Component: Parallelism Version: 3.2
Keywords: Cc:

Description

Introduced two new CPP options to create NetCDF-4/HDF5 format files (NETCDF4) and file compression (DEFLATE). All ROMS standard input scripts were modified to include the file compression parameters:

! NetCDF-4/HDF5 compression parameters for output files.

  NC_SHUFFLE =  1                 ! if non-zero, turn on shuffle filter
  NC_DEFLATE =  1                 ! if non-zero, turn on deflate filter
   NC_DLEVEL =  1                 ! deflate level [0-9]

In order to generate NetCDF-4/HDF5, we needed to change the calls to:

          status=nf90_create(TRIM(ncname), CMODE, ncHISid(ng))

where CMODE is defined in mod_netcdf.F for classic NetCDF-3 files or NetCDF-4/HDF5 format files:

!
!  Netcdf file creation mode flag.
!
#ifdef NETCDF4
      integer :: CMODE = nf90_netcdf4      ! NetCDF-4/HDF5 format file
#else
      integer :: CMODE = nf90_clobber      ! NetCDF classic format file
#endif

The file compression cannot be used in parallel I/O for writing data. This is because the compression makes it impossible for the HDF5 library to exactly map the data to the disk location. However, deflated data can be read with parallel I/O.

Notice that when the USE_NETCDF4 macro is activated in the makefile of build scripts (build.sh or build.bash), we need a serial and parallel version of the NetCDF-4/HDF5 library. The parallel library uses parallel I/O through MPI-I/O so we need compile also with the MPI library. This is fine in ROMS distributed-memory applications. However in serial or shared-memory ROMS applications, we need to use the serial version of the NetCDF-4/HDF5 to avoid conflicts with the compiler. Recall also that the MPI library comes in several flavors: MPICH, MPICH2, and OpenMPI. Therefore, you need to have several versions of the NetCDF library in your computer for all the various configuration possibilities.

Many thanks to Rich Signell for taking a first look to this capability.

Change History (1)

comment:1 Changed 9 years ago by arango

  • Resolution set to Done
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.