Finally, I am done with phase III of the parallel I/O implementation in ROMS. This was very tricky because it required a complete re-design of the I/O structure of ROMS to get a generic parallel interface (mod_netcdf.F
) for all input and output files. It is highly recommended to use the routines in this module for any non-tiled NetCDF I/O processing. Notice that most of the calls to the NetCDF library functions (nf90_***
) are made from the ROMS module (mod_netcdf
). The exceptions are def_dim.F
, and reading (nf_fread*d.F
) and writing (nf_fwrite*d.F
During the update of input (nf_fread*d
) routines, I noticed that we were processing the ghost points during reading in distributed-memory by setting the local variable Nghost
IF (model.eq.iADM) THEN
This is completely removed now and Nghost=0
. That is, the tile ghost points are not filled during reading. These points are processed elsewhere with the appropriate calls to the distributed-memory exchange routines in mp_exchange.F
With parallel I/O there are many combinations of NetCDF
libraries that may need to be built. The build scripts build.sh
have changed accordingly. You may build NetCDF-3
libraries in serial. If creating new NetCDF-4
file format (NETCDF4
C-preprocessing option) you may need serial versions of the NetCDF4/HDF5
libraries. This is the case in serial and shared-memory ROMS applications. If you want parallel I/O, you need to compile NetCDf4/HDF5
with the MPI
library as explained below.Parallel I/O in ROMS: Warning:
script is different for MPICH
, and OpenMPI
implementations of MPI
need to use the appropriate script HDF5 and NetCDF-4 compiling notes:
- As mentioned above, we have been unable to get the Fortran 90 interface of HDF5/NetCDF4 to work with OpenMPI. We have even tried the latest OpenMPI 1.3 that was release January 19, 2009. Unfortunately, all the parallel tests that come with NetCDF4 are C codes and thus don't catch this error.
- At this time we suggest you use MPICH2 to compile HDF5, NetCDF4 and ROMS. If you discover other parallel compilers that work, please let us know.
- You will need a separate HDF5 and NetCDF4 library for each compiler/mpi combination you want to use. For example, if you want to be able to run in serial and parallel you will need a separate serial version of HDF5 and NetCDF4. If you have 2 different MPI implementations with the same compiler (i.e. MPICH2 and MVAPICH2 for your PGI compler), you will need to compile a separate HDF5 and NetCDF4 library for each MPI implementation.
- When configuring HDF5, parallel libraries will be built automatically if your CC environment variable is set to mpicc. If your parallel compiler has a non-standard name, you will probably need to use the --enable-parallel flag when configuring. NetCDF4's configure script will recognize that HDF5 was built for parallel I/O and turn on NetCDF4's parallel I/O features.
- When configuring NetCDF4 you MUST include the --enable-netcdf4 option to build the version 4 interface (including parallel IO) for NetCDF.
- We had a lot of problems compiling NetCDF 4.0 and NetCDF 4.0.1-beta2 releases and ended up using the daily snapshot instead. We are using the snapshot from January 12th 2009.
- If building with gfortran, g95, pgi, or ifort (and possibly others), it is important to set the CPPFLAG -DpgiFortran for HDF5 and NetCDF4 or you will get name mismatches within the resulting libraries
Please use ROMS parallel I/O wisely. Do not overkill by running relatively small grids with a lot of CPU processes. This will decrease the efficiency due to the overhead of MPI communications. Recall that there is always an optimal number of processes for a particular application
Let's start testing a documenting the gains of using parallel I/O in different architectures. Please post your results in this forum.
There is an overhead at the beginning when the output NetCDF files are created. This is due to several of the scalars and application parameters that are defined by def_info.F
and written by wrt_info.F
. I am not too worried about this right now because this only happens when the file is created. I am investigating writing these into a structure or group. Notice that structures and groups are part of HDF
. Processing structures is easy in C but more complicated in Fortran-90. My problem with structures is that we need to know its size in bytes in advance. The groups look promising, check NetCDF-4
new groups, compound and user derived types for more details. By the way, the NetCDF-4
file format is actually an HDF
file with NetCDF
self-describing metadata design.