How to run ROMS.( MPI + OPENMP )

[kissrang]

Hello, I am choi.
I am working on a project for faster running of the ROMS.
I would like to work ROMS using the Xeon Phi. (1 CPU, 8 Xoon Phi.)

But I found that only one of them is possible using MPI or OPENMP.
To use the CPU and Xeon Phi effectively use the MPI and OPENMP at the same time.
So I want to use MPI and OPENMP at the same time.

Modify the contents in the makefile.

in makefile:

#ifdef USE_MPI
# ifdef USE_OpenMP
# $(error You cannot activate USE_MPI and USE_OpenMP at the same time!)
# endif
#endif

And I run after the compile.

The compilation was successful.

compile log:
--------------------------------
.
.
.
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 set_zeta.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/step3d_t.F > Build/step3d_t.f90
ROMS/Bin/cpp_clean Build/step3d_t.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/mpdata_adiff.F > Build/mpdata_adiff.f90
ROMS/Bin/cpp_clean Build/mpdata_adiff.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 mpdata_adiff.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 step3d_t.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/step3d_uv.F > Build/step3d_uv.f90
ROMS/Bin/cpp_clean Build/step3d_uv.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 step3d_uv.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/wvelocity.F > Build/wvelocity.f90
ROMS/Bin/cpp_clean Build/wvelocity.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 wvelocity.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 main3d.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/output.F > Build/output.f90
ROMS/Bin/cpp_clean Build/output.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 output.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Nonlinear/set_data.F > Build/set_data.f90
ROMS/Bin/cpp_clean Build/set_data.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 set_data.f90
ar r Build/libNLM.a Build/bbl.o Build/bc_2d.o Build/bc_3d.o Build/bc_bry2d.o Build/bc_bry3d.o Build/bulk_flux.o Build/bvf_mix.o Build/conv_2d.o Build/conv_3d.o Build/conv_bry2d.o Build/conv_bry3d.o Build/diag.o Build/exchange_2d.o Build/exchange_3d.o Build/forcing.o Build/frc_adjust.o Build/get_data.o Build/get_idata.o Build/gls_corstep.o Build/gls_prestep.o Build/hmixing.o Build/ini_fields.o Build/initial.o Build/interp_floats.o Build/lmd_bkpp.o Build/lmd_skpp.o Build/lmd_swfrac.o Build/lmd_vmix.o Build/main2d.o Build/main3d.o Build/mpdata_adiff.o Build/my25_corstep.o Build/my25_prestep.o Build/nesting.o Build/obc_adjust.o Build/obc_volcons.o Build/omega.o Build/output.o Build/pre_step3d.o Build/prsgrd.o Build/radiation_stress.o Build/rho_eos.o Build/rhs3d.o Build/set_avg.o Build/set_data.o Build/set_depth.o Build/set_massflux.o Build/set_tides.o Build/set_vbc.o Build/set_zeta.o Build/step2d.o Build/step3d_t.o Build/step3d_uv.o Build/step_floats.o Build/t3dbc_im.o Build/t3dmix.o Build/tkebc_im.o Build/u2dbc_im.o Build/u3dbc_im.o Build/uv3dmix.o Build/v2dbc_im.o Build/v3dbc_im.o Build/vwalk_floats.o Build/wetdry.o Build/wvelocity.o Build/zetabc.o
ar: creating Build/libNLM.a
ranlib Build/libNLM.a
ar r Build/libNLM_bio.a Build/biology.o Build/biology_floats.o
ar: creating Build/libNLM_bio.a
ranlib Build/libNLM_bio.a
ar r Build/libNLM_sed.a Build/sed_bed.o Build/sed_bedload.o Build/sed_fluxes.o Build/sediment.o Build/sed_settling.o Build/sed_surface.o
ar: creating Build/libNLM_sed.a
ranlib Build/libNLM_sed.a
ar r Build/libANA.a Build/analytical.o
ar: creating Build/libANA.a
ranlib Build/libANA.a
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' ROMS/Modules/mod_arrays.F > Build/mod_arrays.f90
ROMS/Bin/cpp_clean Build/mod_arrays.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 mod_arrays.f90
ar r Build/libMODS.a Build/mod_arrays.o Build/mod_average.o Build/mod_bbl.o Build/mod_behavior.o Build/mod_biology.o Build/mod_boundary.o Build/mod_clima.o Build/mod_coupler.o Build/mod_coupling.o Build/mod_diags.o Build/mod_eclight.o Build/mod_eoscoef.o Build/mod_floats.o Build/mod_forces.o Build/mod_fourdvar.o Build/mod_grid.o Build/mod_iounits.o Build/mod_kinds.o Build/mod_mixing.o Build/mod_ncparam.o Build/mod_nesting.o Build/mod_netcdf.o Build/mod_ocean.o Build/mod_parallel.o Build/mod_param.o Build/mod_scalars.o Build/mod_sedbed.o Build/mod_sediment.o Build/mod_sources.o Build/mod_stepping.o Build/mod_storage.o Build/mod_strings.o Build/mod_tides.o
ar: creating Build/libMODS.a
ranlib Build/libMODS.a
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' Master/esmf_roms.F > Build/esmf_roms.f90
ROMS/Bin/cpp_clean Build/esmf_roms.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 esmf_roms.f90
/usr/bin/cpp -P -traditional -DMPI -D_OPENMP -DLINUX -DX86_64 -DIFORT -D'ROOT_DIR="/home/test1/ROMS_0818/omp+mpi"' -DUPWELLING -D'HEADER="upwelling.h"' -D'ROMS_HEADER="upwelling.h"' -DNestedGrids= -D'ANALYTICAL_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Functionals"' -D'SVN_REV="756M"' -IROMS/Include -IROMS/Nonlinear -IROMS/Nonlinear/Biology -IROMS/Nonlinear/Sediment -IROMS/Utility -IROMS/Drivers -IROMS/Functionals -IMaster -ICompilers -D'HEADER_DIR="/home/test1/ROMS_0818/omp+mpi/ROMS/Include"' Master/master.F > Build/master.f90
ROMS/Bin/cpp_clean Build/master.f90
cd Build; /opt/intel/impi/5.0.3.048/bin64/mpiifort -c -heap-arrays -fp-model precise -openmp -fpp -ip -O3 master.f90
/opt/intel/impi/5.0.3.048/bin64/mpiifort -heap-arrays -fp-model precise -openmp -fpp -ip -O3 Build/esmf_roms.o Build/master.o Build/ocean_control.o Build/ocean_coupler.o Build/propagator.o Build/roms_export.o Build/roms_import.o -o oceanO Build/libUTIL.a Build/libNLM.a Build/libNLM_bio.a Build/libNLM_sed.a Build/libANA.a Build/libUTIL.a Build/libMODS.a -L/home/netcdf/lib -lnetcdf -lmpifort
rm -f -r /home/test1/make_macros.mk
---------------------------------------------

oceanO file has been created.
Come to run, but the following error has occurred.

NtileI == 4 ! I-direction partition
NtileJ == 3 ! J-direction partition

~>$ mpirun -n 12 -env OMP_NUM_THREADS=2 ./oceanO ROMS/External/ocean_upwelling.in
Error log:
-----------------------------------------------
Process Information:

Node # 0 (pid= 9279) is active.
Node # 1 (pid= 9280) is active.
Node # 2 (pid= 9281) is active.
Node # 4 (pid= 9283) is active.
Node # 5 (pid= 9284) is active.
Node # 6 (pid= 9285) is active.
Node # 7 (pid= 9286) is active.
Node # 8 (pid= 9287) is active.
Node # 9 (pid= 9288) is active.
Node # 10 (pid= 9289) is active.
Node # 11 (pid= 9290) is active.
Node # 3 (pid= 9282) is active.
Node # 3 (pid= 9282) is active.
Node # 0 (pid= 9279) is active.
Node # 1 (pid= 9280) is active.
Node # 2 (pid= 9281) is active.
Node # 4 (pid= 9283) is active.
Node # 5 (pid= 9284) is active.
Node # 6 (pid= 9285) is active.
Node # 7 (pid= 9286) is active.
Node # 8 (pid= 9287) is active.
Node # 9 (pid= 9288) is active.
Node # 10 (pid= 9289) is active.
Node # 11 (pid= 9290) is active.
forrtl: severe (174): SIGSEGV, segmentation fault occurred
forrtl: severe (174): SIGSEGV, segmentation fault occurred
-----------------------------------------------

And it ran the other way.

NtileI == 4 ! I-direction partition
NtileJ == 6 ! J-direction partition

~>$ mpirun -n 12 -env OMP_NUM_THREADS=2 ./oceanO ROMS/External/ocean_upwelling.in
Error log:
-----------------------------------------------
Elapsed CPU time (seconds):


ROMS/TOMS - Output NetCDF summary for Grid 01:

ROMS/TOMS - Partition error ......... exit_flag: 6


ERROR: Illegal domain partition.
-----------------------------------------------

How can you be running to run the MPI and OPENMP at the same time?
I need help. Please advise.

[kate]

It is not possible with the myroms.org version of ROMS without major changes (i.e., a complete rewrite). I know Sasha's code supports mixed mode, so perhaps the French ROMS_Agrif would too. It's worth looking into.

[kissrang]

Thank you very much kate.
It was a great help.

I'll ask a few more.

I received a ROMS_Agrif-down version.
Can I have inputData compatible with? If not, I think it will be hard to use.

And you've told me that "I know Sasha's code supports mixed mode",
Where can I get Sasha's code go?

Thank you !!!!

[kate]

It wants one and only one forcing file, which is a problem for my domains. There are Matlab tools for generating files for ROMS_Agrif, but I haven't used them.

You would have to ask Sasha for his code. You might not be successful in getting a reply.