mpi error on SGI with the latest ROMS

Discussion on computers, ROMS installation and compiling

Moderators: arango, robertson

Post Reply
Message
Author
isopycnal
Posts: 15
Joined: Fri Jan 05, 2007 3:36 pm
Location: Ocean institution,Chinese Academy of Sciences

mpi error on SGI with the latest ROMS

#1 Unread post by isopycnal »

hi,everyone,
I compiled the latest ROMS on SGI linux system with MPI mode and the compilation is OK. however, when I excute ROMS with mpirun, such as

Code: Select all

mpirun -np 12 ./oceanM ./ROMS/External/liuzl_test.in
I got the following errors

Code: Select all

MPI: On host a3700, Program /lun/home/liuzl/latest_roms/oceanM, Rank 11, Process 22290 received signal SIGSEGV(11)

MPI: --------stack traceback-------
sh: line 1: idb: command not found

MPI: -----stack traceback ends-----
MPI: On host a3700, Program /lun/home/liuzhiliang/latest_roms/oceanM, Rank 11, Process 22290: Dumping core on signal SIGSEGV(11) into directory /lun/home/liuzhiliang/latest_roms
MPI: MPI_COMM_WORLD rank 11 has terminated without calling MPI_Finalize()
MPI: aborting job
MPI: Received signal 11
but when I set the debug on in my makefile, the compilation and excution (using oceanG) both have no any problems. I don't know what it is going on. what is the problem when using the oceanM in mpi mode?
thanks in advance!!

gerardo
Posts: 12
Joined: Wed Sep 27, 2006 7:23 pm
Location: SGI

#2 Unread post by gerardo »

isopycnal wrote:
> [...]
> MPI: On host a3700, Program /lun/home/liuzl/latest_roms/oceanM, Rank 11, Process 22290 received signal SIGSEGV(11)
> [...]

SIGSEGV(11) occurs for one of two reasons: a bad memory access
(such as using out-of-range indices or a bad pointer), or trying to grow
the stack beyond the user limit. Since several ROMS applications
have been built and run successfully on SGI Altix systems, it is
more likely that the problem is insufficient stack. What does the
'limit' or 'ulimit -a' command show? (The command that works will
depend on the shell you use.)

Saludos,

Gerardo

isopycnal
Posts: 15
Joined: Fri Jan 05, 2007 3:36 pm
Location: Ocean institution,Chinese Academy of Sciences

#3 Unread post by isopycnal »

Hi, Gerardo
thank you for your reply. when I typed 'ulimit -a' , it shows that

Code: Select all

core file size        (blocks, -c) 0
data seg size         (kbytes, -d) unlimited
file size             (blocks, -f) unlimited
max locked memory     (kbytes, -l) unlimited
max memory size       (kbytes, -m) unlimited
open files                    (-n) 1024
pipe size          (512 bytes, -p) 8
stack size            (kbytes, -s) unlimited
cpu time             (seconds, -t) unlimited
max user processes            (-u) 7675
virtual memory        (kbytes, -v) unlimited
I discussed this proble with the SGi administrator, and he don't know the reason either. but, when I tried to use fewer cpus(originally 16 cpus, now 8 cpus), it works using oceanM. Is this the reason for the error? I really don't know too much on the linux system.

gerardo
Posts: 12
Joined: Wed Sep 27, 2006 7:23 pm
Location: SGI

#4 Unread post by gerardo »

when I tried to use fewer cpus(originally 16 cpus, now 8 cpus), it works using oceanM. Is this the reason for the error? I really don't know too much on the linux system.
I don't understand why the error would occur with *fewer*
processors, since in that case one would expect the application
to require more memory per process. I don't see anything
problematic with your user limits. What is the setting of
ROMS_APPLICATION in your makefile? Would you like me
to try building and running your test case?

Saludos,

Gerardo

isopycnal
Posts: 15
Joined: Fri Jan 05, 2007 3:36 pm
Location: Ocean institution,Chinese Academy of Sciences

#5 Unread post by isopycnal »

hi,gerardo.thank you very much if you can do the similar test.
I am running a case to simulate the seasonal evolution of the circulations in the east China Sea. My study area is [105-135E,15-42N), dl=1/6 degree. And the forcing fields are from the coads in the ROMS package.In the makefile, I only set the MPI on and using the ifort compiler. And my cpp definitions are as belows

Code: Select all

#define ANA_SSFLUX
#define ANA_BTFLUX
#define ANA_BSFLUX
#define UV_ADV
#define UV_COR
#define UV_QDRAG
#define  UV_VIS2
#define  MIX_S_UV
#define  VISC_GRID
#define  TS_DIF2
#undef BULK_FLUXES
#define  MIX_ISO_TS
#define PJ_GRADPQ2
#define SOLVE3D
#define SALINITY
#define NONLIN_EOS
#define CURVGRID
#define MASKING
#define SPLINES
#define SPONGE
#define WESTERN_WALL
#define EAST_FSCHAPMAN
#define EAST_FSNUDGING
#define EAST_M2FLATHER
#define EAST_M3RADIATION
#define EAST_TRADIATION
#undef EAST_M2NUDGING
#define EAST_TNUDGING
#define SOUTH_FSCHAPMAN
#define SOUTH_FSNUDGING
#define SOUTH_M2FLATHER
#define SOUTH_M3RADIATION
#define SOUTH_TRADIATION
#undef SOUTH_M2NUDGING
#define SOUTH_TNUDGING
#define NORTH_FSCHAPMAN
#define NORTH_FSNUDGING
#define NORTH_M2FLATHER
#define NORTH_M3RADIATION
#define NORTH_TRADIATION
#undef NORTH_M2NUDGING
#define NORTH_TNUDGING
#define TS_U3HADVECTION
#define TS_C4VADVECTION
#define TCLIMATOLOGY
#define M2CLIMATOLOGY
#define ZCLIMATOLOGY
#define ZCLM_NUDGING
#define M2CLM_NUDGING
#define M3CLIMATOLOGY
#define M3CLM_NUDGING
#define TCLM_NUDGING
And my .in file are

Code: Select all

! Application title.

       TITLE = liuzl_test

! C-preprocessing Flag.

    MyAppCPP = 

! Input variable information file name.  This file needs to be processed
! first so all information arrays can be initialized properly.

     VARNAME = ROMS/External/varinfo.dat

! Grid dimension parameters. See notes below in the Glossary for how to set
! these parameters correctly.

          Lm ==  179         ! Number of I-direction INTERIOR RHO-points
          Mm ==  186          ! Number of J-direction INTERIOR RHO-points
           N == 10            ! Number of vertical levels

        Nbed =  0             ! Number of sediment bed layers

         NAT =  2             ! Number of active tracers (usually, 2)
         NPT =  0             ! Number of inactive passive tracers
         NCS =  0             ! Number of cohesive (mud) sediment tracers
         NNS =  0             ! Number of non-cohesive (sand) sediment tracers

! Domain decomposition parameters for serial, distributed-memory or
! shared-memory configurations used to determine tile horizontal range
! indices (Istr,Iend) and (Jstr,Jend), [1:Ngrids].

      NtileI ==  2                             ! I-direction partition
      NtileJ == 4                               ! J-direction partition

! Time-Stepping parameters.

      NTIMES == 10104000
          DT == 300.0d0
     NDTFAST == 10

! Model iteration loops parameters.

       ERstr =  1
       ERend =  1
      Nouter =  1
      Ninner =  1
  Nintervals =  1

! Number of eigenvalues (NEV) and eigenvectors (NCV) to compute for the
! Lanczos/Arnoldi problem in the Generalized Stability Theory (GST)
! analysis. NCV must be greater than NEV (see documentation below).

         NEV =  2                             ! Number of eigenvalues
         NCV =  10                              ! Number of eigenvectors

! Input/Output parameters.

       NRREC == 0
   LcycleRST == T
        NRST == 5400
        NSTA == 1
        NFLT == 1
       NINFO == 1

! Output history, average, diagnostic files parameters.

     LDEFOUT == T
        NHIS == 1440
     NDEFHIS == 8640
      NTSAVG == 1440
        NAVG == 8640
     NDEFAVG == 1440
      NTSDIA == 8640
        NDIA == 1440
     NDEFDIA == 8640

! Output tangent linear and adjoint models parameters.

   LcycleTLM == F
        NTLM == 1080
     NDEFTLM == 0
   LcycleADJ == F
        NADJ == 1080
     NDEFADJ == 0

! Output check pointing GST restart parameters.

     LrstGST =  F                               ! GST restart switch
  MaxIterGST =  500                             ! maximun number of iterations
        NGST =  10                              ! check pointing interval

! Relative accuracy of the Ritz values computed in the GST analysis.

    Ritz_tol =  1.0d-15

! Harmonic/biharmonic horizontal diffusion of tracer: [1:NAT+NPT,Ngrids].

        TNU2 == 0.0d0    0.0d0                 ! m2/s
        TNU4 == 0.0d0    0.0d0                  ! m4/s

! Harmononic/biharmonic, horizontal viscosity coefficient: [Ngrids].

       VISC2 == 0.0d0                          ! m2/s
       VISC4 == 0.0d0                         ! m4/s

! Vertical mixing coefficients for active tracers: [1:NAT+NPT,Ngrids]

     AKT_BAK == 1.0d-6 1.0d-6                     ! m2/s

! Vertical mixing coefficient for momentum: [Ngrids].

     AKV_BAK == 1.0d-5                           ! m2/s

! Turbulent closure parameters.

     AKK_BAK == 5.0d-6                          ! m2/s
     AKP_BAK == 5.0d-6                          ! m2/s
      TKENU2 == 0.0d0                           ! m2/s
      TKENU4 == 0.0d0                           ! m4/s

! Generic length-scale turbulence closure parameters.

       GLS_P == 3.0d0                           ! K-epsilon
       GLS_M == 1.5d0
       GLS_N == -1.0d0
    GLS_Kmin == 7.6d-6
    GLS_Pmin == 1.0d-12

    GLS_CMU0 == 0.5477d0
      GLS_C1 == 1.44d0
      GLS_C2 == 1.92d0
     GLS_C3M == -0.4d0
     GLS_C3P == 1.0d0
    GLS_SIGK == 1.0d0
    GLS_SIGP == 1.30d0

! Constants used in surface turbulent kinetic energy flux computation.

  CHARNOK_ALPHA == 1400.0d0         ! Charnok surface roughness
 ZOS_HSIG_ALPHA == 0.5d0            ! roughness from wave amplitude
       SZ_ALPHA == 0.25d0           ! roughness from wave dissipation
      CRGBAN_CW == 100.0d0          ! Craig and Banner wave breaking

! Constants used in momentum stress computation.

        RDRG == 2.65d-04                   ! m/s
       RDRG2 == 3.0d-03                    ! nondimensional
         Zob == 0.02d0                     ! m
         Zos == 0.02d0                     ! m

! Height (m) of atmospheric measurements for Bulk fluxes parameterization.

      BLK_ZQ == 10.0d0                     ! air humidity
      BLK_ZT == 10.0d0                     ! air temperature
      BLK_ZW == 10.0d0                     ! winds

! Minimum depth for wetting and drying.

       DCRIT == 0.10d0                     ! m

! Various parameters.

       WTYPE == 3
     LEVSFRC == 10
     LEVBFRC == 10

! Vertical S-coordinates parameters, [1:Ngrids].

     THETA_S == 3.0d0                      ! 0 < THETA_S < 20
     THETA_B == 0.2d0                      ! 0 < THETA_B <1> 1, nudging on inflow is stronger
! than on outflow (recommended).

      OBCFAC == 20.0d0                      ! nondimensional

! Linear equation of State parameters:

         R0 == 1025.0d0                   ! kg/m3
          T0 == 15.0d0                     ! Celsius
          S0 == 35.0d0                     ! PSU
       TCOEF == 1.7d-4                     ! 1/Celsius
       SCOEF == 7.6d-4                     ! 1/PSU

! Slipperiness parameter: 1.0 (free slip) or -1.0 (no slip)

      GAMMA2 == 1.0d0    ! used for closed wall boundary condition

! Starting (DstrS) and ending (DendS) day for adjoint sensitivity forcing.
! DstrS must be less or equal to DendS. If both values are zero, their
! values are reset internally to the full range of the adjoint integration.

       DstrS == 0.0d0                      ! starting day
       DendS == 0.0d0                      ! ending day

! Starting and ending vertical levels of the 3D adjoint state variables
! whose sensitivity is required.

       KstrS == 1                          ! starting level
       KendS == 1                          ! ending level

! Logical switches (TRUE/FALSE) to specify the adjoint state variables
! whose sensitivity is required.

Lstate(isFsur) == F                        ! free-surface
Lstate(isUbar) == F                        ! 2D U-momentum
Lstate(isVbar) == F                        ! 2D V-momentum
Lstate(isUvel) == F                        ! 3D U-momentum
Lstate(isVvel) == F                        ! 3D V-momentum

! Logical switches (TRUE/FALSE) to specify the adjoint state tracer
! variables whose sensitivity is required (NT values are expected).

Lstate(isTvar) == F F                      ! tracers

! Stochastic optimals time decorrelation scale (days) assumed for
! red noise processes.

    SO_decay == 2.0d0                      ! days

! Logical switches (TRUE/FALSE) to specify the state surface forcing
! variable whose stochastic optimals is required.

SOstate(isUstr) == T                      ! surface u-stress
SOstate(isVstr) == T                       ! surface v-stress

! Logical switches (TRUE/FALSE) to specify the surface tracer forcing
! variable whose stochastic optimals is required (NT values are expected).

SOstate(isTsur) == T T                     ! surface tracer flux

! Stochastic optimals surface forcing standard deviation for
! dimensionalization.

SO_sdev(isUstr) == 1.0d0                   ! surface u-stress
SO_sdev(isVstr) == 1.0d0                   ! surface v-stress
SO_sdev(isTsur) == 1.0d0 1.0d0             ! NT surface tracer flux

! Logical switches (TRUE/FALSE) to activate writing of fields into
! HISTORY output file.

Hout(idUvel) == T                          ! 3D U-velocity
Hout(idVvel) == T                          ! 3D V-velocity
Hout(idWvel) == T                          ! 3D W-velocity
Hout(idOvel) == F                          ! omega vertical velocity
Hout(idUbar) == T                          ! 2D U-velocity
Hout(idVbar) == T                          ! 2D V-velocity
Hout(idFsur) == T                          ! free-surface
Hout(idBath) == F                          ! time-dependent bathymetry

Hout(idTvar) == T T                        ! temperature and salinity

Hout(idUsms) == F                          ! surface U-stress
Hout(idVsms) == F                          ! surface V-stress
Hout(idUbms) == F                          ! bottom U-stress
Hout(idVbms) == F                          ! bottom V-stress

Hout(idUbrs) == F                          ! bottom U-current stress
Hout(idVbrs) == F                          ! bottom V-current stress
Hout(idUbws) == F                          ! bottom U-wave stress
Hout(idVbws) == F                          ! bottom V-wave stress
Hout(idUbcs) == F                          ! bottom max wave-current U-stress
Hout(idVbcs) == F                          ! bottom max wave-current V-stress

Hout(idUbot) == F                          ! bed wave orbital U-velocity
Hout(idVbot) == F                          ! bed wave orbital V-velocity
Hout(idUbur) == F                          ! bottom U-velocity above bed
Hout(idVbvr) == F                          ! bottom V-velocity above bed

Hout(idW2xx) == F                          ! 2D radiation stress, Sxx component
Hout(idW2xy) == F                          ! 2D radiation stress, Sxy component
Hout(idW2yy) == F                          ! 2D radiation stress, Syy component
Hout(idU2rs) == F                          ! 2D radiation U-stress
Hout(idV2rs) == F                          ! 2D radiation V-stress
Hout(idU2Sd) == F                          ! 2D U-Stokes velocity
Hout(idV2Sd) == F                          ! 2D V-Stokes velocity

Hout(idW3xx) == F                          ! 3D radiation stress, Sxx component
Hout(idW3xy) == F                          ! 3D radiation stress, Sxy component
Hout(idW3yy) == F                          ! 3D radiation stress, Syy component
Hout(idW3zx) == F                          ! 3D radiation stress, Szx component
Hout(idW3zy) == F                          ! 3D radiation stress, Szy component
Hout(idU3rs) == F                          ! 3D U-radiation stress
Hout(idV3rs) == F                          ! 3D V-radiation stress
Hout(idU3Sd) == F                          ! 3D U-Stokes velocity
Hout(idV3Sd) == F                          ! 3D V-Stokes velocity

Hout(idWamp) == F                          ! wave height
Hout(idWlen) == F                          ! wave length
Hout(idWdir) == F                          ! wave direction

Hout(idTsur) == F F                        ! surface net heat and salt flux
Hout(idLhea) == F                          ! latent heat flux
Hout(idShea) == F                          ! sensible heat flux
Hout(idLrad) == F                          ! longwave radiation flux
Hout(idSrad) == F                          ! shortwave radiation flux
Hout(idevap) == F                          ! evaporation rate
Hout(idrain) == F                          ! precipitation rate

Hout(idDano) == F                          ! density anomaly
Hout(idVvis) == F                          ! vertical viscosity
Hout(idTdif) == F                          ! vertical T-diffusion
Hout(idSdif) == F                          ! vertical Salinity diffusion
Hout(idHsbl) == F                          ! depth of surface boundary layer
Hout(idHbbl) == F                          ! depth of bottom boundary layer
Hout(idMtke) == F                          ! turbulent kinetic energy
Hout(idMtls) == F                          ! turbulent length scale

! Logical switches (TRUE/FALSE) to activate writing of extra inert passive
! tracers other than biological and sediment tracers. An inert passive tracer
! is one that it is only advected and diffused. Other processes are ignored.
! These tracers include, for example, dyes, pollutants, oil spills, etc.
! NPT values are expected. However, these switches can be activated using
! compact parameter specification.

 Hout(inert) == F                          ! inert passive tracers

! Logical switches (TRUE/FALSE) to activate writing of exposed sediment
! layer properties into HISTORY output file.  Currently, MBOTP properties
! are expected for the bottom boundary layer and/or sediment models:
!
!   Hout(idBott(isd50)),  isd50 = 1        ! mean grain diameter
!   Hout(idBott(idens)),  idens = 2        ! mean grain density
!   Hout(idBott(iwsed)),  iwsed = 3        ! mean settling velocity
!   Hout(idBott(itauc)),  itauc = 4        ! critical erosion stress
!   Hout(idBott(irlen)),  irlen = 5        ! ripple length
!   Hout(idBott(irhgt)),  irhgt = 6        ! ripple height
!   Hout(idBott(ibwav)),  ibwav = 7        ! wave excursion amplitude
!   Hout(idBott(izdef)),  izdef = 8        ! default bottom roughness
!   Hout(idBott(izapp)),  izapp = 9        ! apparent bottom roughness
!   Hout(idBott(izNik)),  izNik = 10       ! Nikuradse bottom roughness
!   Hout(idBott(izbio)),  izbio = 11       ! biological bottom roughness
!   Hout(idBott(izbfm)),  izbfm = 12       ! bed form bottom roughness
!   Hout(idBott(izbld)),  izbld = 13       ! bed load bottom roughness
!   Hout(idBott(izwbl)),  izwbl = 14       ! wave bottom roughness
!   Hout(idBott(iactv)),  iactv = 15       ! active layer thickness
!   Hout(idBott(ishgt)),  ishgt = 16       ! saltation height
!
!                                 1 1 1 1 1 1 1
!               1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6

Hout(idBott) == F F F F F F F F F F F F F F F F

! Generic User parameters, [1:NUSER].

       NUSER =  0
        USER =  0.d0

! Input NetCDF file names, [1:Ngrids].

     GRDNAME == liuzl_grd_test_4.nc
     ININAME == liuzl_ini_test_4.nc
     ITLNAME == 
     IRPNAME == 
     IADNAME == 
     CLMNAME == liuzl_clm_test_4.nc
     BRYNAME == liuzl_bry_test_4.nc
     FWDNAME == 
     ADSNAME == 

! Input forcing NetCDF file name(s).  The USER has the option to enter
! several files names per each nested grid.  For example, the USER may
! have a different files for wind products, heat fluxes, rivers, tides,
! etc.  The model will scan the file list and will read the needed data
! from the first file in the list containing the forcing field. Therefore,
! the order of the file names is very important.  If multiple forcing
! files per grid, enter first all the file names for grid 1, then grid 2,
! and so on.  Use a single line per entry with a continuation (\) symbol
! at the each entry, except the last one.

     NFFILES == 1                          ! number of forcing files

     FRCNAME == liuzl_frc_test_4.nc               ! forcing file 1, grid 1

! Output NetCDF file names, [1:Ngrids].

     GSTNAME == ocean_gst.nc
     RSTNAME == ocean_rst.nc
     HISNAME == ocean_his1.nc
     TLMNAME == ocean_tlm.nc
     TLFNAME == ocean_tlf.nc
     ADJNAME == ocean_adj.nc
     AVGNAME == ocean_avg.nc
     DIANAME == ocean_dia.nc
     STANAME == ocean_sta.nc
     FLTNAME == ocean_flt.nc

! Input ASCII parameter filenames.
by the way, I am very glad if someone can give some suggestions or comments on the settings. Thank you very much in advance.

gerardo
Posts: 12
Joined: Wed Sep 27, 2006 7:23 pm
Location: SGI

#6 Unread post by gerardo »

isopycnal wrote:
thank you very much if you can do the similar test.
I will be glad to run your test case, but I will need to have
copies of your .nc input files. Next week I will be traveling
to Europe (to visit an institution where ROMS is also used),
but the following week I should be able to build and run
your test case on one of our systems. Please e-mail me
(gerardo@sgi.com) so I can send you the details of an ftp
server where you can put the files I will need.

Saludos,

Gerardo

Post Reply