ROMS on XSEDE systems

Post ROMS benchmark results

Moderators: arango, robertson

Post Reply
Message
Author
User avatar
wilkin
Posts: 668
Joined: Mon Apr 28, 2003 5:44 pm
Location: Rutgers University
Contact:

ROMS on XSEDE systems

#1 Post by wilkin »

Does anyone have any recent experience running ROMS on one of the XSEDE (https://www.xsede.org/home Extreme Science and Engineering Discovery Environment - the successor to TERAGRID) high performance computing systems?

I'd appreciate hearing any advice on what hosts have proven useful for running long, modest sized ROMS applications, and whether users encountered any particular difficulties building and running the model, and experience with the bandwidth to these hosts if there a large amount of data to return.

I'm looking at requesting an allocation on host "stampede" at UT Austin.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu

kchen
Posts: 2
Joined: Wed Nov 02, 2011 5:50 am
Location: WHOI

Re: ROMS on XSEDE systems

#2 Post by kchen »

John,

No problem building ROMS on Stampede, and it's fairly fast (IfinityBand FDR, 16-processor/node Xeon E5 (sandy bridge),pretty good I/O hardware ).

Quota of /work directory is 1T. No limit writing on the purged-storage /scratch (7.5P).

However, the max runtime is only 48-hr. The other major downside is that under heavy-load, the wait could be long. It easily takes hours before the job gets running.

Ke

vincentcombesOSU
Posts: 3
Joined: Fri Oct 21, 2011 2:14 am
Location: OSU

Re: ROMS on XSEDE systems

#3 Post by vincentcombesOSU »

Dear John and Ke,

it seems that you were able to run ROMS on stampede from utexas (although your posts are from 2014)
I do have a question for you about this.


While I was able to get ROMS run there (now stampede2) using openmp, I am having trouble making it run using mpi.
The compiling does not complain (using ifort / mpfi90 / mvapich2)
When I run however I get the following errors (depending on the mpfi90 flags):

forrtl: severe (194): Run-Time Check Failure. The variable 'distribute_mod_mp_mp_reduce_1d_$SERROR' is being used in 'distribute.f90(4812,19)' without being defined
when I use the following fflags:
mpif90 -c -fp-model precise -heap-arrays -ip -O3 -traceback -check uninit

or

Fatal error in PMPI_Bcast: Message truncated, error stack:
PMPI_Bcast(1645)............: MPI_Bcast(buf=0x2b6bf2e372e0, count=20, MPI_DOUBLE_PRECISION, root=0, MPI_COMM_WORLD) failed
when I use the following fflags:
mpif90 -c -fp-model precise -heap-arrays -ip -O3


I am guessing that I cannot run it because I don't have the right flags for mpi.


If you were able to run it there using mpi, could you please share your Linux-ifort.mk ?
Thank you

Or any other tips would be great

Vincent

vincentcombesOSU
Posts: 3
Joined: Fri Oct 21, 2011 2:14 am
Location: OSU

Re: ROMS on XSEDE systems

#4 Post by vincentcombesOSU »

Just found what I was missing.

I did not have defined REDUCE_ALLREDUCE

everything is fine now :)

Post Reply