Speed-up advice?

Post ROMS benchmark results

Moderators: arango, robertson

Post Reply
Message
Author
User avatar
kate
Posts: 3718
Joined: Wed Jul 02, 2003 5:29 pm
Location: IMS/UAF, USA

Speed-up advice?

#1 Post by kate » Tue Nov 19, 2019 9:12 pm

We are hoping to have a long simulation of a Pan-Arctic grid with a BGC model. My colleagues have therefore obtained a computing grant at NCAR.
I have done a few timings of a day-long simulation on Cheyenne. I'm finding it no faster than a not-so-new Penguin Supercomputing cluster at home:

On 216 cores: 1.68 hours (6x36 tiling)

On 432 cores: 1.0 hours (8x48 tiling)

Compared to Chinook:

On 192 cores: 1.71 hours (6x32 tiling)

This is a large domain using the ROMS model with a complex BGC component, such that the tracer timestepping is over 65% of the run time. I'm using TS_MPDATA because otherwise I get instabilities in the BGC fields at the open boundaries.

I believe we are limited by the memory bandwidth of the system. This is a very large simulation, taking 340 GB for the 216 cores and 530 GB for the 432 cores (according to the internal reporting). Do you have any advice for ways to possibly speed this thing up on Cheyenne?

jcwarner
Posts: 816
Joined: Wed Dec 31, 2003 6:16 pm
Location: USGS, USA

Re: Speed-up advice?

#2 Post by jcwarner » Wed Nov 20, 2019 7:32 pm

we have been using TS_HSIMT in COAWST. it is a pos def scheme, faster than mpdata. would be nice to know how the tracer advection compares between the 2 schemes. Have you tried different tiling? how many cores / node does it have?

User avatar
kate
Posts: 3718
Joined: Wed Jul 02, 2003 5:29 pm
Location: IMS/UAF, USA

Re: Speed-up advice?

#3 Post by kate » Thu Nov 21, 2019 7:28 am

Thanks, John, I'd forgotten about that TS_HSIMT option. I'll give it a whirl.

I haven't tried other tilings. I heard from their consultants saying the two systems are remarkably similar so they would expect the performance I got. Cheyenne has 36 cores/node.

Post Reply