WRF quilting support in the WRF-ROMS setup in COAWST

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
Serega.Osipov
Posts: 5
Joined: Thu Oct 01, 2015 4:41 am
Location: KAUST

WRF quilting support in the WRF-ROMS setup in COAWST

#1 Unread post by Serega.Osipov »

I'm having problems running WRF with quilting specifically in the COAWST setup.
Normally, when running WRF with a large number of cores (such as 1024 and more) saving NetCDF output becomes slow (about 30 sec). This is easily resolved by using quilting (by setting namelist parameters nio_tasks_per_group=8(16) and nio_groups=2(4,6)). This reduces the saving time to about 0.03 seconds which is significant speed up.
I've tested pure WRF with this quilting settings and it is running fine. However, when WRF is coupled to ROMS, turning on quilting causes the run to hang indefinitely (no errors, doesn't crash). I suspect that MCT expects cores to communicate and since some of them in case of quilting are dedicated for the I/O, this causes the hanging. I've also tried to specify nproc_x and nproc_y, but it didn't help.

Is it possible to run WRF-ROMS setup with WRF quilting option enabled?

jcwarner
Posts: 1172
Joined: Wed Dec 31, 2003 6:16 pm
Location: USGS, USA

Re: WRF quilting support in the WRF-ROMS setup in COAWST

#2 Unread post by jcwarner »

when the wrf + other-model job is submitted, the total number of cpus goes through an mpi_comm_split to allocate the procs for wrf. Then each wrf proc does an MCT init so all the processors for wrf will look to communicate through mct. I dont think the current configuration will allow quilting, but i can run a test to see. i am not sure exactly how wrf determines which procs go to quilting and which to computations. i can take a look but it will be some time til i get to it.

Jingwen_Huang
Posts: 1
Joined: Wed Feb 19, 2020 9:29 pm
Location: Ocean University of China

Re: WRF quilting support in the WRF-ROMS setup in COAWST

#3 Unread post by Jingwen_Huang »

:D I found that COAWST 3.7 supports this setup, but COAWST 3.6 has the same problem as you
Last edited by Jingwen_Huang on Thu Dec 07, 2023 2:12 am, edited 1 time in total.

jcwarner
Posts: 1172
Joined: Wed Dec 31, 2003 6:16 pm
Location: USGS, USA

Re: WRF quilting support in the WRF-ROMS setup in COAWST

#4 Unread post by jcwarner »

this is best to post a message on the coawst forum:
https://github.com/DOI-USGS/COAWST/issues
and the answer is yes, it should work.

Post Reply