Performance issues using floats

Report or discuss software problems and other woes

Moderators: arango, robertson

Post Reply
Message
Author
lcbernardo
Posts: 59
Joined: Wed Oct 01, 2014 8:57 pm
Location: Tokyo Institute of Technology

Performance issues using floats

#1 Post by lcbernardo » Fri May 17, 2019 2:47 am

Dear ROMS users,

I have encountered a strange situation when I use floats in ROMS (or in my case, COAWST using only ROMS). My application's grid size is 300 x 300, with 15 vertical layers. And I am trying to run a 60-day simulation with floats, and the total number of floats released is 149975. I started off using 28 cpus (4 x 7 tiling) and I got some initial results. I then hoped to decrease the run time by using more nodes in our computing cluster, and tried to run using 56 cpus (7 x 8 tiling) and then 112 cpus (8 x 14 tiling). However, what occurred was the complete opposite, and the number of days simulated successfully decreased the more cpus I used. Has anyone encountered such a situation before?

It may have to do with our system's settings, so I'm still checking. But nevertheless, I'd appreciate any help on this issue.

Thanks,
Lawrence

jcwarner
Posts: 798
Joined: Wed Dec 31, 2003 6:16 pm
Location: USGS, USA

Re: Performance issues using floats

#2 Post by jcwarner » Fri May 17, 2019 3:09 am

as you increase the number of procs for the job, then the tile size decreases. 300x300 tiled 8x14 gives a tile 38 x 20 cells. that is not very big. the job is spending more time exchanging information than performing computations.
here are some previous posts in this issue
viewtopic.php?f=29&t=134

lcbernardo
Posts: 59
Joined: Wed Oct 01, 2014 8:57 pm
Location: Tokyo Institute of Technology

Re: Performance issues using floats

#3 Post by lcbernardo » Fri May 17, 2019 3:29 am

Thank you for the reply Dr. Warner. I'll look into these previous posts to see what may be helpful. But just basing on the explanation you provided, if the problem is the added time exchanging information among tiles, will this be particularly true for the floats option? Because if I don't use floats, and also for the case wherein I couple ROMS and SWAN, I get increased performance when I increase the number of cpus from 28 to 56 or 112.

User avatar
kate
Posts: 3678
Joined: Wed Jul 02, 2003 5:29 pm
Location: IMS/UAF, USA

Re: Performance issues using floats

#4 Post by kate » Fri May 17, 2019 4:28 am

The floats implementation works for a modest number of floats, but doesn't scale well. Yes, the floats incur a lot of communications and processing as they figure out which tile each float is on every timestep.

lcbernardo
Posts: 59
Joined: Wed Oct 01, 2014 8:57 pm
Location: Tokyo Institute of Technology

Re: Performance issues using floats

#5 Post by lcbernardo » Fri May 17, 2019 5:13 am

Thanks for the reply Kate! So scaling isn't so good for floats. I guess I'll have to be more strategic in my use of this option then.

Post Reply