faster MPI on single computer / -bind-to core /Ubuntu 18.04

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
jpringle
Posts: 107
Joined: Sun Jul 27, 2003 6:49 pm
Location: UNH, USA

faster MPI on single computer / -bind-to core /Ubuntu 18.04

#1 Unread post by jpringle »

Dear all --

I hope some of you find this useful. I am running ROMS on a dual processor 32 core AMD EPYC machine. I noticed on the latest version of Ubuntu my smaller machines showed good improvement with ROMS runs with openMP parallelization.

So I upgraded the big machine to Ubuntu 18.04, and my ROMS runs became about 15% slower. This new version has a new kernel (4.15) which is supposed to allocate processes to cores in a way that works better with the EPYC memory architecture (it is NUMA with 16 memory channels). It is not doing so by default.

I returned to the old performance (actually, a few percent faster) by invoking mpirun with the "-bind-to core" option -- I run the model with "mpirun -bind-to core -np 32 ./oceanM jnk.in"

Hope this helps someone,
Jamie

jpringle
Posts: 107
Joined: Sun Jul 27, 2003 6:49 pm
Location: UNH, USA

Re: faster MPI on single computer / -bind-to core /Ubuntu 18.04

#2 Unread post by jpringle »

With the same version of Ubuntu, but updated kernels (nothing special, just the default updates), this no longer appears to be true. A reminder to test your tiling and MPI options every so often.

Jamie

Post Reply