Search found 12 matches

by koushik
Fri Apr 03, 2020 6:45 pm
Forum: ROMS Discussion
Topic: Imbalance in Parallel writing in ROMS
Replies: 3
Views: 4098

Re: Imbalance in Parallel writing in ROMS

Hi Koushik, great study, and thank you for the research that you continue doing about ROMS parallel I/O :!: Let me digest the information and come up with a strategy for more efficient parallel I/O. Are you aware that ROMS has the CPP option NO_WRITE_GRID , which suppresses the definition and writi...
by koushik
Fri Apr 03, 2020 6:44 pm
Forum: ROMS Discussion
Topic: Imbalance in Parallel writing in ROMS -- Further Investigation Results
Replies: 0
Views: 997

Imbalance in Parallel writing in ROMS -- Further Investigation Results

Hello All, I have investigated the imbalance further within nf_fwrite2d.F and here are the results --- Imbalance in parallel NetCDF define phase.png Figure 1: Further Parallel writing Imbalance analysis Observations and Conclusion: 1.The imbalance is flowing as below... -- The imbalance is found in ...
by koushik
Tue Mar 24, 2020 6:24 am
Forum: ROMS Bugs
Topic: ROMS Code failing in between when it is creating new NetCDF file to write the output
Replies: 1
Views: 1290

ROMS Code failing in between when it is creating new NetCDF file to write the output

Hello All, I am facing the below issue -- My ROMS code is failing sometimes in the middle while defining the NetCDF files phase. It is completing successfully sometimes and sometimes its failing while defining new quick write files after some time steps (different timesteps in different runs). -----...
by koushik
Tue Mar 24, 2020 6:07 am
Forum: ROMS Discussion
Topic: Imbalance in Parallel writing in ROMS
Replies: 3
Views: 4098

Re: Imbalance in Parallel writing in ROMS

Hello Arango, Thanks for your suggestions. 1.I was not aware of NO_WRITE_GRID flag. I think the imbalance is in writing the GRID arrays. I will perform the analysis with this flag ON and let you know the results. 2.I am doing the analysis for the serial NetCDF writing to compare with the parallel Ne...
by koushik
Sun Mar 22, 2020 8:35 am
Forum: ROMS Discussion
Topic: Imbalance in Parallel writing in ROMS
Replies: 3
Views: 4098

Imbalance in Parallel writing in ROMS

Hello All, Questions are at the last part of this post:(I have explained my oberservations first) I have done IO analysis of writing data into NetCDF files. I have found load imbalance while writing the data. The configuration I have used is as below... 40*36 PEs = 1440 PEs used for simulation in CR...
by koushik
Fri Dec 27, 2019 4:25 am
Forum: ROMS Discussion
Topic: Bottleneck in ROMS application
Replies: 0
Views: 1178

Bottleneck in ROMS application

I am using Cray PAT prfiling tool to find the bottlenecks in ROMS application . I have used NetCDF4-HDF5 parallel with 1440 PEs for this experiment and writing 5 Quick files. ! Time-Stepping parameters ---- NTIMES == 1080 DT == 240.0d0 NDTFAST == 54 I have added these 2 flags in the header file to a...
by koushik
Sat Dec 07, 2019 5:55 am
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Re: Parallel IO not scaling over Serial IO for ROMS

robertson wrote: Fri Dec 06, 2019 9:27 pm Please attach your gbplume.h file where your chosen CPP Definitions are set.
Please find attachec the gbplum.h file
Please let me know if something need to be added as I am new to ROMS.
I think my PARALLEL_IO is not turned on properly although I have turned on USE_PARALLEL_IO in build script.
by koushik
Fri Dec 06, 2019 8:20 pm
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Re: Parallel IO not scaling over Serial IO for ROMS

Where did you get your ROMS code from? That's what I was asking. Hernan's code has PARALLEL_IO, but you say turning that on isn't doing anything. My code does not recognize PARALLEL_IO but instead recognizes PARALLEL_IN and PARALLEL_OUT. A quick check of cppdefs.h should tell you what the options a...
by koushik
Fri Dec 06, 2019 6:56 pm
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Re: Parallel IO not scaling over Serial IO for ROMS

Many years ago our supercomputer center bought a Cray and an IBM. The timings I did without I/O led me to believe that the Cray was the machine for me. However, my real jobs have I/O and that meant that the IBM was actually faster for my problems. At that time, I didn't have a vectorized netCDF lib...
by koushik
Fri Dec 06, 2019 6:40 pm
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Re: Parallel IO not scaling over Serial IO for ROMS

I'm using CRAY XC40 and must admit, quite happy with performance (dragonfly net topology), 24 core nodes. Made some benchmarking and it really depends on your application and grid size. Note that depending on users load on scratch you can have totally different results (and you can tweak parallel w...
by koushik
Fri Dec 06, 2019 5:54 pm
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Re: Parallel IO not scaling over Serial IO for ROMS

Great study. We are missing important information about your simulations. We need the grid size and the parallel domain decomposition. I have mentioned frequently in this forum that ROMS scalability is a function of the application and depends on the domain size and parallel partition. There is alw...
by koushik
Thu Dec 05, 2019 11:25 am
Forum: ROMS Discussion
Topic: Parallel IO not scaling over Serial IO for ROMS
Replies: 14
Views: 12986

Parallel IO not scaling over Serial IO for ROMS

Hello All, I am working on IO scalability for ROMS model in Cray XC40 machine. I have some observations and some questions as below - 1. When I am not writing any netcdf output file , I observed the model is scaling upto 960 processors(PEs) only.Any reason for that? (Strong Scaling Results -- Varing...