Ocean Modeling Discussion

k.alexander

Thanks Hernan for the link to your warning, I hadn't seen it before. I'm not running with volume conservation enabled, but the same principles hold for global reductions.

k.alexander

I have fixed the problem by switching from openmpi 1.8.4 to openmpi 1.10.2. For ROMS 3.6, openmpi 1.8.4 was bit-reproducible. So something about the new kernel is now sensitive to the version of openmpi. Perhaps there should be a warning to users somewhere in the makefile comments to check their ver...

k.alexander

I seem to have resolved the problem by switching from openmpi 1.8.4 to openmpi 1.10.2. I haven't tested all possible tilings, of course, but it is now bit-reproducible for 32x16. I have confirmed this twice. Comparing 32x16 to 1x1, all the variables are the same except for swrad on one patch of the ...

k.alexander

I have confirmed that the same problem exists for the newest version of the code (as of today) with none of my own modifications (CICE coupling, ice shelves, etc.) See viewtopic.php?f=14&t=4470&p=17405#p17405.

k.alexander

I finally got around to setting up standalone ROMS with no modifications. Same grid, forcing, boundary conditions as before, but no sea ice, no ice shelves, none of the other miscellaneous things I've messed with. It is the most recent version of the code as of today. As before, ROMS is bit-reproduc...

k.alexander

Has anyone implemented a GM scheme in ROMS? I know from discussions with Hernan it has never been part of the master code. But I was wondering if it exists somewhere as a non-standard option. If so, I would love to give it a try!

k.alexander

Turns out you can just switch to the old version of distribute.F and nothing breaks! Unfortunately it doesn't fix the problem, it's still not bit-reproducible.

k.alexander

I will try that tomorrow and let you know what happens. Hopefully I can just revert that file without having everything else break.

k.alexander

Hmmm, very interesting. That ticket says "We get identical nesting solutions in serial with partitions, shared-memory (OpenMP), and distributed-memory (MPI) for tile partitions of 1x1, 2x2, and 3x3." So some tile partitions have been tested but not an exhaustive list. Looks like even if ne...

k.alexander

For the ROMS 3.6 distribution that was bit-reproducible for me, distribute.F is stamped with

Code: Select all

!svn $Id: distribute.F 645 2013-01-22 23:21:54Z arango $

and yes, it uses mpi_alltoall rather than mpi_allgather.

k.alexander

I asked the advice of one of my colleagues who is a software engineer working with different ocean models, and he had this to say: Those results are somewhat reassuring, if you can get any parallel run to match the 1x1 results. It is possible that your collectives are not reproducible, like MPI_Redu...

k.alexander

Just to confirm, I have svn 820. I took a look at the updates since then (the trunk is now at svn 839) and I don't see anything relevant, mainly just 4Dvar stuff that I have turned off anyway. I posted as a bug report now that we know it's not just me (i.e. Serega.Osipov has had the same problems).

k.alexander

Since updating from 3.6 to the trunk (svn version 820), my configuration of ROMS is no longer bit-reproducible for certain tilings. For example 1x1, 2x2, 16x8 all give identical output (for multiple simulations with the same tiling as well as comparing different tilings) but 32x1, 32x16 diverge. See...

k.alexander

Thanks for the replies - interesting to see that others are having the same issues. The explanation of how truncation errors propagate makes sense, however I don't think it applies here, because ROMS 3.6 was bit-reproducible for me (and the current version is bit-reproducible for certain parallel ti...

k.alexander

I've never tried it, I expect it would involve a lot of messing with compiler flags, plus gfortran runs more slowly than ifort. I don't think the compiler is the issue anyway.

k.alexander

Just FYI, I will have no internet access for the next week (off to see the Great Barrier Reef while it's still there!) so there won't be any updates from me here.

k.alexander

I'm using the mpif90 compiler from openmpi 1.8.4, which is a wrapper for ifort, using intel-fc version 12.1.9.293. However I used the same compiler for ROMS 3.6 which was bit-reproducible even for 32x16 tiling. It's possible that the intel-fc version has changed on the HPC cluster since then, but th...

k.alexander

Update: this bounds error also happens when running with a single processor. I'm wondering if it's a red herring.

k.alexander

So this is interesting, some parallel tilings are okay and others are not. The following have exactly the same output as 1x1: 2x1 1x2 2x2 16x1 1x16 8x2 24x1 30x1 31x1 8x4 12x4 15x4 16x4 16x8 The following don't: 16x16 16x2 32x1 32x16 (this is what I usually run with) I was confused as to why 16x4 an...

k.alexander

I understand parallel vs serial will usually give output differences though? But parallel vs parallel with the same tiling should be the same.

k.alexander

I just switched to a single processor for ROMS (NtileI=1, NtileJ=1 instead of 32 and 16 as I usually have) and now all variables are bit reproducible. So it is something about the parallel decomposition. Note that USE_MPI, DISTRIBUTE etc are still defined, so all the same calls are made to mp_exchan...

k.alexander

Something else interesting: if I turn off UV_ADV, temperature and some other variables are now unaffected (i.e. bit-reproducible at least in time index 2 of ocean_his). Here is the full list: Affected: ubar, u, v, w, AKt, AKs Unaffected: m, zeta, vbar, temp, salt, rho, Hsbl, AKv, shflux, ssflux, lwr...

k.alexander

Possibly related issue: Since I updated to 3.7 I have also occasionally had segfaults that don't reproduce themselves. That is, it will segfault, I resubmit the job without even recompiling, and then it doesn't segfault. I understand segfaults that show up inconsistently are often related to variabl...

k.alexander

I am running MetROMS (ROMS coupled with CICE) and since updating from ROMS 3.6 to the 3.7 trunk, the model has stopped being bit reproducible. i.e. if I run exactly the same simulation twice it gives me different answers. Not a good sign... I'm almost positive that I've ruled out CICE or the couplin...

k.alexander

Thanks to Ryan Holmes who stopped by my desk to explain all this to me

Consider this ticket closed!

k.alexander

I'm playing around with vertical mixing in the hopes of preventing deep convection in the Weddell Sea. It's a delicate balance between having enough vertical mixing that brine doesn't accumulate in the mixed layer, but not so much that the Winter Water layer gets mixed out. There are a lot of option...

k.alexander

I've just updated from 3.6 and 3.7, and found a bug in the quicksave code. I had quicksave files turned off in roms.in, but a few of my 512 processors were getting caught in the quicksave code anyway. Interestingly enough, this was preventing the next output files (averages or restart, depending on ...

k.alexander

Thanks Kate, this worked for me too. Interestingly, the error showed up differently for me once I'd switched to 3.7 - I got this instead: viewtopic.php?f=14&t=3749 But the root cause was the same, and I fixed it by increasing the character limit on "string".

k.alexander

Fixed this by updating to 3.7

k.alexander

Adding check_multifile just isn't working. I've been at it all day and there are stubborn MPI errors. I think there is some deeply buried structural thing this routine depends on which isn't in ROMS 3.6. I am just about to go to a conference in Melbourne but when I'm back I will update to the latest...

k.alexander

Right now I'm tossing up whether to spend the time and effort fully updating to the newest ROMS trunk, which will be a fair bit of work. Or just fix this multifile issue and stick with 3.6 for simplicity. Given that my system is (hopefully) almost operational, I'm not sure what would be the best inv...

k.alexander

Ok I have found the routine check_multifile in ROMS 3.7. Looks like it wasn't in 3.6 at all. Working on getting it to fit now...

k.alexander

Thanks Kate, but unfortunately that patch doesn't make any difference. I will keep digging. Multifiles have worked fine for me in 3.6 otherwise, as long as you start with the first file it then switches between them just fine. I definitely share your sentiment about version control, and I have used ...

k.alexander

After some print statements it looks like FRC(ifile)%name is initialised to the first name in the multi-file list, and FRC(ifile)%Fcount is initialised to 1. Can't find the bit of code where they are updated...

k.alexander

I am having exactly the same problem as this message: https://www.myroms.org/forum/viewtopic.php?f=14&t=3835 but since I am not using COAWST the Utility code is all different, and I can't find the snippet of code that was the culprit in that message. (I am using ROMS 3.6 with a bunch of sea ice ...

k.alexander

Thanks Hui for your message. Yes, as we discussed over email, I was getting more advection errors with A4 + strong diffusion than I realised. Switching to HSIMT removed the weird "corner" of sea ice in the Ross Sea, and there is thinner sea ice in the outer Weddell Sea now too. I am curren...

k.alexander

I just got exactly the same error. Did you ever solve it?

k.alexander

PS Thank you for spotting this. It's interesting (and a bit dangerous) how geometric errors like this can still compile and run just fine.

k.alexander

Ok that makes sense. I will regenerate that grid. Do you happen to know how CICE handles the periodic boundary conditions? I assume from your previous emails that it doesn't want any overlap cells, i.e. it should be 1440 cells in the x-direction for both the CICE t-grid and u-grid. Dealing with this...

k.alexander

Apologies if the answer to this is obvious somewhere. A quick search on this forum, the wiki, and a browse through the source code didn't yield any obvious answers... I have a periodic boundary (east-west) and as Kate points out, the sea ice coupling I am using (MetROMS) may not support this properl...

k.alexander

Yes I missed exactly what Hernan said about HSIMT, I think it was something about topography? However the COAWST version is apparently a bit different (or so Hui Wu tells me) so maybe that is fixed. I'm just testing it for now, if I end up keeping it I will do more investigation. I am also going to ...

k.alexander

Just got access to HSIMT, and confirmation that it is okay with negative temperatures. I will try that today or tomorrow. However, I don't think my issues are due to advection anymore. That would show up in the frazil fields which are very low in the Ross and Weddell. Instead the ice there is gettin...

k.alexander

Ah ok interesting about MPDATA. I'm not sure how HSIMT handles negative temperatures, Kate do you know? The MetROMS developers in Norway suggest TS_A4HADVECTION + TS_C4VADVECTION with quite a lot of diffusion (TS_DIF2) to smooth out the errors. I'm going to try that next. Feels a bit dirty to use di...

k.alexander

Ah that makes sense. Yes, ROMS and CICE read the forcing fields separately. A bit inefficient but since they aren't coupling every timestep I think it's the only way to do it properly (at least without rewriting a lot of code!) FYI CICE wants specific humidity, whereas ROMS accepts either specific o...

k.alexander

Hi Kate, I'm not getting that problem - in spring (October) shortwave is nonzero, longwave is non-constant, and both look reasonable. I assume it is CICE radiation fields that you mean? Maybe you have the wrong CICE radiation option turned on?

k.alexander

Just read a bit about the HSIMT scheme and it sounds great (I particularly like the bit about being more efficient than MPDATA!) Who knows if it will blow up like MPDATA does, but it's worth a go. What's the best way to get access to the COAWST code so I can merge the option into MetROMS?

k.alexander

Hi Kate, that's interesting about the congelation. I don't have an alternate coupling setup to compare my runs to, but for what it's worth my annually averaged congelation values are generally around 0.2-0.3, with the exception of right at the coast where they are around 1 (up to 1.4). I *think* tha...

k.alexander

I have run for 3 years now and the sea ice is getting really weird. Here is the concentration (top) and ice thickness (bottom) averaged over the last year: https://dl.dropboxusercontent.com/u/7618756/aice_hi.png Notice the giant Weddell Sea polynya opening up, as well as the lack of ice beyond the c...

k.alexander

Okay I have run a whole year now, and the good news is that the crazy thick ice seems to be fixed. It's still a bit thicker than observations in some regions (especially the Ross Sea) but it at least passes the sanity check! I have more coastal polynya activity, much deeper mixed layers especially r...

k.alexander

I ran 4 months with TS_U3HADVECTION (instead of TS_C4HADVECTION) and the thick ice is gone! No more signs of spurious supercooling, and no more crazy thick ice over the continental shelf break. It remains to be seen whether this will fix my lack of coastal polynyas (and resulting erosion of Winter W...

k.alexander

I've fixed a bunch of things since I last tried TS_U3HADVECTION. I'll try it (and a few other advection schemes) next week when I have CPU time again.

Kate, are your frzmlt values very different between the ROMS-CICE "fake coupling" and MetROMS? Or is it just the ice thickness?

k.alexander

The one thing I didn't try from your frazil code was ICE_LOG_LAYER. I don't suppose that would make a difference? Regardless of whether advection is *the* problem, it clearly seems to be *a* problem for my setup at least...I tried TS_U3HADVECTION ages ago and it absolutely destroyed my sea ice every...

k.alexander

Hi Kate, that's interesting that you have the same problem with MetROMS but not your internally coupled ROMS/CICE. It's not the frazil (supercooling) code because one of the things I tried was using your frazil code instead of the (very slightly different) MetROMS code. Were you using the same advec...

k.alexander

My name is Kaitlin Alexander and I am a PhD student in Australia, working with Katrin Meissner, Ben Galton-Fenzi, and Matthew England. As part of my PhD I am using the new MetROMS (ROMS + CICE + MCT) system coupled by the Norwegian Meteorological Institute. I have adapted this to a circumpolar Antar...

Search found 54 matches