Search found 54 matches

by k.alexander
Tue Apr 18, 2017 12:37 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Thanks Hernan for the link to your warning, I hadn't seen it before. I'm not running with volume conservation enabled, but the same principles hold for global reductions.
by k.alexander
Tue Apr 11, 2017 4:55 am
Forum: ROMS Bugs
Topic: ROMS not bit-reproducible for certain tilings
Replies: 2
Views: 1843

Re: ROMS not bit-reproducible for certain tilings

I have fixed the problem by switching from openmpi 1.8.4 to openmpi 1.10.2. For ROMS 3.6, openmpi 1.8.4 was bit-reproducible. So something about the new kernel is now sensitive to the version of openmpi. Perhaps there should be a warning to users somewhere in the makefile comments to check their ver...
by k.alexander
Tue Apr 11, 2017 4:53 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I seem to have resolved the problem by switching from openmpi 1.8.4 to openmpi 1.10.2. I haven't tested all possible tilings, of course, but it is now bit-reproducible for 32x16. I have confirmed this twice. Comparing 32x16 to 1x1, all the variables are the same except for swrad on one patch of the ...
by k.alexander
Mon Apr 10, 2017 2:33 am
Forum: ROMS Bugs
Topic: ROMS not bit-reproducible for certain tilings
Replies: 2
Views: 1843

Re: ROMS not bit-reproducible for certain tilings

I have confirmed that the same problem exists for the newest version of the code (as of today) with none of my own modifications (CICE coupling, ice shelves, etc.) See viewtopic.php?f=14&t=4470&p=17405#p17405.
by k.alexander
Mon Apr 10, 2017 2:31 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I finally got around to setting up standalone ROMS with no modifications. Same grid, forcing, boundary conditions as before, but no sea ice, no ice shelves, none of the other miscellaneous things I've messed with. It is the most recent version of the code as of today. As before, ROMS is bit-reproduc...
by k.alexander
Wed Apr 05, 2017 2:10 am
Forum: ROMS Discussion
Topic: Gent-McWilliams mixing in ROMS
Replies: 0
Views: 788

Gent-McWilliams mixing in ROMS

Has anyone implemented a GM scheme in ROMS? I know from discussions with Hernan it has never been part of the master code. But I was wondering if it exists somewhere as a non-standard option. If so, I would love to give it a try!
by k.alexander
Wed Mar 29, 2017 3:29 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Turns out you can just switch to the old version of distribute.F and nothing breaks! Unfortunately it doesn't fix the problem, it's still not bit-reproducible.
by k.alexander
Tue Mar 28, 2017 5:51 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I will try that tomorrow and let you know what happens. Hopefully I can just revert that file without having everything else break.
by k.alexander
Tue Mar 28, 2017 5:05 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Hmmm, very interesting. That ticket says "We get identical nesting solutions in serial with partitions, shared-memory (OpenMP), and distributed-memory (MPI) for tile partitions of 1x1, 2x2, and 3x3." So some tile partitions have been tested but not an exhaustive list. Looks like even if nesting is o...
by k.alexander
Tue Mar 28, 2017 1:00 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

For the ROMS 3.6 distribution that was bit-reproducible for me, distribute.F is stamped with

Code: Select all

!svn $Id: distribute.F 645 2013-01-22 23:21:54Z arango $
and yes, it uses mpi_alltoall rather than mpi_allgather.
by k.alexander
Mon Mar 27, 2017 11:27 pm
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I asked the advice of one of my colleagues who is a software engineer working with different ocean models, and he had this to say: Those results are somewhat reassuring, if you can get any parallel run to match the 1x1 results. It is possible that your collectives are not reproducible, like MPI_Redu...
by k.alexander
Mon Mar 27, 2017 5:16 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Just to confirm, I have svn 820. I took a look at the updates since then (the trunk is now at svn 839) and I don't see anything relevant, mainly just 4Dvar stuff that I have turned off anyway. I posted as a bug report now that we know it's not just me (i.e. Serega.Osipov has had the same problems).
by k.alexander
Mon Mar 27, 2017 5:13 am
Forum: ROMS Bugs
Topic: ROMS not bit-reproducible for certain tilings
Replies: 2
Views: 1843

ROMS not bit-reproducible for certain tilings

Since updating from 3.6 to the trunk (svn version 820), my configuration of ROMS is no longer bit-reproducible for certain tilings. For example 1x1, 2x2, 16x8 all give identical output (for multiple simulations with the same tiling as well as comparing different tilings) but 32x1, 32x16 diverge. See...
by k.alexander
Mon Mar 27, 2017 4:38 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Thanks for the replies - interesting to see that others are having the same issues. The explanation of how truncation errors propagate makes sense, however I don't think it applies here, because ROMS 3.6 was bit-reproducible for me (and the current version is bit-reproducible for certain parallel ti...
by k.alexander
Fri Mar 17, 2017 6:10 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I've never tried it, I expect it would involve a lot of messing with compiler flags, plus gfortran runs more slowly than ifort. I don't think the compiler is the issue anyway.
by k.alexander
Fri Mar 17, 2017 6:01 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Just FYI, I will have no internet access for the next week (off to see the Great Barrier Reef while it's still there!) so there won't be any updates from me here.
by k.alexander
Fri Mar 17, 2017 5:10 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I'm using the mpif90 compiler from openmpi 1.8.4, which is a wrapper for ifort, using intel-fc version 12.1.9.293. However I used the same compiler for ROMS 3.6 which was bit-reproducible even for 32x16 tiling. It's possible that the intel-fc version has changed on the HPC cluster since then, but th...
by k.alexander
Fri Mar 17, 2017 3:04 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Update: this bounds error also happens when running with a single processor. I'm wondering if it's a red herring.
by k.alexander
Fri Mar 17, 2017 2:19 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

So this is interesting, some parallel tilings are okay and others are not. The following have exactly the same output as 1x1: 2x1 1x2 2x2 16x1 1x16 8x2 24x1 30x1 31x1 8x4 12x4 15x4 16x4 16x8 The following don't: 16x16 16x2 32x1 32x16 (this is what I usually run with) I was confused as to why 16x4 an...
by k.alexander
Thu Mar 16, 2017 7:02 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I understand parallel vs serial will usually give output differences though? But parallel vs parallel with the same tiling should be the same.
by k.alexander
Thu Mar 16, 2017 4:46 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

I just switched to a single processor for ROMS (NtileI=1, NtileJ=1 instead of 32 and 16 as I usually have) and now all variables are bit reproducible. So it is something about the parallel decomposition. Note that USE_MPI, DISTRIBUTE etc are still defined, so all the same calls are made to mp_exchan...
by k.alexander
Thu Mar 16, 2017 3:58 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Something else interesting: if I turn off UV_ADV, temperature and some other variables are now unaffected (i.e. bit-reproducible at least in time index 2 of ocean_his). Here is the full list: Affected: ubar, u, v, w, AKt, AKs Unaffected: m, zeta, vbar, temp, salt, rho, Hsbl, AKv, shflux, ssflux, lwr...
by k.alexander
Thu Mar 16, 2017 3:18 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

Re: ROMS not bit-reproducible: northern boundary?

Possibly related issue: Since I updated to 3.7 I have also occasionally had segfaults that don't reproduce themselves. That is, it will segfault, I resubmit the job without even recompiling, and then it doesn't segfault. I understand segfaults that show up inconsistently are often related to variabl...
by k.alexander
Thu Mar 16, 2017 1:46 am
Forum: ROMS Discussion
Topic: ROMS not bit-reproducible: northern boundary?
Replies: 32
Views: 8604

ROMS not bit-reproducible: northern boundary?

I am running MetROMS (ROMS coupled with CICE) and since updating from ROMS 3.6 to the 3.7 trunk, the model has stopped being bit reproducible. i.e. if I run exactly the same simulation twice it gives me different answers. Not a good sign... I'm almost positive that I've ruled out CICE or the couplin...
by k.alexander
Mon Feb 27, 2017 5:06 am
Forum: ROMS Discussion
Topic: LMD options
Replies: 1
Views: 847

Re: LMD options

Thanks to Ryan Holmes who stopped by my desk to explain all this to me :-) Consider this ticket closed!
by k.alexander
Mon Feb 27, 2017 1:38 am
Forum: ROMS Discussion
Topic: LMD options
Replies: 1
Views: 847

LMD options

I'm playing around with vertical mixing in the hopes of preventing deep convection in the Weddell Sea. It's a delicate balance between having enough vertical mixing that brine doesn't accumulate in the mixed layer, but not so much that the Winter Water layer gets mixed out. There are a lot of option...
by k.alexander
Fri Jan 13, 2017 12:18 am
Forum: ROMS Bugs
Topic: Quicksave bug
Replies: 1
Views: 1503

Quicksave bug

I've just updated from 3.6 and 3.7, and found a bug in the quicksave code. I had quicksave files turned off in roms.in, but a few of my 512 processors were getting caught in the quicksave code anyway. Interestingly enough, this was preventing the next output files (averages or restart, depending on ...
by k.alexander
Thu Jan 12, 2017 11:51 pm
Forum: ROMS Bugs
Topic: Error with Multifiles
Replies: 9
Views: 7229

Re: Error with Multifiles

Thanks Kate, this worked for me too. Interestingly, the error showed up differently for me once I'd switched to 3.7 - I got this instead: viewtopic.php?f=14&t=3749 But the root cause was the same, and I fixed it by increasing the character limit on "string".
by k.alexander
Thu Jan 12, 2017 11:48 pm
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

Fixed this by updating to 3.7 :-)
by k.alexander
Tue Nov 15, 2016 3:52 am
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

Adding check_multifile just isn't working. I've been at it all day and there are stubborn MPI errors. I think there is some deeply buried structural thing this routine depends on which isn't in ROMS 3.6. I am just about to go to a conference in Melbourne but when I'm back I will update to the latest...
by k.alexander
Mon Nov 14, 2016 11:58 pm
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

Right now I'm tossing up whether to spend the time and effort fully updating to the newest ROMS trunk, which will be a fair bit of work. Or just fix this multifile issue and stick with 3.6 for simplicity. Given that my system is (hopefully) almost operational, I'm not sure what would be the best inv...
by k.alexander
Mon Nov 14, 2016 11:19 pm
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

Ok I have found the routine check_multifile in ROMS 3.7. Looks like it wasn't in 3.6 at all. Working on getting it to fit now...
by k.alexander
Mon Nov 14, 2016 10:37 pm
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

Thanks Kate, but unfortunately that patch doesn't make any difference. I will keep digging. Multifiles have worked fine for me in 3.6 otherwise, as long as you start with the first file it then switches between them just fine. I definitely share your sentiment about version control, and I have used ...
by k.alexander
Mon Nov 14, 2016 4:50 am
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Re: Restarting with multi-files

After some print statements it looks like FRC(ifile)%name is initialised to the first name in the multi-file list, and FRC(ifile)%Fcount is initialised to 1. Can't find the bit of code where they are updated...
by k.alexander
Mon Nov 14, 2016 1:49 am
Forum: ROMS Discussion
Topic: Restarting with multi-files
Replies: 9
Views: 2284

Restarting with multi-files

I am having exactly the same problem as this message: https://www.myroms.org/forum/viewtopic.php?f=14&t=3835 but since I am not using COAWST the Utility code is all different, and I can't find the snippet of code that was the culprit in that message. (I am using ROMS 3.6 with a bunch of sea ice and ...
by k.alexander
Mon Nov 07, 2016 12:21 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Thanks Hui for your message. Yes, as we discussed over email, I was getting more advection errors with A4 + strong diffusion than I realised. Switching to HSIMT removed the weird "corner" of sea ice in the Ross Sea, and there is thinner sea ice in the outer Weddell Sea now too. I am currently runnin...
by k.alexander
Thu Nov 03, 2016 4:29 am
Forum: ROMS Bugs
Topic: Error with Multifiles
Replies: 9
Views: 7229

Re: Error with Multifiles

I just got exactly the same error. Did you ever solve it?
by k.alexander
Tue Oct 25, 2016 8:41 am
Forum: ROMS Discussion
Topic: Periodic boundary overlap in input grid
Replies: 4
Views: 1874

Re: Periodic boundary overlap in input grid

PS Thank you for spotting this. It's interesting (and a bit dangerous) how geometric errors like this can still compile and run just fine.
by k.alexander
Tue Oct 25, 2016 8:37 am
Forum: ROMS Discussion
Topic: Periodic boundary overlap in input grid
Replies: 4
Views: 1874

Re: Periodic boundary overlap in input grid

Ok that makes sense. I will regenerate that grid. Do you happen to know how CICE handles the periodic boundary conditions? I assume from your previous emails that it doesn't want any overlap cells, i.e. it should be 1440 cells in the x-direction for both the CICE t-grid and u-grid. Dealing with this...
by k.alexander
Tue Oct 25, 2016 4:14 am
Forum: ROMS Discussion
Topic: Periodic boundary overlap in input grid
Replies: 4
Views: 1874

Periodic boundary overlap in input grid

Apologies if the answer to this is obvious somewhere. A quick search on this forum, the wiki, and a browse through the source code didn't yield any obvious answers... I have a periodic boundary (east-west) and as Kate points out, the sea ice coupling I am using (MetROMS) may not support this properl...
by k.alexander
Mon Oct 24, 2016 5:33 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Yes I missed exactly what Hernan said about HSIMT, I think it was something about topography? However the COAWST version is apparently a bit different (or so Hui Wu tells me) so maybe that is fixed. I'm just testing it for now, if I end up keeping it I will do more investigation. I am also going to ...
by k.alexander
Mon Oct 24, 2016 1:00 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Just got access to HSIMT, and confirmation that it is okay with negative temperatures. I will try that today or tomorrow. However, I don't think my issues are due to advection anymore. That would show up in the frazil fields which are very low in the Ross and Weddell. Instead the ice there is gettin...
by k.alexander
Tue Oct 11, 2016 11:00 pm
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Ah ok interesting about MPDATA. I'm not sure how HSIMT handles negative temperatures, Kate do you know? The MetROMS developers in Norway suggest TS_A4HADVECTION + TS_C4VADVECTION with quite a lot of diffusion (TS_DIF2) to smooth out the errors. I'm going to try that next. Feels a bit dirty to use di...
by k.alexander
Mon Oct 10, 2016 11:46 pm
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Ah that makes sense. Yes, ROMS and CICE read the forcing fields separately. A bit inefficient but since they aren't coupling every timestep I think it's the only way to do it properly (at least without rewriting a lot of code!) FYI CICE wants specific humidity, whereas ROMS accepts either specific o...
by k.alexander
Mon Oct 10, 2016 12:27 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Hi Kate, I'm not getting that problem - in spring (October) shortwave is nonzero, longwave is non-constant, and both look reasonable. I assume it is CICE radiation fields that you mean? Maybe you have the wrong CICE radiation option turned on?
by k.alexander
Fri Oct 07, 2016 5:02 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Just read a bit about the HSIMT scheme and it sounds great (I particularly like the bit about being more efficient than MPDATA!) Who knows if it will blow up like MPDATA does, but it's worth a go. What's the best way to get access to the COAWST code so I can merge the option into MetROMS?
by k.alexander
Fri Oct 07, 2016 12:52 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Hi Kate, that's interesting about the congelation. I don't have an alternate coupling setup to compare my runs to, but for what it's worth my annually averaged congelation values are generally around 0.2-0.3, with the exception of right at the coast where they are around 1 (up to 1.4). I *think* tha...
by k.alexander
Thu Oct 06, 2016 2:42 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

I have run for 3 years now and the sea ice is getting really weird. Here is the concentration (top) and ice thickness (bottom) averaged over the last year: https://dl.dropboxusercontent.com/u/7618756/aice_hi.png Notice the giant Weddell Sea polynya opening up, as well as the lack of ice beyond the c...
by k.alexander
Tue Oct 04, 2016 1:19 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Okay I have run a whole year now, and the good news is that the crazy thick ice seems to be fixed. It's still a bit thicker than observations in some regions (especially the Ross Sea) but it at least passes the sanity check! I have more coastal polynya activity, much deeper mixed layers especially r...
by k.alexander
Mon Oct 03, 2016 10:59 pm
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

I ran 4 months with TS_U3HADVECTION (instead of TS_C4HADVECTION) and the thick ice is gone! No more signs of spurious supercooling, and no more crazy thick ice over the continental shelf break. It remains to be seen whether this will fix my lack of coastal polynyas (and resulting erosion of Winter W...
by k.alexander
Fri Sep 30, 2016 12:52 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

I've fixed a bunch of things since I last tried TS_U3HADVECTION. I'll try it (and a few other advection schemes) next week when I have CPU time again.

Kate, are your frzmlt values very different between the ROMS-CICE "fake coupling" and MetROMS? Or is it just the ice thickness?
by k.alexander
Thu Sep 29, 2016 5:14 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

The one thing I didn't try from your frazil code was ICE_LOG_LAYER. I don't suppose that would make a difference? Regardless of whether advection is *the* problem, it clearly seems to be *a* problem for my setup at least...I tried TS_U3HADVECTION ages ago and it absolutely destroyed my sea ice every...
by k.alexander
Thu Sep 29, 2016 4:17 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Re: Spurious supercooling in ROMS-CICE simulation

Hi Kate, that's interesting that you have the same problem with MetROMS but not your internally coupled ROMS/CICE. It's not the frazil (supercooling) code because one of the things I tried was using your frazil code instead of the (very slightly different) MetROMS code. Were you using the same advec...
by k.alexander
Thu Sep 29, 2016 2:26 am
Forum: ROMS Discussion
Topic: Spurious supercooling in ROMS-CICE simulation
Replies: 33
Views: 8097

Spurious supercooling in ROMS-CICE simulation

My name is Kaitlin Alexander and I am a PhD student in Australia, working with Katrin Meissner, Ben Galton-Fenzi, and Matthew England. As part of my PhD I am using the new MetROMS (ROMS + CICE + MCT) system coupled by the Norwegian Meteorological Institute. I have adapted this to a circumpolar Antar...