Dual NPZD Variables

[pmaccc]

Dear fellow NPZD modelers,

For ongoing projects related to the effects of treated sewage and marine Carbon Dioxide removal we have been running "twin" experiments, computing the difference between runs with standard nutrient or alkalinity boundary fluxes and runs in which those fluxes have been altered. Being able to do this reliably is incredibly valuable for these projects, but we have run into problems with "bit-reproducibility." Typically the problem manifests as small but growing noise-like differences in biogeochemical fields that are distant from the altered sources (and so clearly not the result of advection. Sometimes this noise seems to come from open boundaries but can also arise internally. By being careful about running with the same MPI tiling we are able to achieve bit-reproducibility of the physical fields (velocity, temp, salt) so clearly it is possible in ROMS, but somehow not with the biogeochemical fields. Other info: we are using a modified version of the fennel.h bio code, and we use MPDATA for all tracer advection (including temp and salt). We have seem some improvements in NPZD bit reproducibility by using HSIMT advection, but this degrades the quality of the simulations. Similarly using much stronger nudging to climatology seems to help along the open boundaries, but still the problem is not completely gone. If anyone has a solution to this I would be grateful.

However, my purpose in this post is to explore a different solution. Instead of doing "twin" runs, can we have twin versions of the biogeochemistry running in the same model? We could certainly edit our existing fennel.h code to make extra versions of all the NPZD variables, but before we launch into that substantial task, I am wondering if others may have already done this? I know the MARBL biogeochemical code has this capability. It was designed for CESM, and also runs with UCLA-ROMS, but not our own Rutgers-ROMS.

Thanks for your help everyone!

Cheers,

Parker

[kfennel]

Hi Parker,

We have encountered the same issue and our solution is the same as in MARBL, i.e. running the two versions of our biogeochemistry (a simplified version of fennel.h) in the same code. Happy to share our code, although it might not be quite what you want if you need the full BGC model. Our simplified model is based on parameterizations derived from the full fennel.h model. We’re quite happy with our results and like the efficiency. Preprint should be available within the next few days.

Will watch this thread as I’m curious what others have to say.

Cheers, Katja

[arango]

I don't know what bit-reproduction means in this context. In the past, I have checked all ecosystem kernels in ROMS to produce identical solutions in serial, serial with partitions, shared-memory, and distributed memory with tiles partitioning 1x1, 2x2, and 3x3 with the upwelling test case. Of course, this is a simpler configuration that I have run with various advection schemes. I turned on DEBUGGING so I could compare all the output NetCDF files bit-by-bit in double-precision with the Unix diff command. I have mentioned that trick in this Forum in the past.

With minor changes, various biogeochemical kernels can be called in ROMS in the same application as BIOLOGY, SEDIMENT, BBL, VEGETATION, and WEC can be activated together. We need to worry about the order of the tracer indices and repetitive secondary variables that may be the same. Note the design of the sea ice model. It only has two structures (Fi and Si) instead of the 39 or more state variables needed. I should have designed the ecosystem model framework in the same way. But when we started, we didn't have the capabilities of Fortran 2003. It is now too late to rewrite all those kernels. If you look at the ROMS-JEDI interface, which I released publicly this week, I use CLASS objects to compact and hide everything, including the associated PROCEDURE calls within the structure. We can use inheritance associations to support various biogeochemical processes. I know, it is a bit more complicated to code and debug, but it is worth it because the code is more compact and flexible.

[pmaccc]

Thanks very much for your answers!

Hernan I will try yourDEBUGGING/diff method.

Katja, I'll be interested to see your preprint. I am also interested to learn what pieces of code had to be edited to add the dual bio variables (e.g. varinfo.yaml and what else).

Cheers,

Parker

[stlaur]

This is speculation on my part, but the sensitivity to tiling may arise/disappear depending on the boundary conditions being used by ROMS.
"Upwelling" has closed+periodic, while I suspect Parker is referring to LiveOcean (so RadNud, presumably).