ROMS/TOMS Developers

Algorithms Update Web Log
Next Page »

kate - January 24, 2013 @ 18:09
Ice model debugging fun- Comments (1)

Yes, debugging can be fun. It’s a job where you know there’s an answer to be found in some finite amount of time. It’s not always as quick as you expect, though.

Our Arctic domain has been run for years and doesn’t look too horrible, but there’s a bit too much ice in the summers. I’ve tried a couple of different albedo schemes, then found one more to try. It’s by Ebert and Curry and is the only one to include the effects of melt ponds. Why hadn’t I ever tried it before? A quick test with PGI Fortran showed me why - the thing blew up in a few timesteps. Maybe that had happened before and I found it easier to just go back to another albedo option.

This time, I recompiled for the debugger and fired up totalview. The thing ran this time, no problem. My favorite trick in this situation is to try another compiler. I’ve also got gfortran, so I gave that a whirl. That one ran fine too - but then it blew up three steps after a restart. How strange to run for hundreds of days, then fail right after a restart. Both this and the earlier PGI failure reported trouble from the ice_frazil routine, so I looked to the temperature field.

Totalview and gfortran don’t play nice, so it’s back to PGI and this time it failed in the same way as gfortran, in the debugger even. I focused on the i,j point reported by the ice_frazil error message and I discovered that in the very first timestep, the surface salinity value became nan, then the entire water column of salinity was set to zero. I’m not quite sure how that loss of nan happened... Those zero salinities with sub-zero temperatures generated a bunch of ice at just the one point and the water column then became unstable.

The nan at the surface led me to check the surface tracer forcing, which indeed was nan for salt. On restart, the ice concentration contained a very tiny non-zero value which introduced trouble to the computation of the ice albedo there. I have an ice_limit routine which keeps ice fields well-behaved, but I wasn’t running it from within get_state. I looked into calling it there, but ice_limit works on the MPI tile while get_state does not. I managed to add a call to ice_limit from the ini_fields routine instead. It’s running again - in both PGI and gfortran. The problem this time was due to a single precision restart file, but I already knew I wanted a call to ice_limit on startup.

While I was poking around in there, I also fixed it so that trouble in ice_frazil really does cause the job to end. I had added a check on exit_flag, but was then broadcasting the value from process 0, not broadcasting the bad value to the rest of the nodes - unless process 0 was the one in trouble. It needed a “MAX” reduction so that any process in trouble made that trouble known to the head node.

Anyway, I learned something, which is always fun.

kate - January 19, 2011 @ 14:39
Hong Kong training- Comments (0)

I have to write up these trips for my work, so why not post here too?

Part 1 written Tuesday Jan 4:

Dale Haidvogel invited me to help him teach a ten-day ROMS training
in Hong Kong. He had said he would go early with his family, so Rob
and I bought tickets to arrive early. Dale’s family then changed
their minds and they spent New Year’s in Toronto. We arrived late
afternoon on Dec 30 for a class starting Jan 5. We were met by a
taxi driver with a card holding my name - it was so nice. Gan and
two of his students met us at the apartment where they are putting
us up.

The next morning one of the students, LinLin, came and gave us a
campus tour. HKUST has most everything in one enormous building,
with eating establishments on the lower levels, class rooms on the
main level, and offices above. HKUST is only twenty years old, but
has become one of the best universities in Hong Kong, with the top
business school in all of Asia. LinLin took us to meet Gan, then
we went to visit the computer lab where my part of the class will
be held. The two computer guys who run all the computers in the
Math department were there, explaining that the PCs running windows
would have a virtual Linux, then they could connect to Gan’s cluster
for doing the actual work. Seemed dicey to me, plus Gan hadn’t sent
them the list of software I’d asked for (with all the Python modules),
so that was worrisome as well. We agreed to meet with them again
on Tuesday, the day before the class started. Gan then took us out
for a big dim sum lunch.

That afternoon Rob and I went to Hang Hau to buy Octopus cards and
to look for an extra electric plug converter. The Octopus cards can
be loaded with money, then debitted by the buses and the metro, so
you don’t have to carry tons of change around. My last Seattle trip
was filled with scrambling for bus money.

We spent the evenings and some early mornings working on talks while
touring around during the daylight hours. We got out to Tsuen Wan
on Saturday to see a Taoist temple, then to Stanley Market to see
what was there (I wanted a Chinese chop and we were told that was
the place to go). On Sunday we went out by the airport to see the
largest Buddha in the world at the Po Lin monastery . You can get
there by aerial tramway, which happened to have a two hour wait -
we’d have taken a bus if we’d known! Gan told us there were many
tourists from the mainland over for the holidays and said it should
be quiet now until Chinese New Years.

Dale arrived on Monday the 3rd and we met him, Gan, and Olivia for
lunch. We talked over some of the plans and they told us that 40% of
those signed up are university students, maybe 40% already have some
ROMS experience and want more of the theory (Dale’s part of the
class) and some have absolutely no necessary skills other than
perhaps English, in spite of being told they should know math,
computers, oceanography before coming.

On Tuesday we met with the computer center guys again and were told
that Cook had found a way to get the PCs to run Linux, running one
image from a server. What a relief! Rob then handed over a DVD of
/usr/local on which he had installed everything we need. It’s going
to make everything so much simpler for the students. They can do
everything on the quad-core desktop systems rather than having to
log into Gan’s cluster.

Olivia is coordinating with a copy center and they want our talks
two days in advance for making copies for everyone. Dale and I are
both working hard to prepare things on time. Hernan is expecting me
to put all the talks on the ROMS Wiki, but I told him it has to
wait until the talks are all done. Classes start tomorrow!

Part 2: Fairbanks two weeks later:

Wow, that was one intense class! My talks are now on the wiki:

The first day wasn’t bad, with my part being an intro to Linux and
a tour of online ROMS resources. We had a nice group lunch with
everyone saying their name and where they are from, but I caught
practically none of it. There was a group from Korea, then a person
from Canada and one from Virginia, but I think the rest were Chinese.
Gan had opened up 35 spaces for people, reserving 5 for his group
since the lab has 40 workstations.

The second day we got into the thick of it. Dale had asked me to
make sure they’d be able to run five different test problems. Three
are in the main trunk of the ROMS code, two are not. One of the
latter contains a C language Bessel function for initial conditions
so I hacked the Makefile to link to that thing - but only for that
case. How to do that? Well, with git we have the technology. Download
ROMS with git-svn to a master branch, then create two other branches,
one for the circle problem, one for the user’s own changes. I wrote
down the commands to set that all up and asked the users to go
through them. I went way too fast and we were dealing with weird
Bessel problems for days afterwards.

In the evenings, Rob and I were working on the python tools for
ROMS, aka pyroms. They are still very rough around the edges, but
contain a surprising amount of useful capabilities. They do however
have a frightful list of dependencies which I never quite got working
on my Mac, so Rob set me up with a VirtualBox containing Linux with
the whole works. Fred at Rutgers sent me tips and codes and even a
patch to get me going. My pyroms talks were a bit scattered and
last minute, but by golly, we signed up for the Sunday outing and
were determined to make the most of it. Twenty-three of us got a
quick tour of the Hong Kong highlights, from a reunification park,
to the Peak, to the south side of the island (lunch at Aberdeen,
Repulse Bay, and Stanley), then back to Kowloon for the Hollywood
Parade of Stars.

On Tuesday afternoon I went through setting up a grid in pyroms,
then copying an existing case in ROMS, then modifying that to work
with the new grid. I changed the open boundary from the east side to
the south side and showed them several ways to screw up and get no
flow at all. They then repeated the getting no flow part and
expected me to fix them up. Hence the Thursday slide about becoming
a detective.

On Wednesday afternoon I gave the talks about MPI and fish, but they
really wanted to see the setting up of a 3-D problem, so that’s what
we did on Thursday afternoon. We added tides from Oregon State, with
caveats about the 18.6 year components not being included, then we
added initial conditions from SODA, finally we added atmospheric
forcing which Rob had prepared for me. I showed them recompiling
umpteen times and mentioned that not everyone has the personality
for it. We finished with ROMS seg faulting on reading in one of
those files. So very true to life!

On Friday the morning was a Q&A session with Dale, Gan and I taking
turns answering the questions. Lots and lots of good questions. That
afternoon we were all ready to call it a day, except Rob volunteered
to lead a class on getting VirtualBox going on their laptops with
all the pyroms goodies - it took all afternoon!

The students were very involved, staying in the lab until we had to
throw them out at 5:30 every evening, even on the Saturday. There
were always many questions, though about half the students were
totally silent the whole time. I was asked to slow down, even when
I thought I was using my slow voice for foreigners - it wasn’t slow
enough, especially the hands-on parts. The Koreans invited us to
come visit them - perhaps in the fall. One is a former student of
Dale’s, now a professor (BJ), and two are researchers at KORDI, one
wanting a coupled ROMS-WRF model. It was good to see BJ and he did a
great job of asking the questions he thought his people would want
to hear the answers to. Gan told us half a dozen times some students
are looking for a quick model to learn, a black box where you hit a
button to make it go. Those students were disappointed, but I think
it was overall a positive experience. Gan was very impressed with
what Rob could contribute to the lab portion, answering many
questions, including those about setting stuff up on their home

Gan told us the idea for this training came about when he visits China
and gets bombarded with ROMS questions of all sorts, to the point where
he can’t get to the coffee during breaks. Now he can tell them all they
should have come to the class! If we do it again, it should be easier with
so much material already prepared. Gan says he started a competition between
his students for who can master git first.

P.S. I finally have a better understanding of build.bash, how and why
one would use it. I do feel that it’s unfortunate that it is not
a thing of mean, lean, clean beauty.

kate - July 19, 2010 @ 20:01
More source code control- Comments (3)

I’ve been using git and I’m quite happy with it. Meanwhile, there’s a competing project with many, many of the same attributes of git, called Mercurial. They pride themselves on a cleaner user interface, so you might like it better.

We’ve now got a longer commute and my husband found some podcasts to listen to at (This Week in Tech). We like the TWIT shows, but they tend to focus on things like smart phones and iPads. But they’ve got quite a selection of other shows, including FLOSS, which is about free software. There was a FLOSS show about Mercurial, explaining that it and git were started on the same day to solve the same problem. Mercurial not only has a way to access svn archives, but one to access git archives. Mercurial is written in Python and is being used by the Python community (which explains why Rob Hetland thought Mercurial was getting more traction than git). The result is that both are probably here for the long haul.

kate - June 25, 2010 @ 0:05
Still More About Git- Comments (0)

Note: this is a copy of something submitted to the ARSC HPC newsletter.

I’ve written about git here twice before, where I described needing
to deal with legacy svn repositories. Perhaps it’s time to get a
little more specific about how I’m using it so you can all be
suitably horrified.

First of all, I started playing with git before I found a system
with a working git-svn command. I created a repository with a main
trunk which mimics the svn code, plus a branch with exploratory code
we’re not ready to share via svn just yet. I have these codes in a bare
“origin” repository in my home directory on my desktop system. A
bare repository has the database, but no working files - it is
unsafe to push to a non-bare repository.

I can do a “git clone” of the “origin” out to any of the supercomputers
here at ARSC. The “origin” repository doesn’t know about any of the
clones, but each clone can pull updates from the “origin” and push
changes back into it. Good practice means making sure the “origin”
is always up to date with any changes made anywhere.

It turns out the system with the working git-svn is a laptop, though
security prevents the laptop from hosting the “origin”. The laptop
also has a clone of “origin” in addition to the repository generated
by “git svn clone” and a third repository from “git svn clone” on
a colleague’s trunk version. These three repositories know something
about each other. How does that work? Say we create them with:

     git clone  git_code
     git svn clone  svn_code
     git svn clone  trunk_code

We’ll assume that everything is on the master branch below, but it
could be on some other active branch.


I can go into git_code and type:

     git remote add svn_code ../svn_code
     git remote add trunk_code ../trunk_code

Likewise, I go into svn_code and type:

     git remote add git_code ../git_code
     git remote add trunk_code ../trunk_code

Because my view of trunk_code is read-only, I have no need for it to
know about the others.

Note that the “git clone” operation automatically generates a remote
called “origin”.

Remote Tracking Branches

I’m not quite set up yet to use the remotes. They need to be turned
into remote tracking branches, branches in the local repository
which point to the remotes. To do this, go into each directory with
remotes and type:

    git remote update

This will create a new tracking branch for each of the remote sites.
These branches show up with “git branch -r” or “git branch -a” but
not with simply “git branch”.

Incorporating Svn Updates

Suppose you decide to check for updates from the main trunk, after a
period of weeks (or years) and you want to bring those changes to
your code. Using svn, an “svn update” would bring a copy of that trunk
directory up to date all in one fell swoop. Ditto with a merge from the
old to the current, even if some dozens of updates have been checked in.

Using git, you invoke the following from the trunk_code directory:

     git svn rebase

This will bring over the changes one at a time, with the svn
check-in message attached to each change.

Now, go to the svn_code directory and type:

    git remote update

Now you can apply the changes via “git cherry-pick”, one change at a
time. Conflicts get resolved as they are encountered, commit by commit.
This could get tedious if you let it go too long...

Once that’s done, it’s time to commit these changes to your svn

    git svn dcommit

In the git_code directory, repeat the “git remote update” and the
“git cherry-pick” commands, then:

    git commit
    git push origin master

Now your workstation’s “origin” is up to date as well.

Incorporating Git Updates

Sometimes the changes come from you, over on one of the
supercomputers. On that supercomputer, type:

    git push origin master

Now the changes are on the workstation, but not the laptop. On the
laptop in the git_code directory, type:

    git pull origin master

Now go to the svn_code direcotory and:

    git remote update
    git cherry-pick 
    git svn dcommit

This brings your changes over to your svn branch.

Last Thoughts

If I were to start fresh, I would start with the “git svn clone” operation
and create a bare clone of that on the workstation. I could still do
that if I knew how to pull the development branch code from the git_code
to the svn_code repository (any hints from you all?). I’m still enjoying
using git, though it makes me feel like an old dog faced with many
new tricks!

Also, I talked about git at a ROMS meeting and that talk is available

kate - April 17, 2010 @ 16:17
Hacker’s Dream (Python for ROMS, part 0)- Comments (0)

A friend told me he can always count on me to do something outside the norm. All my colleagues use Matlab? Well, I can dream of a world in which we can do all the ROMS pre- and post-processing in an Open Source manner. I’ve played a bit with NCL and I really like some aspects of it. However, it doesn’t allow you to build tools that have a gui (graphical user interface). Grid generation and land-mask editing are two examples where a gui is really, really nice.

Rob Hetland and others have been working with Python to build up a toolbox for working with ROMS NetCDF files. Before we get to the tools, let’s talk about what they require:

  • Python itself. You will need a version between 2.4 and 2.6, inclusive. There is a newer 3.x series, but it is incompatible and therefore not to be trusted just yet.
  • numpy. Get a version that’s reasonably new, don’t count on the version 1.0.1 that you found already on the system. How do you tell the version? At the interactive prompt from python, type “import numpy” then “numpy.__version__”.
  • scipy.
  • matplotlib. This is a plotting package to reproduce the Matlab plotting, complete with everything you might not like about Matlab plotting. This also contains the gui tools, but it depends on an underlying gui package, be that tk, wx, X11, or qt. If it can’t find any of them, you are out of luck.
  • basemap. These are the map tools for Python, complete with etopo2 in the examples directory (user beware).
  • netCDF4. This sits on top of the hdf5 and netcdf4 libraries.
  • ipython. Fred Castruccio recommends using “ipython -pylab” for interactive fun. It preloads both numpy and the matplotlib pylab package.

So how does one go about acquiring all this stuff? There are choices, from a prebuilt package with everything from Enthought, to fetching each as a package for your OS, to building everything from scratch. I’ve honestly had a bit of a challenge for various reasons, but here are a few things I’ve learned:

  • Python 2.6.5 does not compile on the Mac with either gcc-4.0.1 or gcc-4.4.0, but requires gcc-4.5.0 to get around an internal compiler error when building the datetime module. Matplotlib will not load without datetime.
  • Python itself uses the “standard” method of “configure; make; make install”, but after that, all packages play by the Python rules.
  • The python rules are:
    1. In the directory of the package you want to install, type “python build”. This puts files into a local build directory.
    2. Type “python install” with an optional “--prefix={dir}” option.
    3. Type “python config” to see stuff like what guis matplotlib has found.
    4. If you use the optional non-standard directory, tell Python about it with the PYTHONPATH environment variable.
    5. If you want to link to packages that are themselves in a nonstandard place, use the environment variables CPPFLAGS=-I{dir} and LDFLAGS=-L{dir}. This adds to the list of places it will look - I don’t know how to take away from the list.
  • The Python package for Mac comes as a “universal binary”, working on both ppc and i386 architectures. That means that any other libraries that are involved, such as hdf5 and netcdf4, also have to be compiled this way or you will get an error.
  • If you get rough packages from Fred, make sure to delete the pre-existing build directory and any other binary object files you might find.

See? It really is a hacker’s dream, with fifty ways to mess up. ;)