Ocean Modeling Discussion

ROMS/TOMS

Search for:
It is currently Tue Dec 11, 2018 4:43 am




Post new topic Reply to topic  [ 7 posts ] 

All times are UTC

Author Message
 Post subject: Test suite
PostPosted: Mon May 14, 2007 3:24 am 
Offline
User avatar

Joined: Tue Jul 01, 2003 4:12 am
Posts: 511
Location: NIWA
I think ROMS should have a test suite and I am writing one. At the moment it consists of a script that looks like this:

#! /bin/sh

make clean && make ROMS_APPLICATION=BASIN
make clean && make ROMS_APPLICATION=BENCHMARK
make clean && make ROMS_APPLICATION=BIO_TOY

I'm sure you get the idea. It's amazing how many problems can be identified with something this simple. Just look at my recent bug reports on the Trac system.

Any suggestions about how this could be expanded & made effective but relatively painless are welcome.


Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: Mon May 14, 2007 1:32 pm 
Offline

Joined: Fri Apr 25, 2003 9:22 pm
Posts: 123
Location: USGS
Mark,

We agree that a test suite is a great idea for ROMS! Sachin Bhate, who is working on ROMS software tools under the NOPP Community Sediment Transport Project, has been working on just such a regression testing script (using Perl) to run a list of test cases in scalar, openmp and mpi modes, and do some simple checks on the netcdf results. Hopefully Sachin will post a follow-up here -- I know he had basically finished an alpha version of the script for ROMS 2.2 version when ROMS 3.0 was released, so now he's finishing up that. Perhaps you guys can work together on this!

-Rich


Top
 Profile  
Reply with quote  
 Post subject: Regression Test Package
PostPosted: Mon May 14, 2007 2:06 pm 
Mark,

Elaborating on the point Rich just made, Yes I am working on a Regression test package for ROMS CSTM code. I did an earlier alpha release of the regression test package for ROMS 2.2. But, it was almost too close to the ROMS3.0 release, so decided to put it a hold on final release of the test package and upgrade it to work with ROMS3.0. Currently, I am working on upgrading it to work with ROMS3.0. It should be out for testing sometime this week (hopefully). The testing code is in Perl. You can look at some old documentation on it at this site http://skumar.tiddlyspot.com/ to have an idea about the test package.

You are more than welcome to be an alpha tester for the upcoming release of testing package and provide me with some valuble feedback.

-Sachin.


Top
  
Reply with quote  
 Post subject:
PostPosted: Mon May 14, 2007 2:43 pm 
Offline
Site Admin
User avatar

Joined: Wed Feb 26, 2003 4:41 pm
Posts: 1049
Location: IMCS, Rutgers University
Regression tests are used for production type applications whose configuration is usually the same but the source code evolves in time. However, what I think Mark is talking about is about a simple script to check if the model compiles at all. We need to clear compilation errors and other logic errors that arise as the code evolves. This usually involves turning on and off CPP options and checking for parallel (distributed- or shared-memory) bugs.

A regression test will fail for this type of code pruning. However, regression tests are good to check the vadility of new algorithms when compared with old algorithms. It will tell us that the solution is different, but it will not give us clues on how to resolve such differences.

ROMS has several sanity checks for the nonlinear, tangent linear, and adjoint numerical kernels that usually need to be done manually. There is no way to write a script for it, since we require a very sofisticated script training in algebra, adjoints, numerical kernel, and physics.


Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: Mon May 14, 2007 10:22 pm 
Offline
User avatar

Joined: Tue Jul 01, 2003 4:12 am
Posts: 511
Location: NIWA
My ideas on what a test suite should cover are very preliminary, but I guess I have been thinking of a regression test. I started with a simple script to make each of the idealised test applications in turn, because that is very easy to script with the current directory structure and uncovers enough bugs to keep me busy for a while. However it's clear one also needs to try *running* the model.

I would be happy to be a test tester for Sachin.


Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: Mon May 14, 2007 10:33 pm 
Offline
User avatar

Joined: Wed Jul 02, 2003 5:29 pm
Posts: 3516
Location: IMS/UAF, USA
I have long thought such a test option would be extremely useful. I agree that compiling for all the different cases is a good start, but running serial, OpenMP, and MPI would be a wonderful addition if it were possible. Hernan can tell you that I don't understand the OpenMP parts to be trusted not to break it. Then there's my stuff with all the goodies like tides, ice, point sources, etc, which don't always get tested for each little change.


Top
 Profile  
Reply with quote  
 Post subject:
PostPosted: Tue May 15, 2007 1:39 pm 
Offline
User avatar

Joined: Mon Apr 28, 2003 5:44 pm
Posts: 468
Location: Rutgers University
A testing framework that users could easily adapt to their own application for finer levels of granularity in testing would indeed be valuable. This would allow the communitiy to test the interaction of certain elements of the code which, when one considers all the possible combinations of CPP options, is practically-speaking infinite.

For the CBLAST application for example, I could test operation with/without various open boundary options, with/without surface forcing imposed via bulk fluxes, direct fluxes, or analytical options, with all vertical turbulence closures. These combinations exercise almost all possible netcdf I/O cases. For any given user's application, this sort of test would reassure a user like me that at least for the parts of the code that I care about :P everything is functioning correctly.

So I guess it would take some sort of perl script with a perl database (?) where a user could check of a table or matrix of CPP options to be tested.

_________________
John Wilkin: IMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu


Top
 Profile  
Reply with quote  
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group