gmake

gmake

This article first appeared in the HPC Newsletter. It as been revised and updated to more closely match the ROMS makefile.

Over the years, the community has moved from the stance of writing portable Makefiles to a stance of just using a powerful, portable make. The make section described a portable subset of make features. We now delve into some of the more powerful tools available in GNU make. See also Managing projects with GNU Make by Robert Mecklenburg, 2005.

Pattern Rules

The core of make hasn't changed in decades, but concentrating on gmake allows one to make use of its nifty little extras designed by real programmers to help with real projects. The first change that matters to my Makefiles is change from suffix rules to pattern rules. I've always found the .SUFFIXES list to be odd, especially since .f90 is not in the default list. Good riddance to all of that! For a concrete example, the old way to provide a rule for going from file.f90 to file.o is:

 .SUFFIXES: .o .f90 .F .F90

 .f90.o:
  <TAB> $(FC) -c $(FFLAGS) $<

while the new way is:

 %.o: %.f90
  <TAB> $(FC) -c $(FFLAGS) $<

In fact, going to a uniform make means that we can figure out what symbols are defined and use their standard values - in this case, $(FC) and $(FFLAGS) are the built-in default names for the compiler and its options. If you have any questions about this, you can always run make with the -p (or --print-data-base) option. This prints out all the rules make knows about, such as:

 # default

 COMPILE.f = $(FC) $(FFLAGS) $(TARGET_ARCH) -c

Printing the rules database will show variables that make is picking up from the environment, from the Makefile, and from its built-in rules - and which of these sources is providing each value.

Assignments

In the old days, I only used one kind of assignment: = (equals sign). Gmake has several kinds of assignment (other makes might as well, but I no longer have to know or care). An example of the power of GNU make is shown by an example from my Cray X1 Makefile. There is a routine which runs much more quickly if a short function in another file is inlined. The way to accomplish this is through the -O inlinefrom=file directive to the compiler. I can't add this option to FFLAGS, since the inlined routine won't compile with this directive - it is only the one file that needs it. I had:

   FFLAGS = -O 3,aggress -e I -e m
  FFLAGS2 = -O 3,aggress -O inlinefrom=lmd_wscale.f90 -e I -e m

 lmd_skpp.o:
  <TAB> $(FC) -c $(FFLAGS2) $*.f90

The first change I can make to this using other assignments is:

   FFLAGS := -O 3,aggress -e I -e m
  FFLAGS2 := $(FFLAGS) -O inlinefrom=lmd_wscale.f90

The := assignment means to evaluate the right hand side immediately. In this case, there is no reason not to, as long as the second assigment follows the first one (since it's using the value of $(FFLAGS). For the plain equals, make doesn't evaluate the right-hand side until its second pass through the Makefile. However, gmake supports an assignment that avoids the need for FFLAGS2 entirely:

 lmd_skpp.o: FFLAGS += -O inlinefrom=lmd_wscale.f90

What this means is that for the target lmd_skpp.o only, append the inlining directive to FFLAGS. I think this is pretty cool!

One last kind of assignment is to set the value only if there is no value from somewhere else (the environment, for instance):

       FC ?= mpxlf90_r

If we use := or =, we would override the value from the environment.

Include and a Few Functions

One reasonably portable make feature is the include directive. It can be used to clean up the Makefile by putting bits in an include file. The syntax is simply:

 include file

and we use it liberally to keep the project information neat. We can also include a file with the system/compiler information in it, assuming we have some way of deciding which file to include. We can use uname -s to find out which operating system we're using. We also need to know which compiler we're using.

One user-defined variable is called FORT, the name of the Fortran compiler. This value is combined with the result of uname -s to provide a machine and compiler combination. For instance, ftn on Linux is the Cray cross-compiler. This would link to a different copy of the NetCDF library and use different compiler flags than the Intel compiler which might be on the same system.

 # The user sets Fortran Compiler:

    FORT ?= ftn

      OS := $(shell uname -s| sed 's/[\/ ]/-/g'))

 include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk

We pick one include file at compile time, here picking Linux-ftn.mk, containing the Cray cross-compiler information. The value Linux comes from the \code{uname} command, the ftn comes from the user, and the two are concatenated. The sed command will turn the slash in UNICOS/mp into a dash; the native Cray include file is UNICOS-mp-ftn.mk. Strip is a built-in function to strip away any extra white space.

Another tricky system is CYGWIN, which puts a version number in the uname output, such as CYGWIN_NT-5.1. GNU make has quite a few built-in functions plus allows the user to define their own functions. One of the built-in functions allows us to do string substitution:

 MACHINE := $(patsubst CYGWIN_%,CYGWIN,$(MACHINE))

In make, the % symbol is a sort of wild card, much like * in the shell. Here, we match CYGWIN followed by an underscore and anything else, replacing the whole with simply CYGWIN. Another example of a built-in function is the substitution we do in:

 objects = $(subst .F,.o,$(sources))

where we build our list of objects from the list of sources. There are quite a few other functions, plus the user can defined their own. From the book:

GNU make supports both built-in and user-defined functions.

A function invocation looks much like a variable reference, but

includes one or more parameters separated by commas. Most built-in

functions expand to some value that is then assigned to a variable

or passed to a subshell. A user-defined function is stored in a

variable or macro and expects one or more parameters to be passed

by the caller.

We will show some user-defined functions below.

Conditionals

We used to have way too many Makefiles, a separate one for each of the serial/MPI/OpenMP versions on each system (if supported). For instance, the name of the IBM compiler changes when using MPI; the options change for OpenMP. The compiler options also change when using 64-bit addressing or for debugging, which we were manually setting. A better way to do this is to have the user select 64-bit or not, MPI or not, etc, then use conditionals. A complete list of the user definable make variables is given in build Script.

GNU make supports two kinds of if test, ifdef and ifeq (plus the negative versions ifndef, ifneq). The example from the book is:

 ifdef COMSPEC
 #  We are running Windows
 else
 #  We are not on Windows
 endif

An example from the IBM include file is:

 ifdef USE_DEBUG
   FFLAGS += -g -qfullpath
 else
   FFLAGS += -O3 -qstrict
 endif

To test for equality, an example is:

 ifeq ($(USE_MPI),on)
 #  Do MPI things
 endif

or

 ifeq "$(USE_MPI)" "on"
 #  Do MPI things
 endif

The user has to set values for the USE_MPI, USE_OPENMP, USE_DEBUG, and USE_LARGE switches in the Makefile before the compiler-dependent piece is included:

    USE_MPI ?= on
 USE_OPENMP ?=
  USE_DEBUG ?=
  USE_LARGE ?= on

The Makefile uses the conditional assign ?= in case a build script is used to set them in the environment. Be sure to leave the switches meant to be off set to an empty string - the string off will test true on an ifdef test.

Multiple Source Directories the ROMS Way

There's more than one way to divide your sources into separate directories. The choices we have made include nonrecursive make and putting the temporary files in their own $(SCRATCH_DIR) directory. These include the .f90 files which have been through the C preprocessor, object files, module files, and libraries.

Directory Structure

The directory structure of the source code has the top directory, a Master directory, a ROMS directory with a number of subdirectories, and several other directories. Master contains the main program while the rest contain sources for libraries and other files. Note that the bulk of the source code gets compiled into files that become libraries with the ar command, one library per directory. There is also a Compilers directory for the system- and compiler-specific Makefile components.

Conditionally Including Components

The makefile will build the lists of libraries to create and source files to compile. They start out empty at the top of the makefile:

 sources :=
 libraries :=

That's simple enough, but the list of directories to search for these sources will depend on the options chosen by the user, not just in the make options, but inside the ROMS_HEADER file as well. How does this happen? Once make knows how to find the ROMS_HEADER, it is used by cpp to generate an include file telling make about these other options.

 MAKE_MACROS := Compilers/make_macros.mk
 MACROS := $(shell cpp -P $(ROMS_CPPFLAGS) Compilers/make_macros.h > \
                  $(MAKE_MACROS); $(CLEAN) $(MAKE_MACROS))

The make_macros.h file contains blocks such as:

 #ifdef SWAN_COUPLING
   USE_SWAN := on
 #else
   USE_SWAN :=
 #endif

The resulting make_macros.mk file will simply end up with either

 USE_SWAN := on

or

 USE_SWAN :=

This file can then be included by the makefile and the variable USE_SWAN will have the correct state for this particular compilation. We can now use it and all the similar flags to build a list of directories.

We initialize two lists:

 modules  :=
  includes :=    ROMS/Include

Add the adjoint bits:

 ifdef USE_ADJOINT
   modules  +=    ROMS/Adjoint
 endif
 ifdef USE_ADJOINT
   includes +=    ROMS/Adjoint
 endif

Add the bits we'll always need:

 modules  +=    ROMS/Nonlinear \
                ROMS/Functionals \
                ROMS/Utility \ 
                ROMS/Modules
 includes +=    ROMS/Nonlinear \
                ROMS/Utility \
                ROMS/Drivers

Then we add in some more:

 ifdef USE_SWAN
  modules  +=    Waves/SWAN/Src
  includes +=    Waves/SWAN/Src
 endif

 modules  +=    Master
 includes +=    Master Compilers

Now that our lists are complete, let's put them to use:

 vpath %.F $(modules)
 vpath %.h $(includes)
 vpath %.f90 $(SCRATCH_DIR)
 vpath %.o $(SCRATCH_DIR)

 include $(addsuffix /Module.mk,$(modules))

 CPPFLAGS += $(patsubst %,-I%,$(includes))

1. vpath is a standard make feature for providing a list of directories for make to search for files of different types. Here we are saying that *.F files can be found in the directories provided in the $(modules) list, and so on for the others.
2. For each directory in the $(modules) list, make will include the file Module.mk that is found there. More on these later.
3. For each directory in the $(includes) list, add that directory to the list searched by cpp with the -I flag.

Library Details

The directory structure we are using has the top directory, an Include directory, several directories which contain sources for libraries, and the directory for the main programs (Drivers). There is also a directory for the compiler-specific Makefile components (Compilers).

Here is a complete example of a library Makefile component:

 local_lib := libNLM.a
 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 libraries += $(local_lib)
 sources += $(local_src)

 $(local_lib): $(local_objs)
 $(AR) $(ARFLAGS) $@ $^

The only thing that changes from one to the next is the name of the library to build. I'm actually keeping track of the sources with and without the subdirectory part of their name. The objects will go into the top directory, so they shouldn't have the directory in their list. I only need the path_srcs for creating the dependency information; make itself knows to look in the directories because of a vpath command. We are also updating a libraries variable, adding the local library to the global list.

Main Program

The main program is in a directory called Drivers and its Module.mk is similar to the library one:

 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 sources += $(local_src)

 $(BIN): $(libraries) $(local_objs)
 $(LD) $(FFLAGS) $(LDFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS)

Instead of a rule for building a library, we have a rule for building a binary. In this case, the name of the binary depends on if it's parallel or not and is defined elsewhere. The binary depends on the libraries getting compiled first, as well as the local sources. During the link, the $(libraries) are compiled from the sources in the other directories, while $(LIBS) are exteral libraries such as NetCDF and mpich.

Top Level Makefile

Now we get to the glue that holds it all together. We first initialize some of the global lists:

 #  Initialize some things.

 clean_list := core *.o *.mod *.f90 lib*.a
 sources :=
 path_srcs :=
 libraries :=

 objects = $(subst .F,.o,$(sources)

Next is the subdirectory function we already presented, followed by the user-defined switches and the compiler-dependent includes. Then we have the pattern rules, also shown above. Finally we get to the meat of the includes:

.PHONY: all

 all: $(BIN)

 modules := Adjoint Ice Modules Nonlinear Representer Tangent Utility Drivers

 includes := Include Adjoint Nonlinear Tangent Drivers

 vpath %.F $(modules)
 vpath %.h $(includes)

 include $(addsuffix /Module.mk,$(modules))

 CPPFLAGS += $(patsubst %,-I%,$(includes))

 .PHONY: clean

 clean:
  <TAG>$(RM) $(clean_list) $(BIN)

all is the first target that gets seen by make, making it the default target. In this case, we know there is only the one binary, whose name we know - the book shows how to do more than one binary. The modules are the list of subdirectories containing a Module.mk we need to include. clean is the target that removes all the cruft we don't want to keep. Both all and clean are phony targets in that no files of those names get generated - make has the .PHONY designation for such targets.