gmake

gmake

This article first appeared in the HPC Newsletter. It as been revised and updated to more closely match the ROMS makefile.

Over the years, the community has moved from the stance of writing portable Makefiles to a stance of just using a powerful, portable make. The make section described a portable subset of make features. We now delve into some of the more powerful tools available in GNU make. See also Managing projects with GNU Make by Robert Mecklenburg, 2005.

Pattern Rules

The core of make hasn't changed in decades, but concentrating on gmake allows one to make use of its nifty little extras designed by real programmers to help with real projects. The first change that matters to my Makefiles is change from suffix rules to pattern rules. I've always found the .SUFFIXES list to be odd, especially since .f90 is not in the default list. Good riddance to all of that! For a concrete example, the old way to provide a rule for going from file.f90 to file.o is:

 .SUFFIXES: .o .f90 .F .F90

 .f90.o:
  <TAB> $(FC) -c $(FFLAGS) $<

while the new way is:

 %.o: %.f90
  <TAB> $(FC) -c $(FFLAGS) $<

In fact, going to a uniform make means that we can figure out what symbols are defined and use their standard values - in this case, $(FC) and $(FFLAGS) are the built-in default names for the compiler and its options. If you have any questions about this, you can always run make with the -p (or --print-data-base) option. This prints out all the rules make knows about, such as:

 # default

 COMPILE.f = $(FC) $(FFLAGS) $(TARGET_ARCH) -c

Printing the rules database will show variables that make is picking up from the environment, from the Makefile, and from its built-in rules - and which of these sources is providing each value.

Assignments

In the old days, I only used one kind of assignment: = (equals sign). Gmake has several kinds of assignment (other makes might as well, but I no longer have to know or care). An example of the power of GNU make is shown by an example from my Cray X1 Makefile. There is a routine which runs much more quickly if a short function in another file is inlined. The way to accomplish this is through the -O inlinefrom=file directive to the compiler. I can't add this option to FFLAGS, since the inlined routine won't compile with this directive - it is only the one file that needs it. I had:

   FFLAGS = -O 3,aggress -e I -e m
  FFLAGS2 = -O 3,aggress -O inlinefrom=lmd_wscale.f90 -e I -e m

 lmd_skpp.o:
  <TAB> $(FC) -c $(FFLAGS2) $*.f90

The first change I can make to this using other assignments is:

   FFLAGS := -O 3,aggress -e I -e m
  FFLAGS2 := $(FFLAGS) -O inlinefrom=lmd_wscale.f90

The := assignment means to evaluate the right hand side immediately. In this case, there is no reason not to, as long as the second assigment follows the first one (since it's using the value of $(FFLAGS). For the plain equals, make doesn't evaluate the right-hand side until its second pass through the Makefile. However, gmake supports an assignment that avoids the need for FFLAGS2 entirely:

 lmd_skpp.o: FFLAGS += -O inlinefrom=lmd_wscale.f90

What this means is that for the target lmd_skpp.o only, append the inlining directive to FFLAGS. I think this is pretty cool!

One last kind of assignment is to set the value only if there is no value from somewhere else (the environment, for instance):

       FC ?= mpxlf90_r

If we use := or =, we would override the value from the environment.

Include and a Few Functions

One reasonably portable make feature is the include directive. It can be used to clean up the Makefile by putting bits in an include file. The syntax is simply:

 include file

and we use it liberally to keep the project information neat. We can also include a file with the system/compiler information in it, assuming we have some way of deciding which file to include. We can use uname -s to find out which operating system we're using. We also need to know which compiler we're using.

One user-defined variable is called FORT, the name of the Fortran compiler. This value is combined with the result of uname -s to provide a machine and compiler combination. For instance, ftn on Linux is the Cray cross-compiler. This would link to a different copy of the NetCDF library and use different compiler flags than the Intel compiler which might be on the same system.

 # The user sets Fortran Compiler:

    FORT ?= ftn

      OS := $(shell uname -s| sed 's/[\/ ]/-/g'))

 include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk

We pick one include file at compile time, here picking Linux-ftn.mk, containing the Cray cross-compiler information. The value Linux comes from the \code{uname} command, the ftn comes from the user, and the two are concatenated. The sed command will turn the slash in UNICOS/mp into a dash; the native Cray include file is UNICOS-mp-ftn.mk. Strip is a built-in function to strip away any extra white space.

Another tricky system is CYGWIN, which puts a version number in the uname output, such as CYGWIN_NT-5.1. GNU make has quite a few built-in functions plus allows the user to define their own functions. One of the built-in functions allows us to do string substitution:

 MACHINE := $(patsubst CYGWIN_%,CYGWIN,$(MACHINE))

In make, the % symbol is a sort of wild card, much like * in the shell. Here, we match CYGWIN followed by an underscore and anything else, replacing the whole with simply CYGWIN. Another example of a built-in function is the substitution we do in:

 objects = $(subst .F,.o,$(sources))

where we build our list of objects from the list of sources. There are quite a few other functions, plus the user can defined their own. From the book:

GNU make supports both built-in and user-defined functions.

A function invocation looks much like a variable reference, but

includes one or more parameters separated by commas. Most built-in

functions expand to some value that is then assigned to a variable

or passed to a subshell. A user-defined function is stored in a

variable or macro and expects one or more parameters to be passed

by the caller.

We will show some user-defined functions below.

Conditionals

We used to have way too many Makefiles, a separate one for each of the serial/MPI/OpenMP versions on each system (if supported). For instance, the name of the IBM compiler changes when using MPI; the options change for OpenMP. The compiler options also change when using 64-bit addressing or for debugging, which we were manually setting. A better way to do this is to have the user select 64-bit or not, MPI or not, etc, then use conditionals. A complete list of the user definable make variables is given in build Script.

GNU make supports two kinds of if test, ifdef and ifeq (plus the negative versions ifndef, ifneq). The example from the book is:

 ifdef COMSPEC
 #  We are running Windows
 else
 #  We are not on Windows
 endif

An example from the IBM include file is:

         ifdef USE_DEBUG
   FFLAGS += -g -qfullpath
 else
   FFLAGS += -O3 -qstrict
 endif

To test for equality, an example is:

 ifeq ($(USE_MPI),on)
 #  Do MPI things
 endif

or

 ifeq "$(USE_MPI)" "on"
 #  Do MPI things
 endif

The user has to set values for the USE_MPI, USE_OPENMP, USE_DEBUG, and USE_LARGE switches in the Makefile before the compiler-dependent piece is included:

    USE_MPI ?= on
 USE_OPENMP ?=
  USE_DEBUG ?=
  USE_LARGE ?= on

The Makefile uses the conditional assign ?= in case a build script is used to set them in the environment. Be sure to leave the switches meant to be off set to an empty string - the string off will test true on an ifdef test.

Multiple Directories for Sources

There's more than one way to do it, but ROMS is using the method of putting all the temporary .f90 and .o files in the directory specified in the SCRATCH_DIR definition. The top directory has the master makefile, which includes a Makefile bit from each subdirectory, called Module.mk. The top makefile starts with an empty list of sources. In each subdirectory, we find local sources by simply listing all the .F files and appending this to the master list.

In makefile:

 sources :=
 include somedir/Module.mk

In somedir/Module.mk:

 local_src := $(wildcard $(subdirectory)/*.F)
 sources += $(local_src)

Here, we are using the wildcard function to search the subdirectory for it's local sources. Each subdirectory is resetting the local_src variable, but that's OK because we're saving the values in the global sources variable. The other sneaky thing here is the user-defined subdirectory function, from the Gnu make book:

 subdirectory = $(patsubst %/Module.mk,%,$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))

This does the right thing to figure out which subdirectory we are in from make's internal list of the Makefiles it is parsing. It depends on all the subdirectory include files being called Module.mk.

Library Details

The directory structure we are using has the top directory, an Include directory, several directories which contain sources for libraries, and the directory for the main programs (Drivers). There is also a directory for the compiler-specific Makefile components (Compilers).

Here is a complete example of a library Makefile component:

 local_lib := libNLM.a
 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 libraries += $(local_lib)
 sources += $(local_src)

 $(local_lib): $(local_objs)
 $(AR) $(ARFLAGS) $@ $^

The only thing that changes from one to the next is the name of the library to build. I'm actually keeping track of the sources with and without the subdirectory part of their name. The objects will go into the top directory, so they shouldn't have the directory in their list. I only need the path_srcs for creating the dependency information; make itself knows to look in the directories because of a vpath command. We are also updating a libraries variable, adding the local library to the global list.

Main Program

The main program is in a directory called Drivers and its Module.mk is similar to the library one:

 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 sources += $(local_src)

 $(BIN): $(libraries) $(local_objs)
 $(LD) $(FFLAGS) $(LDFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS)

Instead of a rule for building a library, we have a rule for building a binary. In this case, the name of the binary depends on if it's parallel or not and is defined elsewhere. The binary depends on the libraries getting compiled first, as well as the local sources. During the link, the $(libraries) are compiled from the sources in the other directories, while $(LIBS) are exteral libraries such as NetCDF and mpich.

Top Level Makefile

Now we get to the glue that holds it all together. We first initialize some of the global lists:

 #  Initialize some things.

 clean_list := core *.o *.mod *.f90 lib*.a
 sources :=
 path_srcs :=
 libraries :=

 objects = $(subst .F,.o,$(sources)

Next is the subdirectory function we already presented, followed by the user-defined switches and the compiler-dependent includes. Then we have the pattern rules, also shown above. Finally we get to the meat of the includes:

.PHONY: all

 all: $(BIN)

 modules := Adjoint Ice Modules Nonlinear Representer Tangent Utility Drivers

 includes := Include Adjoint Nonlinear Tangent Drivers

 vpath %.F $(modules)
 vpath %.h $(includes)

 include $(addsuffix /Module.mk,$(modules))

 CPPFLAGS += $(patsubst %,-I%,$(includes))

 .PHONY: clean

 clean:
  <TAG>$(RM) $(clean_list) $(BIN)

all is the first target that gets seen by make, making it the default target. In this case, we know there is only the one binary, whose name we know - the book shows how to do more than one binary. The modules are the list of subdirectories containing a Module.mk we need to include. clean is the target that removes all the cruft we don't want to keep. Both all and clean are phony targets in that no files of those names get generated - make has the .PHONY designation for such targets.