gmake

gmake

This article first appeared in the HPC Newsletter. It as been revised and updated to more closely match the ROMS makefile.

Over the years, the community has moved from the stance of writing portable Makefiles to a stance of just using a powerful, portable make. The make section described a portable subset of make features. We now delve into some of the more powerful tools available in GNU make. See also Managing projects with GNU Make by Robert Mecklenburg, 2005.

Pattern Rules

The core of make hasn't changed in decades, but concentrating on gmake allows one to make use of its nifty little extras designed by real programmers to help with real projects. The first change that matters to my Makefiles is change from suffix rules to pattern rules. I've always found the .SUFFIXES list to be odd, especially since .f90 is not in the default list. Good riddance to all of that! For a concrete example, the old way to provide a rule for going from file.f90 to file.o is:

 .SUFFIXES: .o .f90 .F .F90

 .f90.o:
  <TAB> $(FC) -c $(FFLAGS) $<

while the new way is:

 %.o: %.f90
  <TAB> $(FC) -c $(FFLAGS) $<

In fact, going to a uniform make means that we can figure out what symbols are defined and use their standard values - in this case, $(FC) and $(FFLAGS) are the built-in default names for the compiler and its options. If you have any questions about this, you can always run make with the -p (or --print-data-base) option. This prints out all the rules make knows about, such as:

 # default

 COMPILE.f = $(FC) $(FFLAGS) $(TARGET_ARCH) -c

Printing the rules database will show variables that make is picking up from the environment, from the Makefile, and from its built-in rules - and which of these sources is providing each value.

Assignments

In the old days, I only used one kind of assignment: = (equals sign). Gmake has several kinds of assignment (other makes might as well, but I no longer have to know or care). An example of the power of GNU make is shown by an example from my Cray X1 Makefile. There is a routine which runs much more quickly if a short function in another file is inlined. The way to accomplish this is through the -O inlinefrom=file directive to the compiler. I can't add this option to FFLAGS, since the inlined routine won't compile with this directive - it is only the one file that needs it. I had:

   FFLAGS = -O 3,aggress -e I -e m
  FFLAGS2 = -O 3,aggress -O inlinefrom=lmd_wscale.f90 -e I -e m

 lmd_skpp.o:
  <TAB> $(FC) -c $(FFLAGS2) $*.f90

The first change I can make to this using other assignments is:

   FFLAGS := -O 3,aggress -e I -e m
  FFLAGS2 := $(FFLAGS) -O inlinefrom=lmd_wscale.f90

The := assignment means to evaluate the right hand side immediately. In this case, there is no reason not to, as long as the second assigment follows the first one (since it's using the value of $(FFLAGS). For the plain equals, make doesn't evaluate the right-hand side until its second pass through the Makefile. However, gmake supports an assignment that avoids the need for FFLAGS2 entirely:

 lmd_skpp.o: FFLAGS += -O inlinefrom=lmd_wscale.f90

What this means is that for the target lmd_skpp.o only, append the inlining directive to FFLAGS. I think this is pretty cool!

One last kind of assignment is to set the value only if there is no value from somewhere else (the environment, for instance):

       FC ?= mpxlf90_r

If we use := or =, we would override the value from the environment.

System-dependent Information

One reasonably portable make feature is the include directive. It can be used to clean up the Makefile by putting bits in an include file. The syntax is simply:

 include file

and we are using it to include the list of sources to compile and the list of dependencies. This is how we are now keeping the project information neat. We can also include a file with the system/compiler information in it, assuming we have some way of deciding which file to include. Above is the example from the book, using uname to generate a file name. I decided to modify it slightly for our needs.

I have a make variable called FORT, which is set by the user of the Makefile. This value is combined with the result of uname -s to provide a machine and compiler combination. For instance, ftn on Linux is the Cray cross-compiler. This would link to a different copy of the NetCDF library and use different compiler flags than the Intel compiler on Linux.

 # The user sets Fortran Compiler:

    FORT := ftn

 MACHINE := $(shell uname -s)
 MACHINE += $(FORTRAN)
 MACHINE := $(shell echo $(MACHINE) | sed 's/[\/ ]/-/g')

 include Compilers/$(MACHINE).mk

Now, instead of having 30 Makefiles, we have a collection of include files in the Compilers directory and we pick one at compile time. In this example, we will pick Linux-ftn.mk, containing the Cray cross-compiler information. The value Linux comes from the uname command, the ftn comes from the user, the two are concatenated, then the space is converted to a dash by the sed command. The sed command will also turn the slash in UNICOS/mp into a dash; the native Cray include file is UNICOS-mp-ftn.mk.

The other tricky system is CYGWIN, which puts a version number in the uname output, such as CYGWIN_NT-5.1. GNU make has quite a few built-in functions plus allows the user to define their own functions. One of the built-in functions allows us to do string substitution:

 MACHINE := $(patsubst CYGWIN_%,CYGWIN,$(MACHINE))

In make, the % symbol is a sort of wild card, much like * in the shell. Here, we match CYGWIN followed by an underscore and anything else, replacing the whole with simply CYGWIN. Another example of a built-in function is the substitution we do in:

 objects = $(subst .F,.o,$(sources))

where we build our list of objects from the list of sources. There are quite a few other functions - see the book or the gnu make manual for a complete list.

Conditionals

One reason we had so many Makefiles was having a separate one for each of the serial/MPI/OpenMP versions on each system (if supported). For instance, the name of the IBM compiler changes when using MPI; the options change for OpenMP. The compiler options also change when using 64-bit addressing or for debugging, which we were manually setting. A better way to do this is to have the user select 64-bit or not, MPI or not, etc, then do the right thing later.

GNU make supports two kinds of if test, ifdef and ifeq (plus the negative versions ifndef, ifneq). The example from the book is:

 ifdef COMSPEC
 #  We are running Windows
 else
 #  We are not on Windows
 endif

An example from the IBM include file is:

       FC := xlf95_r
   FFLAGS := -qsuffix=f=f90 -qmaxmem=-1 -qarch=pwr4 -qtune=pwr4
  ARFLAGS := -r -v

 ifdef USE_LARGE
   FFLAGS += -q64
  ARFLAGS += -X 64
  LDFLAGS += -bmaxdata:0x200000000

  NETCDF_INCDIR := /usr/local/pkg/netcdf/netcdf-3.5.0_64/include
  NETCDF_LIBDIR := /usr/local/pkg/netcdf/netcdf-3.5.0_64/lib
 else
  LDFLAGS += -bmaxdata:0x70000000

  NETCDF_INCDIR := /usr/local/include
  NETCDF_LIBDIR := /usr/local/lib
 endif

 ifdef USE_DEBUG
   FFLAGS += -g -qfullpath
 else
   FFLAGS += -O3 -qstrict
 endif

We also test for MPI and OpenMP and change things accordingly. To test for equality, an example is:

 ifeq ($(USE_MPI),on)
 #  Do MPI things
 endif

or

 ifeq "$(USE_MPI)" "on"
 #  Do MPI things
 endif

The user has to set values for the USE_MPI, USE_OPENMP, USE_DEBUG, and USE_LARGE switches in the Makefile before the compiler-dependent piece is included:

    USE_MPI ?= on
 USE_OPENMP ?=
  USE_DEBUG ?=
  USE_LARGE ?= on

Be sure to use the immediate assign with the := or ?= when setting these flags. The conditional features supported by GNU make are very handy and have replaced the need for so many Makefiles in our build system. We have further tidied it up by putting all the include files in their own subdirectory.

Multiple Directories for Sources

There's more than one way to do it, but ROMS is using the method of putting all the temporary .f90 and .o files in the directory specified in the SCRATCH_DIR definition. The top directory has the master makefile, which includes a Makefile bit from each subdirectory, called Module.mk. The top makefile starts with an empty list of sources. In each subdirectory, we find local sources by simply listing all the .F files and appending this to the master list.

In makefile:

 sources :=
 include somedir/Module.mk

In somedir/Module.mk:

 local_src := $(wildcard $(subdirectory)/*.F)
 sources += $(local_src)

Here, we are using the wildcard function to search the subdirectory for it's local sources. Each subdirectory is resetting the local_src variable, but that's OK because we're saving the values in the global sources variable. The other sneaky thing here is the user-defined subdirectory function, from the Gnu make book:

 subdirectory = $(patsubst %/Module.mk,%,$(word $(words $(MAKEFILE_LIST)),$(MAKEFILE_LIST)))

This does the right thing to figure out which subdirectory we are in from make's internal list of the Makefiles it is parsing. It depends on all the subdirectory include files being called Module.mk.

Library Details

The directory structure we are using has the top directory, an Include directory, several directories which contain sources for libraries, and the directory for the main programs (Drivers). There is also a directory for the compiler-specific Makefile components (Compilers).

Here is a complete example of a library Makefile component:

 local_lib := libNLM.a
 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 libraries += $(local_lib)
 sources += $(local_src)

 $(local_lib): $(local_objs)
 $(AR) $(ARFLAGS) $@ $^

The only thing that changes from one to the next is the name of the library to build. I'm actually keeping track of the sources with and without the subdirectory part of their name. The objects will go into the top directory, so they shouldn't have the directory in their list. I only need the path_srcs for creating the dependency information; make itself knows to look in the directories because of a vpath command. We are also updating a libraries variable, adding the local library to the global list.

Main Program

The main program is in a directory called Drivers and its Module.mk is similar to the library one:

 local_src := $(wildcard $(subdirectory)/*.F)
 path_srcs += $(local_src)

 local_src := $(patsubst $(subdirectory)/%.F,%.F,$(local_src))
 local_objs := $(subst .F,.o,$(local_src))

 sources += $(local_src)

 $(BIN): $(libraries) $(local_objs)
 $(LD) $(FFLAGS) $(LDFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS)

Instead of a rule for building a library, we have a rule for building a binary. In this case, the name of the binary depends on if it's parallel or not and is defined elsewhere. The binary depends on the libraries getting compiled first, as well as the local sources. During the link, the $(libraries) are compiled from the sources in the other directories, while $(LIBS) are exteral libraries such as NetCDF and mpich.

Top Level Makefile

Now we get to the glue that holds it all together. We first initialize some of the global lists:

 #  Initialize some things.

 clean_list := core *.o *.mod *.f90 lib*.a
 sources :=
 path_srcs :=
 libraries :=

 objects = $(subst .F,.o,$(sources)

Next is the subdirectory function we already presented, followed by the user-defined switches and the compiler-dependent includes. Then we have the pattern rules, also shown above. Finally we get to the meat of the includes:

.PHONY: all

 all: $(BIN)

 modules := Adjoint Ice Modules Nonlinear Representer Tangent Utility Drivers

 includes := Include Adjoint Nonlinear Tangent Drivers

 vpath %.F $(modules)
 vpath %.h $(includes)

 include $(addsuffix /Module.mk,$(modules))

 CPPFLAGS += $(patsubst %,-I%,$(includes))

 .PHONY: clean

 clean:
  <TAG>$(RM) $(clean_list) $(BIN)

all is the first target that gets seen by make, making it the default target. In this case, we know there is only the one binary, whose name we know - the book shows how to do more than one binary. The modules are the list of subdirectories containing a Module.mk we need to include. clean is the target that removes all the cruft we don't want to keep. Both all and clean are phony targets in that no files of those names get generated - make has the .PHONY designation for such targets.