GUB the Grand Unified Builder

GUB BASICS

ROADMAP

directory contents and use
. (GNUmakefile, *.make) make drivers (try: make help)
bin/ programs (gub, gpkg, gib, ...; try: bin/gub --help)
gub/ python library
gub/specs/ build specifications (gub/specs/hello.py,bison.py,denemo.py,lilypond.py...)
patches/ patch files used in build specifications
target/ build trees
log/ (gub.log, ...) (build) log files
uploads/ installers
target/<platform>/ working tree for <platform>
target/linux-x86/src/ linux-x86 unpacked sources
target/linux-64/build/ linux-64 build output
target/darwin-x86/log/lilypond.log darwin-x86 most recent lilypond build log
target/mingw/packages/ .gup packages for mingw
target/darwin-ppc/log/status/ status of partial darwin-ppc builds
target/freebsd-x86/root/ freebsd-x86 build and install root
target/tools/root/usr/bin/ native tools used for build
sourcefiles/ non-patch files used in build specifications
nsis/ nsis windows installer scripts

BASIC WORKINGS

Checksums and rebuilds

GUB seeks to minimise unnecessary rebuilds while being strict about important changes. Before starting the actual building of a package, the build recipe is executed as a dry-run and a checksum is calculated from these build commands. This checksum includes the environment variables DISPLAY, FTP_PROXY, GUB_TOOLS_PREFIX, HOME, HTTP_PROXY, LIBRESTRICT, PATH [no other environment variables are allowed through].

The packages' build checksum is matched with the previous build and if it changes, the package is rebuilt. Otherwise, only the binary package is reinstalled, if necessary. This means that the .py build specs can be changed and refactored at will and as long as no actual build commands change, nothing will be rebuilt.

This makes that forcing rebuilds can can be tricky: gub refuses to do so if it deems rebuilding unnecessary.

SHARING DIRECTORIES

GUB uses an ABI environment variable to work around some build bugs in some packages. This variable is not checked by the environment-changed function, so problems will probably occur if you share your GUB build directory between a 32-bit and 64-bit OS. If you change your operating system, delete the target/ directory of GUB before trying to build it again.

LIBRESTRICT

A common problem with [making reproducible cross] builds is using files or features that are not controlled by the [GUB] build process, but just happen to be on the build host and happen to differ between most build hosts.

GUB uses the LD_PRELOAD feature with the gub/specs/librestrict.py package and code from ./librestrict/restrict.c to disallow reading from build host files. This makes for much better reprocucibility, especially between different build hosts.

The downside to this LIBRESTRICT feature is that quite a few packages are not prepared for cross building. LIBRESTRICT will catch these errors and they have to be fixed in the package's spec file.

This is what such a LIBRESTRICT assertion could like 
librestrict:error:/home/janneke/vc/gub/target/mingw/root/usr/cross/libexec/gcc/i686-mingw32/4.1.1/cc1plus: tried to open () file /home/janneke/vc/lilypond/lily/main.cc
librestrict:allowed:
  /home/janneke/vc/gub/target/mingw
  /tmp
  /dev/null
  /dev/urandom
  /proc/self
cc1plus: internal compiler error: Aborted
Please submit a full bug report,
with preprocessed source if appropriate.
See  for instructions.
[1]10:42:08 janneke@peder:~/vc/gub
$
you often find these in config.log files when package-dependant code blondly looks in /usr/include or /usr/lib.

stat-restriction

LIBRESTRICT has an experimental strict mode enabled by setting the environment variable 
LIBRESTRICT=open:stat
This will even disallow any STATting outside the target's build tree. It would be nice to get this to work and some effort has been done. However, this is potentially a lot of work as not only upstream packages but also build system authors [autoconf, automake, libtool etc.] look at the build host root file system while cross building.

Simplifying package builds or avoiding code duplication

Another idea of GUB is to identify, fix and generalise build quircks and move them from a package's spec file into one of the generic build classes. This increases maintainability by avoiding code duplication, makes spec files smaller, simpler and also increases the chances of a new package's spec file to only need a source url to specify the build.

BASIC USAGE

Build package hello for platform linux-x86 
bin/gub linux-x86::hello
Examine why package hello wants to rebuild 
less target/linux-x86/log/build.log
    search from end back to `mismatch:' [type: FC-c?mismatch:RET]
Force a rebuild of package hello for platform linux-x86 
rm -rf target/linux-x86/*/hello-*
Force a rebuild of package hello for all platforms 
rm -rf target/*/*/hello-*
Prepare to rebuild entire linux-x86 platform 
rm -rf target/linux-x86
Prepare to rebuild everything, all platforms 
rm -rf target

SPEC FILES

Plain url builds

<--no spec-->

Well behaved packages can be built without spec file, straight from an url. For example, to build bison-2.2 as a build tool, do 
bin/gub tools::http://ftp.gnu.org/pub/gnu/bison/bison-2.2.tar.gz
This produces 
target/tools/packages/bison-2.2.tools.gup
target/tools/packages/bison.tools.hdr
target/tools/packages/bison.checksum
Check files installed under target/tools/root/ by doing 
bin/gpkg -p tools files bison
To build the very latest findutils for your architecture, do 
bin/gub git://git.savannah.gnu.org/findutils.git

Simple spec

gub/specs/hello.py 
from gub import target

class Hello (target.AutoBuild):
    source = 'http://lilypond.org/download/gub-sources/hello-1.0.tar.gz'
build it for mingw by doing 
bin/gub mingw::hello

__TOOLS package with dependencies

gub/specs/autoconf.py 
from gub import tools

class Autoconf__tools (tools.AutoBuild):
    source = 'http://ftp.gnu.org/pub/gnu/autoconf/autoconf-2.63.tar.gz'
    parallel_build_broken = True
    dependencies = [
	'm4',
	'perl',
	]
build it by doing 
bin/gub tools::autoconf

configure flags

gub/specs/icoutils.py 
from gub import tools

class Icoutils__tools (tools.AutoBuild):
    dependencies = ['libpng-devel']
    configure_flags = (tools.AutoBuild.configure_flags
                       + ' --with-libintl-prefix=%(system_prefix)s'
                       + ' --disable-nls')

__<platform>__<arch>-specific instructions

For platform specific instructions, append "__<platform>" to the class name. For platform+architecture-specific instructions, "__<platform>__<arch>" to the class name.

gub/specs/pixman.py 
from gub import target

class Glib (target.AutoBuild):
    ...

class Glib__freebsd (Glib):
    dependencies = Glib.dependencies + ['libiconv-devel']
    configure_variables = Glib.configure_variables + ' CFLAGS=-pthread'

class Glib__freebsd__x86 (Glib__freebsd):
    # Must include -pthread in lib flags, because our most beloved
    # libtool (2.2.6a) thinks it knows best and blondly strips -pthread
    # if it thinks it's a compile flag.
    # FIXME: should add fixup to update_libtool ()
    compile_flags = ' G_THREAD_LIBS=-pthread G_THREAD_LIBS_FOR_GTHREAD=-pthread '

String expansion

In all build commands, standard Python string expansion 
%(string-name)s
can be used. These strings are taken from class Settings in gub/settings.py and from the package's build class and its parent classes.

Adding string expansion variables to a spec build class can be done in several ways

Pre-defined strings

Typical variables defined by settings are 
build_architecture=x86_64-linux
build_os=linux
build_platform=linux-64
platform=target
target_architecture=x86_64-linux
target_bits=64
target_cpu=x86_64
target_os=linux
target_platform=target
Typical variables defined by the build class 
compile_flags=
builddir=.../target/build/...
config_cache_overrides=
configure_binary= .../configure
configure_command= sh .../configure ...
configure_flags= --prefix=... --enable-shared --enable-static
configure_variables= CFLAGS=-I.../target/root/usr/include
install_command= 
install_flags= 
install_prefix=
install_root=
make_flags=
srcdir=.../target/src/...

More examples

See gub/specs/*.py for some more examples.