12.4.1 The function of make doc

The following is a set of notes on how make doc functions.

Preliminary question to be answered some time: where do all the GNUmakefiles come from. They’re in the build directory, but this is not part of source. Must be the configure script. And it looks like this comes from autogen.sh. Must at some point kill the whole git directory, repull and see what is created when.

Anyway, here’s how make doc progresses:

This is the build dependency tree from ‘stepmake/stepmake/generic-targets.make’:

doc: doc-stage-1
  doc-stage-1:
        $(MAKE) -C $(depth)/scripts/build out=
        $(MAKE) out=www WWW-1
          WWW-1: local-WWW-1
        $(LOOP)
          $(MAKE) out=www WWW-2
          WWW-2: local-WWW-2
            $(LOOP)
          $(MAKE) out=www WWW-post
MAKE = make --no-builtin-rules
-C = Change to directory before make

doc-stage-1 does lots of opening and looking in files, but no processing.

Variable LOOP =

+ make PACKAGE=LILYPOND package=lilypond -C python
&& make PACKAGE=LILYPOND package=lilypond -C scripts
&&  make PACKAGE=LILYPOND package=lilypond -C flower
&&  make PACKAGE=LILYPOND package=lilypond -C lily
&&  make PACKAGE=LILYPOND package=lilypond -C mf
&&  make PACKAGE=LILYPOND package=lilypond -C ly
&&  make PACKAGE=LILYPOND package=lilypond -C tex
&&  make PACKAGE=LILYPOND package=lilypond -C ps
&&  make PACKAGE=LILYPOND package=lilypond -C scm
&&  make PACKAGE=LILYPOND package=lilypond -C po
&&  make PACKAGE=LILYPOND package=lilypond -C make
&&  make PACKAGE=LILYPOND package=lilypond -C elisp
&&  make PACKAGE=LILYPOND package=lilypond -C vim
&&  make PACKAGE=LILYPOND package=lilypond -C input
&&  make PACKAGE=LILYPOND package=lilypond -C stepmake
&&  make PACKAGE=LILYPOND package=lilypond -C Documentation
&& true

From git grep:

stepmake/stepmake/generic-vars.make has this:

 
LOOP=+$(foreach i, $(SUBDIRS), $(MAKE) PACKAGE=$(PACKAGE) package=$(package) -C $(i) $@ &&) true

$@ is the name of the target - WWW-1 in this case.

In GNUmakefile.in we find:

SUBDIRS = python scripts \
        flower lily \
        mf ly \
        tex ps scm \
        po make \
        elisp vim \
        input \
        stepmake $(documentation-dir)

So that’s how we get the main make loop...

That loop expands like this:

make PACKAGE=LILYPOND package=lilypond -C python WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scripts WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C flower WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C lily WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C mf WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ly WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C tex WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C ps WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C scm WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C po WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C make WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C elisp WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C vim WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C input WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C stepmake WWW-1 &&
make PACKAGE=LILYPOND package=lilypond -C Documentation WWW-1 &&
true

The directories up to and including vim produce no effect with make in non-debug mode, although debug does show lots of action.

git/build/input/GNUmakefile’ is:

depth=../
include $(depth)/config$(if $(conf),-$(conf),).make
include $(configure-srcdir)/./input/GNUmakefile
MODULE_INCLUDES += $(src-dir)/$(outbase)

The first include is:

..//config.make

(note the // which is strictly wrong)

which has lots of variables to set, but no action occurs.

The second is:

lilypond-git/./input/GNUmakefile

which similarly doesn’t create any actual action.

An error message at the end of build/input/GNUmakefile stops make processing before it moves on to regression - so where does that come from?

And the answer is - make processes all directories in the directory it’s entered (with some exceptions like out and out-www) and so it changes to /regression.

It then seems to consider whether it needs to make/remake loads of makefiles. Don’t understand this yet. Possibly these are all the makefiles it’s processing, and it always checks they’re up to date before processing other files?

Could be correct - some of this output is:

Must remake target `../../make/ly-inclusions.make'.
Failed to remake target file `../../make/ly-inclusions.make'.

Having decided that, it then leaves the directory and re-executes:

make PACKAGE=LILYPOND package=lilypond -C regression WWW-1

The top of this make is:

This program built for i486-pc-linux-gnu
Reading makefiles...
Reading makefile `GNUmakefile'...
Reading makefile `../..//config.make' (search path) (no ~ expansion)...

which looks like it’s re-reading all its known makefiles to check they’re up to date.

(From the make manual:

To this end, after reading in all makefiles, make will consider each as a goal target and attempt to update it. If a makefile has a rule which says how to update it (found either in that very makefile or in another one) or if an implicit rule applies to it (see Chapter 10 [Using Implicit Rules], page 103), it will be updated if necessary. After all makefiles have been checked, if any have actually been changed, make starts with a clean slate and reads all the makefiles over again. (It will also attempt to update each of them over again, but normally this will not change them again, since they are already up to date.)

So my assumption seems correct)

There appear to be about 74 of them. After all the makefile checking, we get this:

 
Updating goal targets....
Considering target file `WWW-1'.
File `WWW-1' does not exist.
Considering target file `local-WWW-1'.
File `local-WWW-1' does not exist.
Considering target file `out-www/collated-files.texi'.
File `out-www/collated-files.texi' does not exist.
Looking for an implicit rule for `out-www/collated-files.texi'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files.texi'.
Trying implicit prerequisite `collated-files.texi.in'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `collated-files.tely'.
Trying pattern rule with stem `collated-files'.
Trying implicit prerequisite `out-www/collated-files.tely'.
Trying rule prerequisite `out-www/version.itexi'.
Found prerequisite `out-www/version.itexi' as VPATH `/home/phil/lilypond-git/input/regression/out-www/version.itexi'

grep finds this if searching for local-WWW-1:

make/lysdoc-targets.make:
  local-WWW-1: $(outdir)/collated-files.texi $(outdir)/collated-files.pdf

which means that local-WWW-1 depends on coll*.texi and coll*.pdf and so these will need to be checked to see if they’re up to date. So make needs to find rules for both of those and (as it says) it certainly needs to make coll*.texi, since it doesn’t exist.

In ly-rules.make we have:

.SUFFIXES: .doc .tely .texi .ly

which I’ll work out at some point, and also this rule:

 
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<

Note that the recipe is a very long line - it could probably benefit from splitting. The same makefile also has:

 
$(outdir)/%.texi: $(outdir)/%.tely $(outdir)/version.itexi $(DOCUMENTATION_LOCALE_TARGET) $(INIT_LY_SOURCES) $(SCHEME_SOURCES)
  LILYPOND_VERSION=$(TOPLEVEL_VERSION) $(PYTHON) $(LILYPOND_BOOK) $(LILYPOND_BOOK_INCLUDES) --process='$(LILYPOND_BOOK_PROCESS) $(LILYPOND_BOOK_INCLUDES) $(LILYPOND_BOOK_LILYPOND_FLAGS)' --output=$(outdir) --format=$(LILYPOND_BOOK_FORMAT) $(LILYPOND_BOOK_FLAGS) $<

which seems to be an almost exact duplicate. Whatever, the first one is executed first. Have not checked if the second executes.

The first recipe translates as this:

LILYPOND_VERSION=2.15.0 /usr/bin/python   --process=' ' \
  --output=./out-www --format= --lily-output-dir \
    /home/phil/lilypond-git/build/out/lybook-db

if we stop the build with an $(error), but I think this is because we need to allow it to process the dependencies first. It looks like foo.texi is shown as being dependent on foo.tely, plus a load of other files.

DOCUMENTATION_LOCALE_TARGET is blank
INIT_LY_SOURCES = /home/phil/lilypond-git/scm/auto-beam.scm \
  /home/phil/lilypond-git/scm/autochange.scm

plus 10s (100s?) of other .scm files.

SCHEME_SOURCES = /home/phil/lilypond-git/ly/Welcome-to-LilyPond-MacOS.ly \
  /home/phil/lilypond-git/ly/Welcome_to_LilyPond.ly

ditto .ly files. This does seem a teency bit wrong - it looks like the .ly and .scm files have been interchanged. ly-vars.make has these 2 lines:

INIT_LY_SOURCES = $(wildcard $(top-src-dir)/scm/*.scm)
SCHEME_SOURCES = $(wildcard $(top-src-dir)/ly/*.ly)

Looks like a bug.....

So it now works its way through all these files, checking if they need to be remade. This is 100s of lines of the debug listing, although none in the normal list. Clearly none has to be made since they’re source files. It concludes:

Must remake target `out-www/collated-files.tely'

lysdoc-rules.make’ has this:

 
$(outdir)/collated-files.tely: $(COLLATED_FILES)
        $(LYS_TO_TELY) --name=$(outdir)/collated-files.tely --title="$(TITLE)" --author="$(AUTHOR)" $^

lysdoc-vars.make’ has:

COLLATED_FILES = $(sort $(TEXINFO_SOURCES) $(LY_FILES) $(OUT_LY_FILES) )

We find that:

TEXINFO_SOURCES = AAA-intro-regression.tely
OUT_LY_FILES is empty

so LY_FILES has the big long list of all the .ly files in the regression directory.

This kicks off

/home/phil/lilypond-git/build/scripts/build/out/lys-to-tely

with a list of all the files in the regression test directory. This should (I believe) create the file collated-files.tely.

So the next rule in make is for ‘version.itexi’, and make duly checks this. There’s a rule in ‘doc-i18n-root-rules.make’ that this depends on ‘git/VERSION’:

 
$(outdir)/version.%: $(top-src-dir)/VERSION
        $(PYTHON) $(top-src-dir)/scripts/build/create-version-itexi.py > $ 

This causes create-version-itexi.py to run and create version.itexi.

Once that’s done, all the other *.scm and *.ly files are checked and since they have no rules associated, they aren’t remade (just as well for source files, really). Since version.itexi was remade make concludes that collated-files.texi must be remade. To do this, it runs lilypond-book.py on collated-files.tely, as below:

LILYPOND_VERSION=2.15.0
  /usr/bin/python
  /home/phil/lilypond-git/scripts/lilypond-book.py
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    --process='/home/phil/lilypond-git/build/out/bin/lilypond
    -I /home/phil/lilypond-git/input/regression/
    -I ./out-www
    -I /home/phil/lilypond-git/input
    -I /home/phil/lilypond-git/Documentation
    -I /home/phil/lilypond-git/Documentation/snippets
    -I /home/phil/lilypond-git/input/regression/
    -I /home/phil/lilypond-git/Documentation/included/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/build/mf/out/
    -I /home/phil/lilypond-git/Documentation/pictures
    -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
    -dbackend=eps
    --formats=ps,png,pdf
    -dinclude-eps-fonts
    -dgs-load-fonts
    --header=doctitle
    --header=doctitlecs
    --header=doctitlede
    --header=doctitlees
    --header=doctitlefr
    --header=doctitlehu
    --header=doctitleit
    --header=doctitleja
    --header=doctitlenl
    --header=doctitlezh
    --header=texidoc
    --header=texidoccs
    --header=texidocde
    --header=texidoces
    --header=texidocfr
    --header=texidochu
    --header=texidocit
    --header=texidocja
    --header=texidocnl
    --header=texidoczh
    -dcheck-internal-types
    -ddump-signatures
    -danti-alias-factor=2'
    --output=./out-www
    --format=texi-html
    --verbose
    --lily-output-dir /home/phil/lilypond-git/build/out/lybook-db
    out-www/collated-files.tely

So - lilypond-book runs on:

input/regression/out-www/collated-files.tely

Note the –verbose flag - this is from the make variable LILYPOND_BOOK_VERBOSE which is added to the make variable LILYPOND_BOOK_FLAGS.

Now found the invocation to write some of the image files. It’s like this:

/home/phil/lilypond-git/build/out/bin/lilypond
  -I /home/phil/lilypond-git/input/regression/
  -I ./out-www -I /home/phil/lilypond-git/input
  -I /home/phil/lilypond-git/Documentation
  -I /home/phil/lilypond-git/Documentation/snippets
  -I /home/phil/lilypond-git/input/regression/
  -I /home/phil/lilypond-git/Documentation/included/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/build/mf/out/
  -I /home/phil/lilypond-git/Documentation/pictures
  -I /home/phil/lilypond-git/build/Documentation/pictures/./out-www
  -dbackend=eps
  --formats=ps,png,pdf
  -dinclude-eps-fonts
  -dgs-load-fonts
  --header=doctitle
  --header=doctitlecs
  --header=doctitlede
  --header=doctitlees
  --header=doctitlefr
  --header=doctitlehu
  --header=doctitleit
  --header=doctitleja
  --header=doctitlenl
  --header=doctitlezh
  --header=texidoc
  --header=texidoccs
  --header=texidocde
  --header=texidoces
  --header=texidocfr
  --header=texidochu
  --header=texidocit
  --header=texidocja
  --header=texidocnl
  --header=texidoczh
  -dcheck-internal-types
  -ddump-signatures
  -danti-alias-factor=2
  -I  "/home/phil/lilypond-git/build/out/lybook-db"
  -I  "/home/phil/lilypond-git/build/input/regression"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/build/input/regression/out-www"
  -I  "/home/phil/lilypond-git/input"
  -I  "/home/phil/lilypond-git/Documentation"
  -I  "/home/phil/lilypond-git/Documentation/snippets"
  -I  "/home/phil/lilypond-git/input/regression"
  -I  "/home/phil/lilypond-git/Documentation/included"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/build/mf/out"
  -I  "/home/phil/lilypond-git/Documentation/pictures"
  -I  "/home/phil/lilypond-git/build/Documentation/pictures/out-www"
  --formats=eps
  --verbose
  -deps-box-padding=3.000000
  -dread-file-list
  -dno-strip-output-dir
  "/home/phil/lilypond-git/build/out/lybook-db/snippet-names--415419468.ly"'

Note the –verbose. This causes 100s of lines of Lily debug output. But at present I can’t work out where the flag comes from. Later.


LilyPond — Contributor’s Guide v2.19.10 (development-branch).