9.5 Pixel-based regtest comparison

As an alternative to the make test method for regtest checking (which relies upon .signature files created by a LilyPond run and which describe the placing of grobs) there is a script which compares the output of two LilyPond versions pixel-by-pixel. To use this, start by checking out the version of LilyPond you want to use as a baseline, and run make. Then, do the following:

cd $LILYPOND_GIT/scripts/auxiliar/
./make-regtest-pngs.sh -j9 -o

The -j9 option tells the script to use 9 CPUs to create the images - change this to your own CPU count+1. -o means this is the "old" version. This will create images of all the regtests in

$LILYPOND_BUILD_DIR/out-png-check/old-regtest-results/

Now checkout the version you want to compare with the baseline. Run make again to recreate the LilyPond binary. Then, do the following:

cd $LILYPOND_GIT/scripts/auxiliar/
./make-regtest-pngs.sh -j9 -n

The -n option tells the script to make a "new" version of the images. They are created in

$LILYPOND_BUILD_DIR/out-png-check/new-regtest-results/

Once the new images have been created, the script compares the old images with the new ones pixel-by-pixel and prints a list of the different images to the terminal, together with a count of how many differences were found. The results of the checks are in

$LILYPOND_BUILD_DIR/out-png-check/regtest-diffs/

To check for differences, browse that directory with an image viewer. Differences are shown in red. Be aware that some images with complex fonts or spacing annotations always display a few minor differences. These can safely be ignored.