9.4 Regtest comparison

Before modified code is committed to master (via staging), a regression test comparison must be completed to ensure that the changes have not caused problems with previously working code. The comparison is made automatically upon compiling the regression test suite twice.

  1. Run make with current git master without any of your changes.
  2. Before making changes to the code, establish a baseline for the comparison by going to the ‘$LILYPOND_GIT/build/’ directory and running:
    make test-baseline
    
  3. Make your changes, or apply the patch(es) to consider.
  4. Compile the source with ‘make’ as usual.
  5. Check for unintentional changes to the regtests:
    make check
    

    After this has finished, a regression test comparison will be available (relative to the current ‘build/’ directory) at:

    out/test-results/index.html
    

    For each regression test that differs between the baseline and the changed code, a regression test entry will be displayed. Ideally, the only changes would be the changes that you were working on. If regressions are introduced, they must be fixed before committing the code.

    Note: The special regression test ‘test-output-distance.ly’ will always show up as a regression. This test changes each time it is run, and serves to verify that the regression tests have, in fact, run.

  6. If you are happy with the results, then stop now.

    If you want to continue programming, then make any additional code changes, and continue.

  7. Compile the source with ‘make’ as usual.
  8. To re-check files that differed between the initial ‘make test-baseline’ and your post-changes ‘make check’, run:
    make test-redo
    

    This updates the regression list at ‘out/test-results/index.html’. It does not redo ‘test-output-distance.ly’.

  9. When all regressions have been resolved, the output list will be empty.
  10. Once all regressions have been resolved, a final check should be completed by running:
    make test-clean
    make check
    

    This cleans the results of the previous ‘make check’, then does the automatic regression comparison again.

Advanced note: Once a test baseline has been established, there is no need to run it again unless git master changed. In other words, if you work with several branches and want to do regtests comparison for all of them, you can make test-baseline with git master, checkout some branch, make and make check it, then switch to another branch, make test-clean, make and make check it without doing make test-baseline again.


LilyPond — Contributor’s Guide v2.17.97 (development-branch).