Today I released a new beta of Template::Benchmark (v0.99_07) and one of the changes, along with the addition of 4 new template engine plugins (Tenjin, Template::Tiny, Text::Template::Simple and NTS::Template), is that it splits the author/release tests away from the install tests.

This was a royal PITA, so I thought I'd cover the how and why of what was done.

First up, I'd like to make it clear that nothing in this blog entry is intended as a criticism of the fine work done by all the authors who have contributed to making the various perl toolchains the wonderfuly useful pieces of software that they are.

Without their work, being an author of a perl module would be a vastly more unpleasant task, and the frustrations in this article should be taken as gripes that not everything is as easy as they've succeeded in making most tasks.

Secondly, I've only uploaded the distribution to CPAN today, I'll probably find some horribly broken thing when the smoke-tests start coming in, but I'll tempt fate and tell the world about my attempts as if they succeeded. Ho ho.

With that out of the way, back to the main plot...

As a module author, like many on CPAN I'm sure, I use Module::Build to handle the installation of my distribution, and Module::Starter to initially build the skeletal framework of my distribution.

This dumps some automatic tests in your distribution's test directory, including the now infamous pod.t and pod-coverage.t.

It's been covered elsewhere why this can be a bad thing, so I won't repeat the arguments for or against. I'll just state that I come down on the side that the tests are useful for improving the quality of perl distributions, and that bundling them in with the install tests was a neccessary evil that we should have outgrown by now: by having some standard framework for release-testing to complement our highly successful framework for install-testing.

Unfortunately, it ain't so.

So, until now, I'd carried on doing things the old way, hoping and waiting for the various toolchain authors to get their heads together and come up with some nice simple and consistent Right Way Of Doing Things for us poor module authors. (Aren't we precious?)

I expect I'd have lazily carried on like this until some consensus did (or didn't) coalesce, but then I came to the 0.99_07 release of Template::Benchmark, a release where the slowly-expanding test-suite hit a snag.

Now the problem with benchmarking is that it needs to run for some length of time, and I had the minimum length of time that each benchmark could run set to 1 second - not unreasonable a figure. Or at least, not unreasonable until you have 19 plugins with 6 benchmark types and 24 template features to individually test.

Bear in mind that this is just an isolated test of each, so the "simple" cartesian product of those options, not the horrors that a test of all the combinations would be.

Now rigorous testing is good, and having tests that isolate their cause are really invaluable, but I think I'd be pushing the patience of all the kind smoke-testers if I made them run 19 * 6 * 24 = 2736 seconds = 45 minutes of tests on their machines each time I drop a point-release on CPAN.

Of course, I could run it with a 1/10th of a second benchmark, but even so that's still nearly 5 minutes, and as it happens each benchmark has around a second of overhead to set up anyway.

Either way, I had a test that was extremely useful for me to be able to run, to be sure I hadn't broken anything, but that I really didn't want to inflict on anyone else unless they really wanted to run it.

So I decided to put it in the extra tests directory as a slow test that would only be run if you asked for it.

"Easy enough", I thought, I'll put it in xt/slow/something.t and while I'm at it I might as well get around to putting my release tests in xt/release/*.t.

Oh dear, oh dear. It was a typical "how hard can it be?" moment. Let's just say it wasn't pretty.

When the smoke settled, I'd given up entirely on getting Module::Build to do it in anything approaching a nice manner: it appears to give you support for groups of tests, but then forces you to do it by grouping them by unique file extension.

I'm sure there's good reasons for that, but grouping stuff by file extension, or by crafted filenames just sends shivers down my spine. We have devices for logically grouping files together already, they're called directories, you may have heard of them.

But, I rather liked the API onto the test groups, so I started digging into the guts of Module::Build to see if I could finesse matters so it used different directories instead of extensions.

It seems everything about it is nicely customizable... except the reliance on file extensions. Damn.

Meanwhile, I had found the Elliot Loves Perl blog entry on author tests, but unfortunately his solution breaks use of the --test_files option.

Given I heavily rely on --test_files while writing new tests, this was a show-stopper for me.

It did make me face up to the fact that I probably wasn't going to find a decent pre-made solution to my problem. The closest I'd come was finding the Module::Install plans to run things in xt as release tests, including when running automated tests (but running everything under xt put me back in the position of running the slow tests on smoke-boxes).

So, and sorry for the long back-story, I hacked together my own way of doing things.

First of all I used Test::XT to produce 'nice' versions of the standard release tests in xt/release/*.t. I'm not happy with the amount of boilerplate in those tests, but it seems to be unavoidable, and it has the benefit of working.

This means that the tests degrade gracefully if the modules they need to perform the test aren't available: when release testing they'll cause a fail, when automated testing they'll skip if the modules aren't available, and under installation testing they'll never be run (or if they are somehow run they'll skip).

Then I put my slow test under xt/slow/*.t, this never gets run unless you explicitly ask for it via ./Build testslow or ./Build testall, it also attempts to use Test::Slow in case they set their preferences that way.

OK, test structure aside, then I had to change Build.PL to be able to run these, and to run them at the right times:

#!/usr/bin/perl -w

use strict;
use warnings;
use Module::Build;

#  This subclassing adds the following build commands:
#    testrelease - runs the release tests in xt/release
#    testslow    - runs the painfully slow tests in xt/slow
#    testall     - runs the tests both in t and xt (recursively)
#  It also customises the "test" command so that it also runs the
#  release tests if either the RELEASE_TESTING or AUTOMATED_TESTING
#  environment variables are set, in line with what appears to be
#  the current consensus "best-practice" as of Feb 2010.
#
#  All xt tests are assumed to do their own requirements checking
#  and to gracefully skip their tests if the requirements are not
#  available: under no circumstances should they add requirements
#  to the end-user build or install processes.

my $class = Module::Build->subclass(
    class => 'Module::Build::SGRAHAM',
    code  => <<'END_OF_CODE',
sub ACTION_test
{
    my ( $self ) = @_;

    if( $ENV{ RELEASE_TESTING } or $ENV{ AUTOMATED_TESTING } )
    {
        #  Checking $self->{ properties } breaks the black-box but
        #  won't clobber use of --test_files args.
        #  Can't call $self->test_files() to find if any were manually
        #  supplied, because that autoexpands the default setting.
        $self->test_files( 't', 'xt/release' )
            unless $self->{ properties }->{ test_files };
    }
    return $self->SUPER::ACTION_test();
}

sub ACTION_testrelease
{
    my ( $self ) = @_;

    $self->depends_on( 'build' );
    local $ENV{ RELEASE_TESTING } = 1;
    $self->test_files( qw( xt/release ) );
    $self->depends_on( 'test' );
}

sub ACTION_testslow
{
    my ( $self ) = @_;

    $self->depends_on( 'build' );
    $self->test_files( qw( xt/slow ) );
    $self->depends_on( 'test' );
}

sub ACTION_testall
{
    my ( $self ) = @_;

    $self->depends_on( 'build' );
    $self->test_files( qw( t xt ) );
    $self->recursive_test_files( 1 );
    $self->depends_on( 'test' );
}

sub ACTION_distdir
{
    my ( $self ) = @_;

    $self->depends_on( 'testrelease' );

    return( $self->SUPER::ACTION_distdir() );
}
END_OF_CODE
    );

my $builder = $class->new(
    module_name         => 'Template::Benchmark',
    license             => 'perl',
    dist_author         => q{Sam Graham <skip skipity skip>},
    dist_version_from   => 'lib/Template/Benchmark.pm',
    script_files        => 'script',
    configure_requires => {
        'Module::Build'    => 0.23,
    },
    build_requires => {
        'Config'           => 0,
        'Cwd'              => 0,
        'File::Spec'       => 0,
        'FindBin'          => 0,
        'Test::More'       => 0,
        'Test::Command'    => 0.08,
    },
    requires => {
        'perl'                      => '5.6.1',
        #  For the module.
        'Module::Pluggable'         => 0,
        'Benchmark'                 => 0,
        'POSIX'                     => 0,
        'File::Path'                => 0,
        'File::Spec'                => 0,
        'IO::File'                  => 0,
        #  For the script.
        'FindBin'                   => 0,
        'Getopt::Long'              => 0,
        'Pod::Usage'                => 0,
        'Text::Wrap'                => 0,
    },
    sign => 1,
    dynamic_config => 0,
);

$builder->create_build_script();

Fun huh?

The comments should be fairly self-explanatory, the key points being that the following work the way I want them to:

#  Install test.
./Build test

#  Force release-testing.
RELEASE_TESTING=1 ./Build test

#  Pretend I'm a smoker.
AUTOMATED_TESTING=1 ./Build test

#  Just run one test script.
./Build test --test_files=t/50-script-run.t

#  Run my release tests specifically.
./Build testrelease

#  Make _sure_ I've run my release tests, I'm forgetful.
./Build dist

#  Run the horribly slow tests.
./Build testslow

#  Run all the tests.
./Build testall

Currently ./Build testall doesn't force the release tests to actually do anything, so without RELEASE_TESTING=1, it will skip all the release tests, telling you that you don't need to run them during install-testing.

I should probably force them to run, but for now it's "good enough".

I'm fairly happy with how this works, I'm decidedly unhappy that I'm having to do it in what feels like an unsafe manner rather than just supplying something simple like:

    test_type_dirs => {
        release => 'xt/release',
        slow    => 'xt/slow',
        },
    extra_automated_testing_test_types => [ qw/release/ ],

...and have it Just Work.

I guess I'm just spoiled by how well the rest of the test framework is integrated into the toolchain, but I can dream can't I?