[prev in list] [next in list] [prev in thread] [next in thread] 

List:       perl-module-build
Subject:    Re: testing alpha M::B releases
From:       Eric Wilhelm <scratchcomputing () gmail ! com>
Date:       2008-07-10 22:21:37
Message-ID: 200807101521.37226.ewilhelm () cpan ! org
[Download RAW message or body]

# from David Golden
# on Thursday 10 July 2008 10:57:

>(That said, I'm not sure I'd suggest holding up a release waiting for
>me/others to get it all written in the next couple weeks.)

How long has it been since the last release?  ;-)

>> Especially handy would be some way to characterize the failures
>> which share a root cause.  Perhaps a nightly aggregation/report run
>> could be posted to the list?
>
>That's a hard problem right now.  CPAN Testers 2.0 (still in the
>design stage) might well support capturing TAP archives, allowing more
>detailed analysis of exactly which tests fail, but that still a long
>ways off.

We don't really need to capture the TAP archive to know whether 
Module::Build is broken.  The failure case we need to catch is that a 
dist built/installed fine with the stable M::B and failed with the 
alpha.  This is a quite different test scheme vs what's currently 
happenning with most smokers.

The characteristics of import are then the OS, perl version, version 
numbers of various related modules, CPAN(PLUS) config, and possibly 
some details about the filesystem layout or what-not.

Which distribution being built is not so important as long as it was 
known-good with the stable M::B (or known to fail due to a bug which we 
fixed -- we actually do that sometimes.)  So, e.g. a cross-section of 
multiple platforms/perls building File::Fu would be a basically useful 
check on non-XS functionality.  Some dists use a compatibilly 
Makefile.PL (with or without passthrough) and some write that by hand, 
etc.

Actually, someone could fabricate some fake distributions (TestMe::MB, 
TestMe::MBCompat, TestMe::XS, etc.)  Of course, real authors are going 
to create failure cases we didn't think of.

In short, we'll never be able to be sure that we didn't break something 
because there is a lot of behavior which comes from platforms and other 
external factors.  We can only try really hard to discover in alpha 
what we *did* break.  The current situation is that we won't be making 
that discovery until too late.

So, help finding and fixing these sorts of bugs would go a long way 
toward getting regular stable releases.

Thanks,
Eric
-- 
Turns out the optimal technique is to put it in reverse and gun it.
--Steven Squyres (on challenges in interplanetary robot navigation)
---------------------------------------------------
    http://scratchcomputing.com
---------------------------------------------------

[prev in list] [next in list] [prev in thread] [next in thread]