[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] MC/DC coverage explained wrong in the home page?
From:       Richard Hipp <drh () sqlite ! org>
Date:       2011-09-23 18:05:40
Message-ID: CALwJ=My-z6tiejBXdUhwJCdqmA6vB5HKUpcKg_CKvyiDieNHxg () mail ! gmail ! com
[Download RAW message or body]

On Fri, Sep 23, 2011 at 1:51 PM, Sami Liedes <sliedes@cc.hut.fi> wrote:

>
> > For a decision of the form:
> >
> >      if( A && B )...
> >
> > The test suite for SQLite tries at least three cases:
> >
> >    (1)  A false
> >    (2)  A true and B false
> >    (3)  A true and B true
> >
> > You seem to be saying that this is only MC/DC if we split case (1) into
> two
> > subcases:
> >
> >    (1a) A false and B false
> >    (1b) A false and B true
>
> Hmm, no, not quite, although you do have a point here that in
> shortcutting languages it's different. I would concede that in a
> shortcutting language you do not need to test for (1a) and (1b).
>
> I think it means that for the decision of the form if( A && B ), you
> additionally need to show that A and B both can independently affect
> the outcome;


For case (1), since B is uncomputable, we can deem it to be true.  Then
since case (3) has a different outcome from cases (1) and (2), we do show
that each term is independent of the other.  The fact that B is deemed true
in case (1) might raise eyebrows, but in a shortcutting language, that's the
best you can do, I think.



> that is,
>
> For A: Either (2) or (3) results in different branch being taken from
> (1) (which in this case would be true for (3), but the analysis is not
> always so straightforward)
>
> For B: The branches taken in (2) and (3) are different (which is the
> case here).
>
> So in case of if( A && B ), I think you are correct that these three
> test cases suffice for MC/DC. But you *have* to consider the outcome
> too for it to be MC/DC.
>
> Taking a look at the Wikipedia page[1] for MC/DC, I believe what you
> describe is plain Condition/Decision Coverage, not MC/DC:
>
> ------------------------------------------------------------
>  * Condition/Decision Coverage
>
>        Every point of entry and exit in the program has been invoked
>        at least once, every condition in a decision in the program
>        has taken all possible outcomes at least once, and every
>        decision in the program has taken all possible outcomes at
>        least once.
>
>  * Modified Condition/Decision Coverage
>
>        Every point of entry and exit in the program has been invoked
>        at least once, every condition in a decision in the program
>        has taken on all possible outcomes at least once, and each
>        condition has been shown to affect that decision outcome
>        independently. A condition is shown to affect a decision's
>        outcome independently by varying just that condition while
>        holding fixed all other possible conditions. The
>        condition/decision criterion does not guarantee the coverage
>        of all conditions in the module because in many test cases,
>        some conditions of a decision are masked by the other
>        conditions. Using the modified condition/decision criterion,
>        each condition must be shown to be able to act on the decision
>        outcome by itself, everything else being held fixed. The MC/DC
>        criterion is thus much stronger than the condition/decision
>        coverage.
> ------------------------------------------------------------
>
> > Your argument is that there might be redundancy; that if B is always
> false
> > when A is false, then it would be possible to simplify the decision to
> just:
> >
> >     if( B )...
> >
> > True enough.  But there are countless ways to refactor a boolean
> > expression.  I don't think that MC/DC has any requirement that the number
> of
> > conditions be minimized, does it?
>
> I think that's one of the central ideas of MC/DC. Well, I think it
> doesn't require *minimizing* per se, only that each condition
> independently is enough to flip the outcome. (I now think I was wrong
> to talk about subexpressions in my original post; The "conditions" are
> the atomic conditions, not subexpressions.) I certainly cannot imagine
> of another plausible reading of the fourth condition. Perhaps you have
> a different plausible interpretation?
>
> In fact nearly all definitions of MC/DC I can find contain this
> paragraph, which explains it quite well:
>
>   "A condition is shown to affect a decision's outcome independently
>    by varying just that decision while holding fixed all other
>    possible conditions."
>
> > One of the key benefits we derive from testing SQLite to 100% branch
> > coverage is that we end up testing the object code, not the source code.
>  So
> > even if the compiler makes a mistake, we will still likely catch it.  I
> > don't think your more rigorous definition of MC/DC is necessary to
> achieve
> > that goal, is it?
>
> I certainly think that even 100% branch coverage (also called Decision
> Coverage) is quite impressive in itself without MC/DC, and if what the
> SQLite test harness does is plain Condition/Decision Coverage, I'm
> sure that already places SQLite within the best tested 1% of software.
> I'm happy and impressed with how tested it is, I just don't think it's
> MC/DC :)
>
> As far as I can tell 100% MC/DC coverage is only required in highly
> safety critical systems, most notably the DO-178B standard for
> avionics for the most critical (Class A) software only, where failure
> would be "catastrophic", i.e. result in planes falling from the sky. I
> don't know how much it would make sense to strive for such a standard
> in SQLite, unless someone really wants to use it in such a highly
> critical system. MC/DC testing is often criticized for its high
> cost, after all[2].
>
> [1] http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage
> [2] http://sunnyday.mit.edu/papers/dupuy.pdf
>
>        Sami
> _______________________________________________
> sqlite-users mailing list
> sqlite-users@sqlite.org
> http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users
>



-- 
D. Richard Hipp
drh@sqlite.org
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[prev in list] [next in list] [prev in thread] [next in thread]