[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] MC/DC coverage explained wrong in the home page?
From:       Sami Liedes <sliedes () cc ! hut ! fi>
Date:       2011-09-23 17:51:11
Message-ID: 20110923175111.GM31808 () sli ! homeunix ! net
[Download RAW message or body]

On Fri, Sep 23, 2011 at 11:16:37AM -0400, Richard Hipp wrote:
> Opinions vary on the exact meaning of MC/DC for a language (such as C) that
> has short-circuit boolean operators.  You are advocating a more rigorous
> view of MC/DC that what I have heard before.  This is not to say it is
> wrong, only different.

I think you are describing (plain) Condition/Decision Coverage, not
Modified Condition/Decision Coverage (see below).

> For a decision of the form:
> 
>      if( A && B )...
> 
> The test suite for SQLite tries at least three cases:
> 
>    (1)  A false
>    (2)  A true and B false
>    (3)  A true and B true
> 
> You seem to be saying that this is only MC/DC if we split case (1) into two
> subcases:
> 
>    (1a) A false and B false
>    (1b) A false and B true

Hmm, no, not quite, although you do have a point here that in
shortcutting languages it's different. I would concede that in a
shortcutting language you do not need to test for (1a) and (1b).

I think it means that for the decision of the form if( A && B ), you
additionally need to show that A and B both can independently affect
the outcome; that is,

For A: Either (2) or (3) results in different branch being taken from
(1) (which in this case would be true for (3), but the analysis is not
always so straightforward)

For B: The branches taken in (2) and (3) are different (which is the
case here).

So in case of if( A && B ), I think you are correct that these three
test cases suffice for MC/DC. But you *have* to consider the outcome
too for it to be MC/DC.

Taking a look at the Wikipedia page[1] for MC/DC, I believe what you
describe is plain Condition/Decision Coverage, not MC/DC:

------------------------------------------------------------
  * Condition/Decision Coverage

	Every point of entry and exit in the program has been invoked
	at least once, every condition in a decision in the program
	has taken all possible outcomes at least once, and every
	decision in the program has taken all possible outcomes at
	least once.

  * Modified Condition/Decision Coverage

	Every point of entry and exit in the program has been invoked
	at least once, every condition in a decision in the program
	has taken on all possible outcomes at least once, and each
	condition has been shown to affect that decision outcome
	independently. A condition is shown to affect a decision's
	outcome independently by varying just that condition while
	holding fixed all other possible conditions. The
	condition/decision criterion does not guarantee the coverage
	of all conditions in the module because in many test cases,
	some conditions of a decision are masked by the other
	conditions. Using the modified condition/decision criterion,
	each condition must be shown to be able to act on the decision
	outcome by itself, everything else being held fixed. The MC/DC
	criterion is thus much stronger than the condition/decision
	coverage.
------------------------------------------------------------

> Your argument is that there might be redundancy; that if B is always false
> when A is false, then it would be possible to simplify the decision to just:
> 
>     if( B )...
> 
> True enough.  But there are countless ways to refactor a boolean
> expression.  I don't think that MC/DC has any requirement that the number of
> conditions be minimized, does it?

I think that's one of the central ideas of MC/DC. Well, I think it
doesn't require *minimizing* per se, only that each condition
independently is enough to flip the outcome. (I now think I was wrong
to talk about subexpressions in my original post; The "conditions" are
the atomic conditions, not subexpressions.) I certainly cannot imagine
of another plausible reading of the fourth condition. Perhaps you have
a different plausible interpretation?

In fact nearly all definitions of MC/DC I can find contain this
paragraph, which explains it quite well:

   "A condition is shown to affect a decision's outcome independently
    by varying just that decision while holding fixed all other
    possible conditions."

> One of the key benefits we derive from testing SQLite to 100% branch
> coverage is that we end up testing the object code, not the source code.  So
> even if the compiler makes a mistake, we will still likely catch it.  I
> don't think your more rigorous definition of MC/DC is necessary to achieve
> that goal, is it?

I certainly think that even 100% branch coverage (also called Decision
Coverage) is quite impressive in itself without MC/DC, and if what the
SQLite test harness does is plain Condition/Decision Coverage, I'm
sure that already places SQLite within the best tested 1% of software.
I'm happy and impressed with how tested it is, I just don't think it's
MC/DC :)

As far as I can tell 100% MC/DC coverage is only required in highly
safety critical systems, most notably the DO-178B standard for
avionics for the most critical (Class A) software only, where failure
would be "catastrophic", i.e. result in planes falling from the sky. I
don't know how much it would make sense to strive for such a standard
in SQLite, unless someone really wants to use it in such a highly
critical system. MC/DC testing is often criticized for its high
cost, after all[2].

[1] http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage
[2] http://sunnyday.mit.edu/papers/dupuy.pdf

	Sami
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[prev in list] [next in list] [prev in thread] [next in thread]