[prev in list] [next in list] [prev in thread] [next in thread] 

List:       sqlite-users
Subject:    Re: [sqlite] MC/DC coverage explained wrong in the home page?
From:       Sami Liedes <sliedes () cc ! hut ! fi>
Date:       2011-09-25 13:03:09
Message-ID: 20110925130309.GR31808 () sli ! homeunix ! net
[Download RAW message or body]

On Sat, Sep 24, 2011 at 10:05:36PM -0400, Pavel Ivanov wrote:
> No, that's exactly the point that Richard tries to explain to you and
> you don't seem to understand. SQLite's testing coverage is a coverage
> of the resulting object code, not some source code lines. And so it

Yes, I now realize that (in fact realized it yesterday from the idea
that shortcuts are considered branches for branch coverage). And that
means that SQLite really is tested more thoroughly than I thought,
which is obviously good :)

Then it seems a bit... ingenious to navigate around the independent
effect analysis requirement (and reducing MC/DC to plain C/DC) by
analyzing object code.

Incidentally the only standard I know that requires 100% MC/DC
coverage (DO-178B, 100% MC/DC coverage required for avionics software
that can cause plane crashes) appears to clarify this precise issue in
a clarification of the standard (DO-248A, FAQ #42, as quoted in [1]
page 11, under the heading "A Note on Source versus Object Code
Coverage"). They explicitly take the position that if the analysis is
done from object code, you still need to demonstrate MC/DC coverage at
the source code level:

------------------------------------------------------------
There has been debate as to whether structural coverage, MC/DC in
particular, can be demonstrated by analyzing the object code in lieu
of the source code. According to FAQ #42 in DO-248A (ref. 9),
structural coverage, including MC/DC, can be demonstrated at the
object code level...

   as long as analysis can be provided which demonstrates that the
   coverage analysis conducted at the object code will be equivalent
   to the same coverage analysis at the source code level. In fact,
   for Level A software coverage, DO-178B/ED-12B Section 6.4.4.2b
   states that if "...the compiler generates object code that is not
   directly traceable to Source Code statements. Then, additional
   verification should be performed on the object code..." This is
   often satisfied by analyzing the object code to ensure that it is
   directly traceable to the source code. Hence, DO-178B/ED-12B
   determines the conditions for analysis of the source code for
   structural coverage, and it does not prevent one from performing
   analysis directly on the object code.
------------------------------------------------------------

> doesn't matter how the code was written, it doesn't matter if it used
> "if" statement or not. What matters is if resulting assembler has
> conditional jump instruction or not, if there is branching in the
> object code or not. And as Richard have shown you there could be
> pretty legitimate reasons for a "dead code" which is not a bug. And

On the source code level it's not dead code if it can be executed in
some configuration of the program. And if it cannot, it's truly
redundant and suggestive of a bug.

> > Dead code is indicative of a bug. Usually programmers write code only
> > in the expectation that it be run. The same can be said of conditions
> > in an if statement, and what the compiler does to it doesn't change
> > that.
> 
> Programmers write code expecting it to be executed, right. But also
> some programs have like 10 or 20 years of history. And all those years
> were spent not only on writing a new code, but also on doing some
> changes to the existing one. And sometimes changes in one place can
> result in a dead code in another. And that doesn't mean that there's a
> bug. It just means that this code is not needed anymore but nobody
> deleted it yet.

Sorry, I meant it's only suggestive of a bug. No, it's not necessarily
a bug.

But for the kind of processes that require MC/DC coverage, "forgotten"
code that is never executed most definitely _is_ a bug. 100% MC/DC
coverage is a heavy hammer. It shouldn't be surprising that it
requires a lot. It is only used in the most safety-critical software
projects.

> > MC/DC is just a clever way of saying that you must show that each
> > condition in an if statement really matters.
> 
> So, your point is if some condition in an if statement doesn't matter
> developer MUST remove it from if statement before it can claim MC/DC?
> Well, that's a straight road to a buggy software.

Yes, that's exactly what achieving 100% MC/DC coverage requires. I'm
sorry you don't like it. Who forces you to use it?

Note that I'm decidedly not trying to claim MC/DC coverage as a metric

  * makes any sense
  * is reasonable in the case of SQLite
  * is ever reasonable for any program

Actually in my opinion it's probably too heavy a hammer, as it also
does require designing the software to be testable in such a way.
Treat 100% MC/DC coverage requirement like a mandated coding style
that sometimes forecloses doing things you would like to do. If you
really need to code to those standards, you are hopefully generously
compensated for that nuisance. But if something in the definition of
MC/DC doesn't make sense, that's still no reason to ignore that and
yet claim 100% MC/DC coverage.

	Sami


[1] A Practical Tutorial on Modified Condition/ Decision Coverage
    (NASA),
    http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.6.1317&rep=rep1&type=pdf
_______________________________________________
sqlite-users mailing list
sqlite-users@sqlite.org
http://sqlite.org:8080/cgi-bin/mailman/listinfo/sqlite-users

[prev in list] [next in list] [prev in thread] [next in thread]