[prev in list] [next in list] [prev in thread] [next in thread]
List: fwknop-discuss
Subject: Re: [Fwknop-discuss] fwknop test coverage
From: "Sebastien J." <s.j () mac ! com>
Date: 2014-05-04 18:31:42
Message-ID: 82B5C8CA-6A0B-4F05-9CD6-4F5D7B811495 () mac ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi Mike,
This is excellent stuff. More open source projects should strive to do this!
Thanks for your efforts.
// Sebastien
> On 4 May 2014, at 02:57 pm, Michael Rash <michael.rash@gmail.com> wrote:
>
>
> Hi all,
>
> Thought I would provide some detail on an fwknop development effort around test \
> coverage. The fwknop test suite (test/test-fwknop.pl) has done a pretty good job \
> of validating functionality for a while now, but a lot more tests are being added \
> to try and get to 100% test coverage on a function, line, and branch basis. It \
> turns out that achieving "100%" coverage for all three of these is extremely hard, \
> and that is a general statement for all software - not just fwknop.
> Cutting to the chase, here are the current test coverage results as measure by the \
> "gcov" tool with these .html views generated by "lcov":
> http://www.cipherdyne.org/fwknop/lcov-results/
>
> So, in terms of functions, the test suite does fairly well with slightly over 99% \
> of all functions covered. Line coverage is less at about 85% and branch coverage \
> is even harder at about 75%. While this may not sound very good, it turns out that \
> the usage of defensive checks (good for security) inflate these numbers. That is, \
> fwknop is careful to validate all sorts of things at various stages, and sometimes \
> these checks are redundant making later checks harder to trigger than earlier ones. \
> Another example is that all functions that deal with heap allocated memory have \
> NULL checks as one would expect, but in practice it is hard to trigger a NULL \
> result unless there is extreme memory pressure and this would affect test suite \
> operations at the same time (and worse). In some cases, it would be possible to \
> account for these problems by building in special test coverage support into the \
> fwknop code base, and some of this has already started. For example, the new \
> python fuzzer (see test/spa_fuzzer.py) leverages special #define's to assist with \
> test coverage. This approach could be extended to do things like maintain a global \
> counter that is incremented with any malloc() (and friends) calls, and then force a \
> NULL result whenever this counter hits any of a set of values during normal fwknop \
> operations.
> All of the above is run time testing, and is complementary to using static \
> analyzers like Coverity. The goal is to maintain an extremely high standard for \
> the fwknop code base.
> Thanks,
>
> --Mike
> ------------------------------------------------------------------------------
> "Accelerate Dev Cycles with Automated Cross-Browser Testing - For FREE
> Instantly run your Selenium tests across 300+ browser/OS combos. Get
> unparalleled scalability from the best Selenium testing platform available.
> Simple to use. Nothing to install. Get started now for free."
> http://p.sf.net/sfu/SauceLabs
> _______________________________________________
> Fwknop-discuss mailing list
> Fwknop-discuss@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/fwknop-discuss
[Attachment #5 (text/html)]
<html><head><meta http-equiv="content-type" content="text/html; \
charset=utf-8"></head><body dir="auto"><div>Hi Mike,</div><div><br></div><div>This is \
excellent stuff. More open source projects should strive to do \
this!</div><div><br></div><div>Thanks for your efforts. <br><br>// \
Sebastien</div><div><br>On 4 May 2014, at 02:57 pm, Michael Rash <<a \
href="mailto:michael.rash@gmail.com">michael.rash@gmail.com</a>> \
wrote:<br><br></div><blockquote type="cite"><div><div \
dir="ltr"><div><div><div><div><div><div><br></div>Hi all,<br><br></div>Thought I \
would provide some detail on an fwknop development effort around test coverage. \
The fwknop test suite (test/<a href="http://test-fwknop.pl">test-fwknop.pl</a>) has \
done a pretty good job of validating functionality for a while now, but a lot more \
tests are being added to try and get to 100% test coverage on a function, line, and \
branch basis. It turns out that achieving "100%" coverage for all three of \
these is extremely hard, and that is a general statement for all software - not just \
fwknop.<br> <br></div>Cutting to the chase, here are the current test coverage \
results as measure by the "gcov" tool with these .html views generated by \
"lcov":<br><br><a href="http://www.cipherdyne.org/fwknop/lcov-results/">http://www.cipherdyne.org/fwknop/lcov-results/</a><br>
<br></div>So, in terms of functions, the test suite does fairly well with slightly \
over 99% of all functions covered. Line coverage is less at about 85% and \
branch coverage is even harder at about 75%. While this may not sound very \
good, it turns out that the usage of defensive checks (good for security) inflate \
these numbers. That is, fwknop is careful to validate all sorts of things at \
various stages, and sometimes these checks are redundant making later checks harder \
to trigger than earlier ones. Another example is that all functions that deal with \
heap allocated memory have NULL checks as one would expect, but in practice it is \
hard to trigger a NULL result unless there is extreme memory pressure and this would \
affect test suite operations at the same time (and worse). In some cases, it \
would be possible to account for these problems by building in special test coverage \
support into the fwknop code base, and some of this has already started. For \
example, the new python fuzzer (see test/spa_fuzzer.py) leverages special #define's \
to assist with test coverage. This approach could be extended to do things like \
maintain a global counter that is incremented with any malloc() (and friends) calls, \
and then force a NULL result whenever this counter hits any of a set of values during \
normal fwknop operations.<br> <br></div>All of the above is run time testing, and is \
complementary to using static analyzers like Coverity. The goal is to maintain \
an extremely high standard for the fwknop code \
base.<br><br>Thanks,<br><br></div>--Mike<br> </div>
</div></blockquote><blockquote \
type="cite"><div><span>------------------------------------------------------------------------------</span><br><span>"Accelerate \
Dev Cycles with Automated Cross-Browser Testing - For FREE</span><br><span>Instantly \
run your Selenium tests across 300+ browser/OS combos. Get \
</span><br><span>unparalleled scalability from the best Selenium testing platform \
available.</span><br><span>Simple to use. Nothing to install. Get started now for \
free."</span><br><span><a \
href="http://p.sf.net/sfu/SauceLabs">http://p.sf.net/sfu/SauceLabs</a></span></div></blockquote><blockquote \
type="cite"><div><span>_______________________________________________</span><br><span>Fwknop-discuss \
mailing list</span><br><span><a \
href="mailto:Fwknop-discuss@lists.sourceforge.net">Fwknop-discuss@lists.sourceforge.net</a></span><br><span><a \
href="https://lists.sourceforge.net/lists/listinfo/fwknop-discuss">https://lists.sourc \
eforge.net/lists/listinfo/fwknop-discuss</a></span><br></div></blockquote></body></html>
------------------------------------------------------------------------------
Is your legacy SCM system holding you back? Join Perforce May 7 to find out:
• 3 signs your SCM is hindering your productivity
• Requirements for releasing software faster
• Expert tips and advice for migrating your SCM now
http://p.sf.net/sfu/perforce
_______________________________________________
Fwknop-discuss mailing list
Fwknop-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/fwknop-discuss
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic