[prev in list] [next in list] [prev in thread] [next in thread] 

List:       freebsd-hackers
Subject:    Re: Developers' Handbook: Chapter on testing
From:       Yoshihiro Ota <ota () j ! email ! ne ! jp>
Date:       2020-09-16 4:16:00
Message-ID: 20200916001600.9741e3230466be3a0e1ad098 () j ! email ! ne ! jp
[Download RAW message or body]

Hi,

My responses are embedded below.

By the way, is it okay to continue here or should I join freebst-testing?

On Mon, 14 Sep 2020 08:25:21 +0000
Lawrence Greening <lg49152@outlook.com> wrote:

> Hi.
> 〓
> A variation of the following was posted to freebsd-testing. I would generally avoid \
> cross-posting but I'm reposting here because of recent interest in testing on this \
> list... 
> I am writing a chapter on testing for the FreeBSD Developers' Handbook.
> 
> The following topics are expected to be covered:
> 
> * FreeBSD Test Suite
> * Kyua
> * Automated Testing Framework (ATF)
> * Organization of current FreeBSD tests.
> * Using automation and virtualization to reduce the overhead in setting up the test
> infrastructure.
> * Examples of different types of testing.
> * Reproducibility in building and testing.
> 
> This list of topics is just a starting point. The scope and organization will \
> evolve as the chapter is written.
> 
> After reading the chapter, the reader should be able to do at least the following:
> 
> * Install and configure the necessary testing tools, test suites, etc. for the \
> purposes of testing FreeBSD.
> * Run tests.
> * Write tests.
> * Submit tests.
> 
> The chapter will include walkthroughs and guided or cumulative examples.

That sounds a good starting point and what I started looking around.

I found some extra documents below after posting my previous question.
Mark and Warner's responses were also helpful to understand further.

https://wiki.freebsd.org/TestSuite
https://wiki.freebsd.org/TestSuite/DeveloperHowTo
https://wiki.freebsd.org/TestSuite/Structure

These wiki covers some of topics above.

> At this preliminary stage, I am interested in the following feedback:
> 
> (1) What material do you want to see covered?

In addition to the list you have, topics on kernel code will be great.

> (2) What material do you want to see prioritized?

My personal interest is step by step guide to setup a test environment and one or few \
examples of how to start writing additional test cases.

> (3) What general approach(es) do you want to see taken towards the subject matter?

We need a good test coverage and start writing more test cases.  My suggestion is \
lowering bar with good documents with examples to encourage people writing more test \
cases.

Then, I'd like to have code-coverage tool run together.  Then, I can check how/if my \
new code reduced test coverage and my new test cases increased test coverage.

It will be even nicer if all tests are run by Jenkins and phabricator bounces back \
ones with broken test cases.

> (4) Are there good example tests or areas of the code base that deserve focus, \
> either because of their importance, usefulness as examples, history of regressions, \
> etc.?

Not familiar enough on FreeBSD test case but I recently realized some of user land \
programs actively adding more test cases from svn mailing list.  That's a good trend.

> (5) Anything else?

In my experiences, I like unit test cases at C/C++ API level.  These allow me to test \
same scenarios over and over again each time I make a change.  These also detect \
un-intended behavior changes.  Unit test cases also pays off very quickly.

I also think integration tests are important but some programs like client/server \
have extra difficulties.

> Cheers,
> Lawrence

Regards,
Hiro
_______________________________________________
freebsd-hackers@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org"


[prev in list] [next in list] [prev in thread] [next in thread]