[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ltp-list
Subject:    Re: [LTP] Handle Bad Test Cases
From:       Daniel Gollub <dgollub () suse ! de>
Date:       2008-10-23 18:09:03
Message-ID: 200810232009.03788.dgollub () suse ! de
[Download RAW message or body]

On Thursday 23 October 2008 19:47:56 CAI Qian wrote:
> Hi,
> 
> Ptrace06 has lots of issues. It fails to build on IA-64, unable to compile on earlier Kernels
> (more details on those two issues could be found on previous threads) and there is new issue I
> have just spotted.
> 
> In spawn_ptrace_child.h,
> 
[...]

> 
> The parent returns from the function in which vfork() was called. As the results, the test result
> is random on x86-64,
> 
> ptrace06    1  PASS  :  ptrace(PEEKDATA, ..., (nil), (nil)) failed as expected
> 
> or
> 
> ptrace06    1  FAIL:  ptrace(PEEKDATA, ..., (nil), (nil)) expected errno EIO or EFAULT; actual: No
> such process
> 
> or
> 
> ptrace06    1  FAIL:  ptrace(PEEKDATA, ..., (nil), (nil)) returned 0 instead of -1
> 
> According vfork() manpage, the code is does not look right.
> 
> "The vfork() function has the same effect as fork(2), except that the behavior is undefined if 
> the process created  by  vfork()  either modifies  any  data  other than a variable of type pid_t
> used to store the return value from vfork(), or returns from the function in which vfork() was
> called,"
> 
> Also, it leaves lots of processes after the test completes.
> 
> qcai      4503  0.0  0.0   4092   408 pts/8    S    01:22   0:00 ./ptrace06 child
> qcai      4541  0.0  0.0   4092   408 pts/8    S    01:23   0:00 ./ptrace06 child
> qcai      4584  0.0  0.0   4092   408 pts/8    S    01:23   0:00 ./ptrace06 child
> qcai      4606  0.0  0.0   4092   404 pts/8    S    01:23   0:00 ./ptrace06 child

I have seen something similar in the past with times03 (iirc) which was about
about a compiler optimization which could tight assembler infinite loop and
interfered with further running tests -> bad impact on testresults.
(This  got already fixed in beginning of the year...)

Is there already some code in runltp to catch such processes and abort the
entire testrun?  Test processes leaking out of the scope of the testdriver
is a pretty serious thing - isn't it? We should definitely ban such broken
tests and prepare runltp to detect such processes? In my opinion aborting
the testrun immediately is better to have wrong/invalid testresults...


> The test has been disabled in Makefile at the moment, but I would like to discuss how we could
> handle the similar case in the future. In this case, it is unnecessary to remove it from CVS
> entirely, but it is kind of untidy and confusion to users to leave in the test directory, but
> never been compiled.

Fully agree.

I'm pretty new to LTP development, but what about the open-posix-testsuite?
It's also disable by default - looking into open-posix-testsuite show also
definitely some issues ...

(And those "grep -v" are really not very obvious - i have to admin i didn't
 actively checked the commit logs or documentation - maybe i'm missing some
 obvious documentation about this...)

> How about to create a separate directory (testcases/working_in_progress?) to
> move all those broken test code there, and each of them has an associate file to document why it
> is here and what need to be done to move it to the test directory it should belong?

Good idea!

How to handle testcases which build fine on all architectures and all
kernel versions - but can't trigger a certain error-path for a kernel
interface for a specific kernel version? Or not on specific Architectures?
Or combined: build on all kernels, but is not testable on 2.6.27 on x86_64?

Example: testcases/open_posix_testsuite/conformance/interfaces/mmap/31-1.c
I'm looking into this one right now - i guess it's not possible to trigger
EOVERFLOW on x86_64 with current (or even any?) kernel on x86_64.

best regards,
Daniel

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Ltp-list mailing list
Ltp-list@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ltp-list
[prev in list] [next in list] [prev in thread] [next in thread]