[prev in list] [next in list] [prev in thread] [next in thread]
List: libreswan-dev
Subject: Re: [Swan-dev] changes in test results
From: Paul Wouters <paul () nohats ! ca>
Date: 2019-02-21 15:55:09
Message-ID: alpine.LRH.2.21.1902211047170.21330 () bofh ! nohats ! ca
[Download RAW message or body]
On Thu, 21 Feb 2019, D. Hugh Redelmeier wrote:
> PLEASE FIX THESE APPARENTLY TRIVIAL PROBLEMS.
SADLY NOT ALL OF THEM ARE TRIVIAL :)
Let me comment on the ones I can from the top of my head
> testing/pluto/nss-cert-nosecret failed east:output-different west:output-different
This is a bug. The test case shows the problem. A code fix is required,
and not trivial. This is a reminder we need to fix this (let's not start
another 'we need anothe bug category' discussion now.
> testing/pluto/nss-cert-ocsp-02 failed east:output-different
> testing/pluto/nss-cert-ocsp-02-ikev2 failed east:output-different
> testing/pluto/nss-cert-ocsp-07-nourl failed east:output-different
> testing/pluto/nss-cert-ocsp-08-post failed east:output-different
Usually due to not regenerating certificates?
> Somehow some cert errors are duplicate.
That should probably just be cleaned up in code.
> testing/pluto/newoe-21-liveness-clear failed east:output-different
>
> Timing changed withing log message. Is this just non-determinism?
> Does this need sanitization?
all dpd/liveness tests need a better approach to reduce false positives.
They are all too fragile.
> (4)
>
> testing/pluto/newoe-27-replace-sa-auth-authnull failed road:output-different
>
> EVENT_SA_REKEY is now used where EVENT_SA_REPLACE was previously used.
> Reference log should be updated.
If that's the only thing sure. This test did indicate a recent
regression that was fixed, so care should be taken it is still properly
working.
> testing/pluto/interop-ikev1-strongswan-01-xauth failed east:output-different west:output-different
> +strongswan 5.7.1 must be installed
>
> What's this about? I seem to have 5.7.2 installed.
A few days ago, fedora updates pushed 5.7.2. I will bump the versions
required.
> - should that not be good enough?
It is not about good enough. When people use different versions, they
keep overwriting each others test results with minor variances. That
is why I forced a single version or completely fail the test.
> - should the message not report what version is actually installed?
>
> Test requires strongswan 5.7.1 but 5.7.2 is installed
patches accepted :)
Paul
_______________________________________________
Swan-dev mailing list
Swan-dev@lists.libreswan.org
https://lists.libreswan.org/mailman/listinfo/swan-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic