[prev in list] [next in list] [prev in thread] [next in thread] 

List:       python-ideas
Subject:    Re: [Python-ideas] PEP 484: Generating test inputs from type hints
From:       David MacIver <david () drmaciver ! com>
Date:       2015-03-31 7:10:59
Message-ID: CADZYRLd37CYewza8kL8HG1A3_1SQK773ihmNGnW03vef8qZu=Q () mail ! gmail ! com
[Download RAW message or body]

[Attachment #2 (multipart/alternative)]


(Sorry for the manual thread continuation, I wasn't previously on the
mailing list)

I presume 'support' could and would include a version of given() that
> accessed .__annotations__.


This is on my list of planned features independently of PEP 484 FWIW. It's
on the long list of nice to have but minor things that didn't make it into
1.0.

One of the problems is that I am for the moment somewhat committed to
supporting Python 2.7 as well as Python 3 (I really wish I wasn't), or I
would have probably built Hypothesis on using annotations from the
beginning.

To a degree though it's fortunate that I didn't given PEP 484. It doesn't
look like I would be able to build the full range of features Hypothesis
needs for @given using only type annotations, and 484 suggests the goal is
to move to those being the only valid ones.

The reason for this is that Hypothesis deliberately lets you mix strategies
in with the specifiers used for annotation. So for example you can do
something like:

Palindromes = strategy(str).map(lambda x: x + str(reversed(x)))

This is now a SearchStrategy object that still returns strings, but only
ones that are palindromes.

You can now do something like

@given([Palindromes])
def test_this_is_a_list_of_palindromes(xs):
   # I'm not actually sure what interesting things you can say about a list
of palindromes...

So the point is that you can use a SearchStrategy deep inside the "type"
signatures that Hypothesis uses, despite SearchStrategy not actually being
at all type-like.

(It actually used to be the case that an earlier version of SearchStrategy
allowed you to test whether a value could have come from that strategy,
which would have made them a bit more "type-like", but this proved to be a
bad idea and the work that let me remove it was one of the best decisions I
made in the design of Hypothesis)

So basically I would fully intend for Hypothesis to become a consumer of
types and annotations, but I don't think they can be used as a primary
building block of it.

(Note: I don't consider this a problem with PEP 484, which looks like a
good proposal. Hypothesis's requirements are just weird and specialized and
don't quite match type checking)


> Of course, if every 'test_xyz' were to be
> decorated with the same decorator, then the decorator could be omitted
> and the fuzzing moved into the test runner.


I'd be moderately against this being the primary mode of using Hypothesis,
even without the above caveats. One of the things I like about Hypothesis
right now is that because it just gives you Python functions it can be very
unopinionated about what you're using to run your tests (even if in
practice basically everyone is using unittest or pytest).  However it
should be relatively easy to support alongside once I've done some much
needed refactoring of the core example exploration code.

David

[Attachment #5 (text/html)]

<div dir="ltr">(Sorry for the manual thread continuation, I wasn&#39;t previously on \
the mailing list)<div><br></div><div><blockquote style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" \
class="gmail_quote">I presume &#39;support&#39; could and would include a version of \
given() that <br>accessed .__annotations__. </blockquote><div><br></div><div>This is \
on my list of planned features independently of PEP 484 FWIW. It&#39;s on the long \
list of nice to have but minor things that didn&#39;t make it into \
1.0.</div><div><br></div><div>One of the problems is that I am for the moment \
somewhat committed to supporting Python 2.7 as well as Python 3 (I really wish I \
wasn&#39;t), or I would have probably built Hypothesis on using annotations from the \
beginning.</div><div><br></div><div>To a degree though it&#39;s fortunate that I \
didn&#39;t given PEP 484. It doesn&#39;t look like I would be able to build the full \
range of features Hypothesis needs for @given using only type annotations, and 484 \
suggests the goal is to move to those being the only valid \
ones.</div><div><br></div><div>The reason for this is that Hypothesis deliberately \
lets you mix strategies in with the specifiers used for annotation. So for example \
you can do something like:</div><div><br></div><div>Palindromes = \
strategy(str).map(lambda x: x + str(reversed(x)))</div><div><br></div><div>This is \
now a SearchStrategy object that still returns strings, but only ones that are \
palindromes.</div><div><br></div><div>You can now do something \
like</div><div><br></div><div>@given([Palindromes])</div><div>def \
test_this_is_a_list_of_palindromes(xs):</div><div>     # I&#39;m not actually sure \
what interesting things you can say about a list of \
palindromes...</div><div><br></div><div>So the point is that you can use a \
SearchStrategy deep inside the &quot;type&quot; signatures that Hypothesis uses, \
despite SearchStrategy not actually being at all \
type-like.</div><div><br></div><div>(It actually used to be the case that an earlier \
version of SearchStrategy allowed you to test whether a value could have come from \
that strategy, which would have made them a bit more &quot;type-like&quot;, but this \
proved to be a bad idea and the work that let me remove it was one of the best \
decisions I made in the design of Hypothesis)</div><div><br></div><div>So basically I \
would fully intend for Hypothesis to become a consumer of types and annotations, but \
I don&#39;t think they can be used as a primary building block of \
it.</div><div><br></div><div>(Note: I don&#39;t consider this a problem with PEP 484, \
which looks like a good proposal. Hypothesis&#39;s requirements are just weird and \
specialized and don&#39;t quite match type checking)</div><div>  </div><blockquote \
style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex" \
class="gmail_quote"> Of course, if every &#39;test_xyz&#39; were to be <br>decorated \
with the same decorator, then the decorator could be omitted <br>and the fuzzing \
moved into the test runner.</blockquote><div><br></div><div>I&#39;d be moderately \
against this being the primary mode of using Hypothesis, even without the above \
caveats. One of the things I like about Hypothesis right now is that because it just \
gives you Python functions it can be very unopinionated about what you&#39;re using \
to run your tests (even if in practice basically everyone is using unittest or \
pytest).   However it should be relatively easy to support alongside once I&#39;ve \
done some much needed refactoring of the core example exploration \
code.</div></div><div><br></div><div>David</div></div>



_______________________________________________
Python-ideas mailing list
Python-ideas@python.org
https://mail.python.org/mailman/listinfo/python-ideas
Code of Conduct: http://python.org/psf/codeofconduct/

[prev in list] [next in list] [prev in thread] [next in thread] 

Configure | About | News | Add a list | Sponsored by KoreLogic