[prev in list] [next in list] [prev in thread] [next in thread]
List: python-ideas
Subject: [Python-ideas] PEP Draft: Power Assertion
From: "Noam Tenne" <noam () 10ne ! org>
Date: 2021-09-24 11:04:21
Message-ID: 9c901651-4d0e-4f5d-9bce-cb464f0249ba () www ! fastmail ! com
[Download RAW message or body]
Hi All,
Following the discussions on "Power Assertions: Is it PEP-able?", I've drafted this \
PEP. Your comments are most welcome.
PEP: 9999
Title: Power Assertion
Author: Noam Tenne <noam@10ne.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Sep-2021
Abstract
========
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the
`Groovy language`_ and is an extension to the core lib's ``assert``
keyword.
When an assertion expression evaluates to ``False``, the output shows
not only the failure, but also a breakdown of the evaluated expression
from the inner part to the outer part.
.. _Groovy language: \
http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#_power_assertions
Motivation
=========
Every test boils down to the binary statement "Is this true or false?",
whether you use the built-in assert keyword or a more advanced
assertion method provided by a testing framework.
When an assertion fails, the output is binary too —
"Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more
verbose breakdown of the difference and answer the question
"What exactly is the difference between x and y?".
This is extremely helpful, but it still focuses on the difference
between the values.
Keep in mind that a given state is normally an outcome of a series of
states, that is, one outcome is a result of multiple conditions and causes.
This is where Power Assertion comes in. It allows us to better
understand what led to the failure.
The Community Wants This
------------------------
As mentioned in the abstract, this feature was borrowed from Groovy.
It is a very popular feature within the Groovy community, also used by
projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the
`Nimoy`_ testing framework
* There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses
were overall positive:
* "This is cool." - Guido van Rossum
* "I was actually thinking exactly the opposite: this would more
useful in production than in testing."
- 2QdxY4RzWzUUiLuEï¼ potatochowder.com
* "What pytest does is awesome. I though about implementing it in the
standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/
.. _Power Assertion was explicitly requested: \
https://stackoverflow.com/a/58536986/198825
.. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html
.. _discussed in the python-ideas: \
https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4BMPG5EOB3A2ELVEWQPYRENRXHM/
Rational
========
Code Example
------------
> >
class SomeClass:
def __init__(self):
self.val = {'d': 'e'}
def __str__(self):
return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
> >
Assertion failed:
sc.val['d'] == f
| | |
| e False
|
{'d': 'e'}
Display
-------
In the output above we can see the value of every part of the
expression from left to right, mapped to their expression fragment
with the pipe (``|``).
The number of rows that are printed depend on the value of each
fragment of the expression.
If the value of a fragment is longer than the actual fragment
(``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``)
will be printed on a new line which will appear above.
Values are appended to the same line until it overflows in length to
horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the
output offered by pytest's solution.
The information that's displayed is dictated by the type.
If the type is a constant value, it will be displayed as is.
If the type implements `__str__`, then the return value of that will
be displayed.
Mechanics
---------
Reference Implementation
''''''''''''''''''''''''
The reference implementation uses AST manipulation because this is
the only way that this level of involvement can be achieved by a
third party library.
It iterates over every subexpression in the assert statement.
Subexpressions are parts of the expression separated by a lookup
(``map[key]``), an attribute reference (``key.value``) or a binary
comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain
the order of the operations in the original code, and tracks the
original code of the subexpression together with the AST code of the
subexpression and the original indices.
It then rewrites the AST of the original expression to call a
specialised assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering
function is called to render the assertion message as per the example
above.
Actual Implementation
'''''''''''''''''''''
To be discussed.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be
compiled to keep all immediate results of subexpressions on the stack.
If the final result is true, immediate results are dropped. If it is
false, the second argument of assert is evaluated and its value together
with all immediate results of the first expression, together with
references to corresponding subexpressions (as strings, ranges or AST
nodes) are passed to the special handler. That handler can be
implemented in a third-party library, because formatting and outputting
a report is a complex task. The default handler can just raise an
AttributeError.
Caveats
-------
It is important to note that expressions with side effects are affected by this \
feature. This is because in order to display this information, we must store \
references to the instances and not just the values.
Reference Implementation
========================
There's a `complete implementation`_ of this enhancement in the
`Nimoy`_ testing framework.
It uses AST manipulation to remap the expression to a `data structure`_
at compile time, so that it can then be `evaluated and printed`_ at runtime.
.. _Nimoy: https://browncoat-ninjas.github.io/nimoy/
.. _complete implementation: \
https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta
.. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expression_transformer.py#L77
.. _evaluated and printed: \
https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/power.py
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
["pep-9999.rst" (pep-9999.rst)]
PEP: 9999
Title: Power Assertion
Author: Noam Tenne <noam@10ne.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 24-Sep-2021
Abstract
========
This PEP introduces a language enhancement named "Power Assertion".
The Power Assertion is inspired by a similar feature found in the
`Groovy language`_ and is an extension to the core lib's ``assert``
keyword.
When an assertion expression evaluates to ``False``, the output shows
not only the failure, but also a breakdown of the evaluated expression
from the inner part to the outer part.
.. _Groovy language: \
http://docs.groovy-lang.org/next/html/documentation/core-testing-guide.html#_power_assertions
Motivation
=========
Every test boils down to the binary statement "Is this true or false?",
whether you use the built-in assert keyword or a more advanced
assertion method provided by a testing framework.
When an assertion fails, the output is binary too —
"Expected x, but got y".
There are helpful libraries like Hamcrest which give you a more
verbose breakdown of the difference and answer the question
"What exactly is the difference between x and y?".
This is extremely helpful, but it still focuses on the difference
between the values.
Keep in mind that a given state is normally an outcome of a series of
states, that is, one outcome is a result of multiple conditions and causes.
This is where Power Assertion comes in. It allows us to better
understand what led to the failure.
The Community Wants This
------------------------
As mentioned in the abstract, this feature was borrowed from Groovy.
It is a very popular feature within the Groovy community, also used by
projects such as `Spock`_.
On top of that, it is very much needed in the Python community as well:
* `Power Assertion was explicitly requested`_ as a feature in the
`Nimoy`_ testing framework
* There's a `similar feature in pytest`_
And when `discussed in the python-ideas`_ mailing list, the responses
were overall positive:
* "This is cool." - Guido van Rossum
* "I was actually thinking exactly the opposite: this would more
useful in production than in testing."
- 2QdxY4RzWzUUiLuEï¼ potatochowder.com
* "What pytest does is awesome. I though about implementing it in the
standard compiler since seen it the first time." - Serhiy Storchaka
.. _Spock: https://spockframework.org/
.. _Power Assertion was explicitly requested: \
https://stackoverflow.com/a/58536986/198825
.. _similar feature in pytest: https://docs.pytest.org/en/latest/how-to/assert.html
.. _discussed in the python-ideas: \
https://mail.python.org/archives/list/python-ideas@python.org/thread/T26DR4BMPG5EOB3A2ELVEWQPYRENRXHM/
Rational
========
Code Example
------------
> >
class SomeClass:
def __init__(self):
self.val = {'d': 'e'}
def __str__(self):
return str(self.val)
sc = SomeClass()
assert sc.val['d'] == 'f'
This routine will result in the output:
> >
Assertion failed:
sc.val['d'] == f
| | |
| e False
|
{'d': 'e'}
Display
-------
In the output above we can see the value of every part of the
expression from left to right, mapped to their expression fragment
with the pipe (``|``).
The number of rows that are printed depend on the value of each
fragment of the expression.
If the value of a fragment is longer than the actual fragment
(``{'d': 'e'}`` is longer than ``sc``), then the next value (``e``)
will be printed on a new line which will appear above.
Values are appended to the same line until it overflows in length to
horizontal position of the next fragment.
This way of presentation is clearer and more human friendly than the
output offered by pytest's solution.
The information that's displayed is dictated by the type.
If the type is a constant value, it will be displayed as is.
If the type implements `__str__`, then the return value of that will
be displayed.
Mechanics
---------
Reference Implementation
''''''''''''''''''''''''
The reference implementation uses AST manipulation because this is
the only way that this level of involvement can be achieved by a
third party library.
It iterates over every subexpression in the assert statement.
Subexpressions are parts of the expression separated by a lookup
(``map[key]``), an attribute reference (``key.value``) or a binary
comparison operator (``==``).
It then builds an AST in the structure of a tree to maintain
the order of the operations in the original code, and tracks the
original code of the subexpression together with the AST code of the
subexpression and the original indices.
It then rewrites the AST of the original expression to call a
specialized assertion function, which accepts the tree as a parameter.
At runtime the expression is executed. If it fails, a rendering
function is called to render the assertion message as per the example
above.
Actual Implementation
'''''''''''''''''''''
To be discussed.
In the python-ideas mailing list, Serhiy Storchaka suggests:
It needs a support in the compiler. The condition expression should be
compiled to keep all immediate results of subexpressions on the stack.
If the final result is true, immediate results are dropped. If it is
false, the second argument of assert is evaluated and its value together
with all immediate results of the first expression, together with
references to corresponding subexpressions (as strings, ranges or AST
nodes) are passed to the special handler. That handler can be
implemented in a third-party library, because formatting and outputting
a report is a complex task. The default handler can just raise an
AttributeError.
Caveats
-------
It is important to note that expressions with side effects are affected by this \
feature. This is because in order to display this information, we must store \
references to the instances and not just the values.
Reference Implementation
========================
There's a `complete implementation`_ of this enhancement in the
`Nimoy`_ testing framework.
It uses AST manipulation to remap the expression to a `data structure`_
at compile time, so that it can then be `evaluated and printed`_ at runtime.
.. _Nimoy: https://browncoat-ninjas.github.io/nimoy/
.. _complete implementation: \
https://browncoat-ninjas.github.io/nimoy/examples/#power-assertions-beta
.. _data structure: https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/ast_tools/expression_transformer.py#L77
.. _evaluated and printed: \
https://github.com/browncoat-ninjas/nimoy/blob/develop/nimoy/assertions/power.py
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
_______________________________________________
Python-ideas mailing list -- python-ideas@python.org
To unsubscribe send an email to python-ideas-leave@python.org
https://mail.python.org/mailman3/lists/python-ideas.python.org/
Message archived at https://mail.python.org/archives/list/python-ideas@python.org/message/TNMUISK7Y24E62B2AIBIPNPFRDVLTWIH/
Code of Conduct: http://python.org/psf/codeofconduct/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic