[prev in list] [next in list] [prev in thread] [next in thread]
List: asterisk-dev
Subject: [asterisk-dev] [Code Review] 3489: testsuite: Improve logging
From: "Matt Jordan" <reviewboard () asterisk ! org>
Date: 2014-04-28 20:18:41
Message-ID: 20140428201841.16090.78307 () sonic ! digium ! api
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3489/
-----------------------------------------------------------
Review request for Asterisk Developers.
Repository: testsuite
Description
-------
This patch changes up the logging in the Asterisk Test Suite. It dramatically reduces \
the amount of logging done in the logs/ directory by default (although this can \
always be bumped up), and it redirects the vast majority of the test suite logs to \
the actual sandboxed test run directories themselves.
When we first added Python logging (it didn't always used to be there!) there were \
far fewer tests. Having all tests dump into the same log files actually felt like a \
benefit, since during test development (or when developing many tests) it made it \
easier to debug test failures. Over time, however, this approach has run into a \
number of problems: (1) Many of the tests are much 'chattier' than they used to be. \
Having twisted, starpy, websockets, requests, and other libraries dump information \
has greatly increased the number of log messages. Not to mention logging out every \
SIPp screen received... (2) There are a lot more tests than there were when this was \
added. (3) Tailing a log file is not always the best way to debug. Sometimes you have \
to search through an entire log file for one run. Finding the error becomes \
problematic when your editor of choice chokes on the size of the file.
Hence, this patch.
Logging now works in the following way:
(1) The test_case.TestCase class now always sets up the logging. The logging set up \
done by test_runner was removed (as it only logged out a few messages before an \
instance of TestCase was created anyway). (2) A global logger can still be configured \
in logger.conf. It now only sets up a logger of INFO messages and higher. This allows \
a test executor to watch which tests are being run, without getting spammed. During \
test development, the log message level can be increased to DEBUG. (3) TestCase now \
places the Asterisk directories created by the test execution in a further subfolder, \
run_N (where N increases with each execution of that test). Where before you might \
have:
tests/my_test/ast1
tests/my_test/ast2
tests/my_test/ast3
tests/my_test/ast4
You will now have:
tests/my_test/run_1/ast1
tests/my_test/run_1/ast2
tests/my_test/run_2/ast1
tests/my_test/run_2/ast2
This lets you determine which Asterisk instances belong together, and also makes it \
possible for only the erroring test run to be archived when a test fails (as opposed \
to every Asterisk directory) (4) TestCase now creates a DEBUG and INFO log file(s) in \
the run directory. These contain the full Asterisk logs for the test run.
Diffs
-----
/asterisk/trunk/runtests.py 5002
/asterisk/trunk/logger.conf 5002
/asterisk/trunk/lib/python/asterisk/test_runner.py 5002
/asterisk/trunk/lib/python/asterisk/test_case.py 5002
/asterisk/trunk/lib/python/asterisk/asterisk.py 5002
Diff: https://reviewboard.asterisk.org/r/3489/diff/
Testing
-------
I've actually been running an instance of the test suite with this set up for several \
months, using it for development of several tests and generally tweaking it. I \
finally felt like it was "good enough".
Tested:
* Failing tests archive appropriately
* Crashing Asterisk gets archived
* Global log file is still created with expected message levels
* Test specific log files are created appropriately with expected message levels
* Tested a pluggable module based test (pbx/dialplan) and a 'traditional' Python test \
(channels/SIP/sip_hold)
Thanks,
Matt Jordan
[Attachment #5 (text/html)]
<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 \
solid;"> <tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/3489/">https://reviewboard.asterisk.org/r/3489/</a>
</td>
</tr>
</table>
<br />
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" \
style="background-image: \
url('https://reviewboard.asterisk.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); \
background-position: left top; background-repeat: repeat-x; border: 1px black \
solid;"> <tr>
<td>
<div>Review request for Asterisk Developers.</div>
<div>By Matt Jordan.</div>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
testsuite
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" \
style="border: 1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">This patch changes up the logging in the Asterisk Test Suite. It \
dramatically reduces the amount of logging done in the logs/ directory by default \
(although this can always be bumped up), and it redirects the vast majority of the \
test suite logs to the actual sandboxed test run directories themselves.
When we first added Python logging (it didn't always used to be there!) there \
were far fewer tests. Having all tests dump into the same log files actually felt \
like a benefit, since during test development (or when developing many tests) it made \
it easier to debug test failures. Over time, however, this approach has run into a \
number of problems: (1) Many of the tests are much 'chattier' than they used \
to be. Having twisted, starpy, websockets, requests, and other libraries dump \
information has greatly increased the number of log messages. Not to mention logging \
out every SIPp screen received... (2) There are a lot more tests than there were when \
this was added. (3) Tailing a log file is not always the best way to debug. Sometimes \
you have to search through an entire log file for one run. Finding the error becomes \
problematic when your editor of choice chokes on the size of the file.
Hence, this patch.
Logging now works in the following way:
(1) The test_case.TestCase class now always sets up the logging. The logging set up \
done by test_runner was removed (as it only logged out a few messages before an \
instance of TestCase was created anyway). (2) A global logger can still be configured \
in logger.conf. It now only sets up a logger of INFO messages and higher. This allows \
a test executor to watch which tests are being run, without getting spammed. During \
test development, the log message level can be increased to DEBUG. (3) TestCase now \
places the Asterisk directories created by the test execution in a further subfolder, \
run_N (where N increases with each execution of that test). Where before you might \
have:
tests/my_test/ast1
tests/my_test/ast2
tests/my_test/ast3
tests/my_test/ast4
You will now have:
tests/my_test/run_1/ast1
tests/my_test/run_1/ast2
tests/my_test/run_2/ast1
tests/my_test/run_2/ast2
This lets you determine which Asterisk instances belong together, and also makes it \
possible for only the erroring test run to be archived when a test fails (as opposed \
to every Asterisk directory) (4) TestCase now creates a DEBUG and INFO log file(s) in \
the run directory. These contain the full Asterisk logs for the test run. </pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: \
1px solid #b8b5a0"> <tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: \
-moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: \
break-word;">I've actually been running an instance of the test suite with this \
set up for several months, using it for development of several tests and generally \
tweaking it. I finally felt like it was "good enough".
Tested:
* Failing tests archive appropriately
* Crashing Asterisk gets archived
* Global log file is still created with expected message levels
* Test specific log files are created appropriately with expected message levels
* Tested a pluggable module based test (pbx/dialplan) and a 'traditional' \
Python test (channels/SIP/sip_hold)</pre> </td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/asterisk/trunk/runtests.py <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/logger.conf <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/test_runner.py <span style="color: \
grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/test_case.py <span style="color: \
grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/asterisk.py <span style="color: \
grey">(5002)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/3489/diff/" style="margin-left: \
3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-dev
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic