[prev in list] [next in list] [prev in thread] [next in thread]
List: asterisk-users
Subject: Re: [asterisk-users] Asterisk crash on core show channel
From: Marcus Kvarsell <Marcus.Kvarsell () fogwise ! se>
Date: 2018-02-22 8:29:41
Message-ID: HE1PR0601MB2617BD15C2764DE53416B45289CD0 () HE1PR0601MB2617 ! eurprd06 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Från: asterisk-users-bounces@lists.digium.com \
[mailto:asterisk-users-bounces@lists.digium.com] För Marcus Kvarsell
Skickat: den 22 februari 2018 09:14
Till: Asterisk Users Mailing List - Non-Commercial Discussion \
<asterisk-users@lists.digium.com> Ämne: Re: [asterisk-users] Asterisk crash on core \
show channel
Den här avsändaren har underkänts i bedrägerikontrollen och kanske egentligen är \
någon annan. Läs mer om förfalskning<http://aka.ms/LearnAboutSpoofing>
Feedback<http://aka.ms/SafetyTipsFeedback>
,
Från: asterisk-users-bounces@lists.digium.com<mailto:asterisk-users-bounces@lists.digium.com> \
[mailto:asterisk-users-bounces@lists.digium.com] För Marcus Kvarsell
Skickat: den 22 februari 2018 08:29
Till: Asterisk Users Mailing List - Non-Commercial Discussion \
<asterisk-users@lists.digium.com<mailto:asterisk-users@lists.digium.com>> Ämne: Re: \
[asterisk-users] Asterisk crash on core show channel
Den här avsändaren har underkänts i bedrägerikontrollen och kanske egentligen är \
någon annan. Läs mer om förfalskning<http://aka.ms/LearnAboutSpoofing>
Feedback<http://aka.ms/SafetyTipsFeedback>
Hi,
I have had alot of issues in asterisk 13 on different servers, we use them mostly \
for webrtc, confbridge and upgrading to 15 has not yet failed us after 1 month or so.
/ Marcus
Från: asterisk-users-bounces@lists.digium.com<mailto:asterisk-users-bounces@lists.digium.com> \
[mailto:asterisk-users-bounces@lists.digium.com] För Patrick Wakano
Skickat: den 21 februari 2018 23:34
Till: Asterisk Users Mailing List - Non-Commercial Discussion \
<asterisk-users@lists.digium.com<mailto:asterisk-users@lists.digium.com>> Ämne: Re: \
[asterisk-users] Asterisk crash on core show channel
Thanks for you answer Marcus,
So maybe this means some bug was fixed? Anyone aware of something related?
From the release notes, I couldn't find any direct change that could fix this....
Thanks,
Kind regards,
Patrick Wakano
On 21 February 2018 at 20:29, Marcus Kvarsell \
<Marcus.Kvarsell@fogwise.se<mailto:Marcus.Kvarsell@fogwise.se>> wrote: Hello, i found \
upgrading to asterisk 15 helped.
Från: asterisk-users-bounces@lists.digium.com<mailto:asterisk-users-bounces@lists.digium.com> \
[mailto:asterisk-users-bounces@lists.digium.com<mailto:asterisk-users-bounces@lists.digium.com>] \
För Patrick Wakano
Skickat: den 21 februari 2018 04:29
Till: Asterisk Users Mailing List - Non-Commercial Discussion \
<asterisk-users@lists.digium.com<mailto:asterisk-users@lists.digium.com>> Ämne: \
[asterisk-users] Asterisk crash on core show channel
Hello Asterisk list,
I am facing some Asterisk crashes which are consistently pointing to the same \
backtrace, which is the following (using DONT_OPTIMIZE, BETTER_BACKTRACES and \
MALLOC_DEBUG): Thread 1 (Thread 0x7f1f08be8700 (LWP 1767)):
#0 0x00007f1f9bed3395 in __strcasecmp_l_sse42 () from /lib64/libc.so.6
#1 0x00000000004a91ca in cdr_object_get_by_name_cb ()
#2 0x0000000000463c60 in internal_ao2_traverse ()
#3 0x0000000000463fc3 in __ao2_callback ()
#4 0x00000000004a9b7d in cdr_object_get_by_name ()
#5 0x00000000004a9d80 in ast_cdr_serialize_variables ()
#6 0x00000000004de411 in handle_showchan ()
#7 0x00000000004e15a5 in ast_cli_command_full ()
#8 0x00000000004e175b in ast_cli_command_multiple_full ()
#9 0x0000000000455246 in netconsole ()
#10 0x000000000060f192 in dummy_start ()
#11 0x00007f1f9cd3b9d1 in start_thread () from /lib64/libpthread.so.0
#12 0x00007f1f9be908fd in clone () from /lib64/libc.so.6
So this happens after issuing a "core show channel <channel>". This command is being \
executed by one script that periodically checks the channels and some of its \
variables. Thing is that sometimes the channel has something which causes this crash \
(happens around 2 times per month). Does anyone have any idea of what may be \
happening/wrong? I am running Asterisk version 13.6.0 on CentOS 6.6 kernel \
2.6.32-504.el6.x86_64 and on CentOS 6.9 kernel 2.6.32-696.el6.x86_64. Any idea is \
very appreciated! Best regards,
Patrick Wakano
Virus-free. www.avg.com \
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Check out the new Asterisk community forum at: https://community.asterisk.org/
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users<http://www.avg.com/email-si \
gnature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Segoe UI";
panose-1:2 11 5 2 4 2 4 2 2 3;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.E-postmall18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.E-postmall19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.E-postmall21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="SV" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Från:</b> asterisk-users-bounces@lists.digium.com \
[mailto:asterisk-users-bounces@lists.digium.com] <b>För </b>Marcus Kvarsell<br>
<b>Skickat:</b> den 22 februari 2018 09:14<br>
<b>Till:</b> Asterisk Users Mailing List - Non-Commercial Discussion \
<asterisk-users@lists.digium.com><br> <b>Ämne:</b> Re: [asterisk-users] \
Asterisk crash on core show channel<o:p></o:p></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" \
width="100%" style="width:100.0%"> <tbody>
<tr>
<td style="background:#910A19;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt \
11.25pt;word-wrap:break-word"> <div>
<p class="MsoNormal" \
style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-e \
lement-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe \
UI",sans-serif;color:#212121">Den här avsändaren har underkänts i \
bedrägerikontrollen och kanske egentligen är någon annan. Läs mer om <a \
href="http://aka.ms/LearnAboutSpoofing">förfalskning</a><o:p></o:p></span></p> \
</div> </td>
<td width="75" style="width:56.25pt;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt \
3.75pt;word-wrap:break-word;align:left"> <p class="MsoNormal" \
style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-e \
lement-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe \
UI",sans-serif;color:#212121"><a \
href="http://aka.ms/SafetyTipsFeedback">Feedback</a><o:p></o:p></span></p> </td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:EN-US">, \
<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b>Från:</b> <a \
href="mailto:asterisk-users-bounces@lists.digium.com"> \
asterisk-users-bounces@lists.digium.com</a> [<a \
href="mailto:asterisk-users-bounces@lists.digium.com">mailto:asterisk-users-bounces@lists.digium.com</a>]
<b>För </b>Marcus Kvarsell<br>
<b>Skickat:</b> den 22 februari 2018 08:29<br>
<b>Till:</b> Asterisk Users Mailing List - Non-Commercial Discussion <<a \
href="mailto:asterisk-users@lists.digium.com">asterisk-users@lists.digium.com</a>><br>
<b>Ämne:</b> Re: [asterisk-users] Asterisk crash on core show \
channel<o:p></o:p></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" align="left" \
width="100%" style="width:100.0%"> <tbody>
<tr>
<td style="background:#910A19;padding:5.25pt 1.5pt 5.25pt 1.5pt"></td>
<td width="100%" style="width:100.0%;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt \
11.25pt;word-wrap:break-word"> <div>
<p class="MsoNormal" \
style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-e \
lement-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe \
UI",sans-serif;color:#212121">Den här avsändaren har underkänts i \
bedrägerikontrollen och kanske egentligen är någon annan. Läs mer om <a \
href="http://aka.ms/LearnAboutSpoofing">förfalskning</a><o:p></o:p></span></p> \
</div> </td>
<td width="75" style="width:56.25pt;background:#FDF2F4;padding:5.25pt 3.75pt 5.25pt \
3.75pt;word-wrap:break-word;align:left"> <p class="MsoNormal" \
style="mso-element:frame;mso-element-frame-hspace:2.25pt;mso-element-wrap:around;mso-e \
lement-anchor-vertical:paragraph;mso-element-anchor-horizontal:column;mso-height-rule:exactly">
<span style="font-size:9.0pt;font-family:"Segoe \
UI",sans-serif;color:#212121"><a \
href="http://aka.ms/SafetyTipsFeedback">Feedback</a><o:p></o:p></span></p> </td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span \
style="mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p> <p \
class="MsoNormal"><span style="mso-fareast-language:EN-US">I have had alot of \
issues in asterisk 13 on different servers, we use them mostly for webrtc, confbridge \
and upgrading to 15 has not yet failed us after 1 month or so.<o:p></o:p></span></p> \
<p class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span style="mso-fareast-language:EN-US">/ \
Marcus<o:p></o:p></span></p> <p class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><b>Från:</b> <a \
href="mailto:asterisk-users-bounces@lists.digium.com"> \
asterisk-users-bounces@lists.digium.com</a> [<a \
href="mailto:asterisk-users-bounces@lists.digium.com">mailto:asterisk-users-bounces@lists.digium.com</a>]
<b>För </b>Patrick Wakano<br>
<b>Skickat:</b> den 21 februari 2018 23:34<br>
<b>Till:</b> Asterisk Users Mailing List - Non-Commercial Discussion <<a \
href="mailto:asterisk-users@lists.digium.com">asterisk-users@lists.digium.com</a>><br>
<b>Ämne:</b> Re: [asterisk-users] Asterisk crash on core show \
channel<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal">Thanks for you answer Marcus,<o:p></o:p></p>
</div>
<p class="MsoNormal">So maybe this means some bug was fixed? Anyone aware of \
something related?<o:p></o:p></p> </div>
<p class="MsoNormal">From the release notes, I couldn't find any direct change that \
could fix this....<o:p></o:p></p> </div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Kind regards,<o:p></o:p></p>
</div>
<p class="MsoNormal">Patrick Wakano<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 21 February 2018 at 20:29, Marcus Kvarsell <<a \
href="mailto:Marcus.Kvarsell@fogwise.se" \
target="_blank">Marcus.Kvarsell@fogwise.se</a>> wrote:<o:p></o:p></p> <blockquote \
style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0cm;margin-bottom:5.0pt"> <div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello, i found upgrading \
to asterisk 15 helped.<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>Från:</b>
<a href="mailto:asterisk-users-bounces@lists.digium.com" \
target="_blank">asterisk-users-bounces@lists.digium.com</a> [mailto:<a \
href="mailto:asterisk-users-bounces@lists.digium.com" \
target="_blank">asterisk-users-bounces@lists.digium.com</a>] <b>För </b>Patrick \
Wakano<br> <b>Skickat:</b> den 21 februari 2018 04:29<br>
<b>Till:</b> Asterisk Users Mailing List - Non-Commercial Discussion <<a \
href="mailto:asterisk-users@lists.digium.com" \
target="_blank">asterisk-users@lists.digium.com</a>><br> <b>Ämne:</b> \
[asterisk-users] Asterisk crash on core show channel<o:p></o:p></p> <div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
<div> <div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hello \
Asterisk list,<o:p></o:p></p> </div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">I am facing \
some Asterisk crashes which are consistently pointing to the same backtrace, which is \
the following (using DONT_OPTIMIZE, BETTER_BACKTRACES and MALLOC_DEBUG):<br> Thread 1 \
(Thread 0x7f1f08be8700 (LWP 1767)):<br> #0 0x00007f1f9bed3395 in \
__strcasecmp_l_sse42 () from /lib64/libc.so.6<br> #1 0x00000000004a91ca in \
cdr_object_get_by_name_cb ()<br> #2 0x0000000000463c60 in internal_ao2_traverse \
()<br> #3 0x0000000000463fc3 in __ao2_callback ()<br>
#4 0x00000000004a9b7d in cdr_object_get_by_name ()<br>
#5 0x00000000004a9d80 in ast_cdr_serialize_variables ()<br>
#6 0x00000000004de411 in handle_showchan ()<br>
#7 0x00000000004e15a5 in ast_cli_command_full ()<br>
#8 0x00000000004e175b in ast_cli_command_multiple_full ()<br>
#9 0x0000000000455246 in netconsole ()<br>
#10 0x000000000060f192 in dummy_start ()<br>
#11 0x00007f1f9cd3b9d1 in start_thread () from /lib64/libpthread.so.0<br>
#12 0x00007f1f9be908fd in clone () from /lib64/libc.so.6<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">So this \
happens after issuing a "core show channel <channel>". This command \
is being executed by one script that periodically checks the channels and some of its \
variables. Thing is that sometimes the channel has something which causes this crash \
(happens around 2 times per month). Does anyone have any idea of what may be \
happening/wrong?<br> I am running Asterisk version 13.6.0 on CentOS 6.6 kernel \
2.6.32-504.el6.x86_64 and on CentOS 6.9 kernel 2.6.32-696.el6.x86_64.<o:p></o:p></p> \
</div> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Any idea is very \
appreciated!<o:p></o:p></p> </div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Best \
regards,<o:p></o:p></p> </div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Patrick \
Wakano<o:p></o:p></p> <div>
<div>
<div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
</div> </div>
</div>
</div>
</div>
<div id="m_-8534169556919179230DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2">
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
<table class="MsoNormalTable" border="1" cellpadding="0" \
style="border:none;border-top:solid #D3D4DE 1.0pt"> <tbody>
<tr>
<td width="55" style="width:41.25pt;border:none;padding:9.75pt .75pt .75pt .75pt">
</td>
<td width="470" style="width:352.5pt;border:none;padding:9.0pt .75pt .75pt .75pt">
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;line-height:13.5pt"> <span \
style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#41424E"><a \
href="http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" \
target="_blank"><span style="color:#41424E;text-decoration:none">Virus-free. \
</span><span style="color:#4453EA">www.avg.com</span><span \
style="color:#41424E;text-decoration:none"> </span></a></span><span \
class="MsoHyperlink"><span \
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p> </td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a \
href="http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" \
target="_blank"><span \
style="color:windowtext;text-decoration:none"> </span></a><span \
class="MsoHyperlink"><span \
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p> </div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><a \
href="http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail" \
target="_blank"><span \
style="color:windowtext;text-decoration:none"><br>
--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by </span>http://www.api-digital.com<span \
style="color:windowtext;text-decoration:none"> --<br> <br>
Check out the new Asterisk community forum at: \
</span>https://community.asterisk.org/<span \
style="color:windowtext;text-decoration:none"><br> <br>
New to Asterisk? Start here:<br>
\
</span>https://wiki.asterisk.org/wiki/display/AST/Getting+Started<span \
style="color:windowtext;text-decoration:none"><br> <br>
asterisk-users mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
</span>http://lists.digium.com/mailman/listinfo/asterisk-users</a><span \
class="MsoHyperlink"><span \
style="color:windowtext;text-decoration:none"><o:p></o:p></span></span></p> \
</blockquote> </div>
</div>
</div>
</div>
</div>
</body>
</html>
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Check out the new Asterisk community forum at: https://community.asterisk.org/
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
--===============5117059994114557033==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic