[prev in list] [next in list] [prev in thread] [next in thread]
List: serusers
Subject: Re: [SR-Users] Recommended openSSL version
From: Henning Westerholt <hw () gilawa ! com>
Date: 2022-10-19 17:36:51
Message-ID: AM6PR05MB5409C2FA135EC0BA4F5A79B1BF2B9 () AM6PR05MB5409 ! eurprd05 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (text/plain)]
Hello,
thanks for reporting your test results.
Regarding the "exit" topic – if you want to close tcp connections from the cfg \
script, 5.6.x has tcp_close_connection(..) in tcpops available.
Cheers,
Henning
--
Henning Westerholt – https://skalatan.de/blog/
Kamailio services – https://gilawa.com<https://gilawa.com/>
From: sr-users <sr-users-bounces@lists.kamailio.org> On Behalf Of Ihor Olkhovskyi
Sent: Tuesday, October 18, 2022 2:17 PM
To: Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org>
Subject: Re: [SR-Users] Recommended openSSL version
Hello,
Sorry for bumping this old up, but some outcome from my research.
1. CentOS 7 provided OpenSSL (1.0.2k-fips 26 Jan 2017) really leads Kamailio 5.x.x \
crash on high load (tested with 5.4 - 5.6) with \
sippts<https://github.com/Pepelux/sippts> tool.
2. Good results are obtained with Kamailio 5.6.2 with tlsa flavour statically linked \
with openssl 1.1.1q (here I have problem with lacking of TLS connections, but it's \
something different)
And with this result I have a question, when I'm invoking
exit;
on Kamailio script it's not "freeing" TCP connection as I got, I've managed "freeing" \
(or not occupying) connection with iptables
-j REJECT --reject-with tcp-reset
Is there anything same for Kamailio or I need to add smth like fail2ban on top?
Thanks in advance!
Le 24/06/2022 à 14:15, Igor Olhovskiy a écrit :
Daniel,
Thanks for clarifying this!
And to ask, is websocket module also uses libssl indirectly or should not be the \
cause in this one? (I'm not using http or so).
Le ven. 24 juin 2022 Ã 08:36, Daniel-Constantin Mierla \
<miconda@gmail.com<mailto:miconda@gmail.com>> a écrit :
Hello,
to add to this topic: tls module runs smooth when no other module uses an external \
library that is linked also with tls, I didn't have issue with in the past few years.
But if another module that indirectly links also the libssl, I also got random \
crashes, usually during events when kamailio code is not involved at all. For \
example, a while ago using the http_client module (which uses libcurl that linked \
also libssl) resulted in sporadic crashes during tls handshake -- that's all in \
libssl, nothing to do with sip traffic at that stage. And actually there were also \
crashes when opening the connection to the https server. The behaviour was \
non-deterministic, months without any issue, then 1-2 crashes in a week or so, then \
all good as well. I somehow related it to minor updates of the operating system.
After all, I ended up writing ruxc module to have an alternative http_client() \
function and from that moment no libssl related crash on the respective system. \
Strange that on another customer having same OS and using http_client() function, all \
was and still is fine. So it could be also related to tls settings in both sides of \
the connection (e.g., ciphers, renegotiation, tls version, ...).
If you migrate to kamailio 5.6.x, then you can also try using tlsa module instead of \
tls, that should isolate the global libssl contexts, one inside the tlsa and one in \
those modules linking dynamically libssl.
Cheers,
Daniel
On 23.06.22 16:46, Karsten Horsmann wrote:
Hi Igor,
I jumped from 5.3 to 5.5.x so I read carefull the changelog and migrate steps.
https://www.kamailio.org/wiki/features/new-in-5.5.x
Show a bit about tls.
Igor Olhovskiy <igorolhovskiy@gmail.com<mailto:igorolhovskiy@gmail.com>> schrieb am \
Mi., 22. Juni 2022, 21:08:
Karsten,
Thanks for your answer!
Out of your head, were there any significant changes in TCP/TLS on 5.4 -> 5.5 change?
Regards,
Igor
Le 22.06.2022 à 18:11, Karsten Horsmann a écrit :
Hi Igor,
I also use CentOS 7 with the same openssl version and between 1000 up to 2000 tls/wss \
connections.
Works for me. Main difference I use Kamailio 5.5.x
Kind regards
Karsten Horsmann
Igor Olhovskiy <igorolhovskiy@gmail.com<mailto:igorolhovskiy@gmail.com>> schrieb am \
Mi., 22. Juni 2022, 10:36: Hello!
Due to I still experience irregular Kamailio 5.4 crashes (like 1/month) related to \
SSL (using websockets and SIPS) I'm wondering, could openSSL upgrade change the \
situation? As of now in CentOS 7 I have 1.0.2k version.
Does anyone have experience to fix crash-related to TLS problems with openSSL \
upgrade?
Or maye some tuneup of TCP parameters can help here?My current setup is quite simple:
children=4
enable_tls=yes
tcp_accept_no_cl=yes
tcp_connection_lifetime=600
tcp_max_connections=998976 # 1000000 - 1024, so we're leaving 1k for system reserve
tls_max_connections=998976
Number of clients ~ 200 constantly connected to websocket.
--
Best regards,
Igor
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com<http://www.asipto.com>
www.twitter.com/miconda<http://www.twitter.com/miconda> -- \
www.linkedin.com/in/miconda<http://www.linkedin.com/in/miconda>
Kamailio Advanced Training - Online: June 20-23, 2022
* https://www.asipto.com/sw/kamailio-advanced-training-online/
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
* sr-users@lists.kamailio.org<mailto:sr-users@lists.kamailio.org>
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
* https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Best regards,
Igor
[Attachment #3 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Vorformatiert Zchn";
margin:0cm;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLVorformatiertZchn
{mso-style-name:"HTML Vorformatiert Zchn";
mso-style-priority:99;
mso-style-link:"HTML Vorformatiert";
font-family:Consolas;}
span.E-MailFormatvorlage21
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="DE" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span \
style="mso-fareast-language:EN-US">Hello,<o:p></o:p></span></p> <p \
class="MsoNormal"><span \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">thanks for \
reporting your test results.<o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-GB" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Regarding the \
"exit" topic – if you want to close tcp connections from the cfg script, 5.6.x has \
tcp_close_connection(..) in tcpops available.<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US">Cheers,<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US">Henning<o:p></o:p></span></p> <p \
class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">-- <o:p>
</o:p></span></p>
<p class="MsoNormal"><span lang="EN-GB" style="mso-fareast-language:EN-US">Henning \
Westerholt – </span><span style="mso-fareast-language:EN-US"><a \
href="https://skalatan.de/blog/"><span lang="EN-GB" \
style="color:#0563C1">https://skalatan.de/blog/</span></a></span><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p></o:p></span></p> <p class="MsoNormal"><span \
lang="EN-GB" style="mso-fareast-language:EN-US">Kamailio services – </span><span \
style="mso-fareast-language:EN-US"><a href="https://gilawa.com/"><span lang="EN-GB" \
style="color:#0563C1">https://gilawa.com</span></a></span><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p></o:p></span></p> </div>
<p class="MsoNormal"><span lang="EN-GB" \
style="mso-fareast-language:EN-US"><o:p> </o:p></span></p> <div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" style="margin-left:35.4pt"><b>From:</b> sr-users \
<sr-users-bounces@lists.kamailio.org> <b>On Behalf Of </b>Ihor Olkhovskyi<br>
<b>Sent:</b> Tuesday, October 18, 2022 2:17 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List \
<sr-users@lists.kamailio.org><br> <b>Subject:</b> Re: [SR-Users] Recommended \
openSSL version<o:p></o:p></p> </div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<p style="margin-left:35.4pt">Hello,<o:p></o:p></p>
<p style="margin-left:35.4pt">Sorry for bumping this old up, but some outcome from my \
research.<o:p></o:p></p> <p style="margin-left:35.4pt">1. CentOS 7 provided OpenSSL \
(1.0.2k-fips 26 Jan 2017) really leads Kamailio 5.x.x crash on high load \
(tested with 5.4 - 5.6) with <a href="https://github.com/Pepelux/sippts">sippts</a> \
tool.<o:p></o:p></p> <p style="margin-left:35.4pt">2. Good results are obtained with \
Kamailio 5.6.2 with tlsa flavour statically linked with openssl 1.1.1q (here I have \
problem with lacking of TLS connections, but it's something different)<o:p></o:p></p> \
<p style="margin-left:35.4pt">And with this result I have a question, when I'm \
invoking <o:p></o:p></p>
<p style="margin-left:35.4pt">exit;<o:p></o:p></p>
<p style="margin-left:35.4pt">on Kamailio script it's not "freeing" TCP \
connection as I got, I've managed "freeing" (or not occupying) connection \
with iptables <o:p></o:p></p>
<p style="margin-left:35.4pt">-j REJECT --reject-with tcp-reset<o:p></o:p></p>
<p style="margin-left:35.4pt">Is there anything same for Kamailio or I need to add \
smth like fail2ban on top?<o:p></o:p></p> <p style="margin-left:35.4pt">Thanks in \
advance!<o:p></o:p></p> <p style="margin-left:35.4pt">Le 24/06/2022 Ã 14:15, Igor \
Olhovskiy a écrit :<o:p></o:p></p> <blockquote \
style="margin-top:5.0pt;margin-bottom:5.0pt"> <div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Daniel,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Thanks for clarifying \
this!<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">And to ask, is websocket module also \
uses libssl indirectly or should not be the cause in this one? (I'm not using http or \
so).<o:p></o:p></p> </div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Le ven. 24 juin 2022 Ã \
08:36, Daniel-Constantin Mierla <<a \
href="mailto:miconda@gmail.com">miconda@gmail.com</a>> a \
écrit :<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-right:0cm"> <div>
<p style="margin-left:35.4pt">Hello,<o:p></o:p></p>
<p style="margin-left:35.4pt">to add to this topic: tls module runs smooth when no \
other module uses an external library that is linked also with tls, I didn't have \
issue with in the past few years.<o:p></o:p></p> <p style="margin-left:35.4pt">But if \
another module that indirectly links also the libssl, I also got random crashes, \
usually during events when kamailio code is not involved at all. For example, a while \
ago using the http_client module (which uses libcurl that linked also libssl) \
resulted in sporadic crashes during tls handshake -- that's all in libssl, nothing to \
do with sip traffic at that stage. And actually there were also crashes when opening \
the connection to the https server. The behaviour was non-deterministic, months \
without any issue, then 1-2 crashes in a week or so, then all good as well. I somehow \
related it to minor updates of the operating system.<o:p></o:p></p> <p \
style="margin-left:35.4pt">After all, I ended up writing ruxc module to have an \
alternative http_client() function and from that moment no libssl related crash on \
the respective system. Strange that on another customer having same OS and using \
http_client() function, all was and still is fine. So it could be also related to \
tls settings in both sides of the connection (e.g., ciphers, renegotiation, tls \
version, ...).<o:p></o:p></p> <p style="margin-left:35.4pt">If you migrate to \
kamailio 5.6.x, then you can also try using tlsa module instead of tls, that should \
isolate the global libssl contexts, one inside the tlsa and one in those modules \
linking dynamically libssl.<o:p></o:p></p> <p style="margin-left:35.4pt">Cheers,<br>
Daniel<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">On 23.06.22 16:46, Karsten Horsmann \
wrote:<o:p></o:p></p> </div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Igor, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I jumped from 5.3 to 5.5.x so I read \
carefull the changelog and migrate steps. <o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><a \
href="https://www.kamailio.org/wiki/features/new-in-5.5.x" \
target="_blank">https://www.kamailio.org/wiki/features/new-in-5.5.x</a><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
Show a bit about tls. <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Igor Olhovskiy <<a \
href="mailto:igorolhovskiy@gmail.com" target="_blank">igorolhovskiy@gmail.com</a>> \
schrieb am Mi., 22. Juni 2022, 21:08:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-right:0cm"> <div>
<p style="margin-left:35.4pt">Karsten,<o:p></o:p></p>
<p style="margin-left:35.4pt">Thanks for your answer!<o:p></o:p></p>
<p style="margin-left:35.4pt">Out of your head, were there any significant changes in \
TCP/TLS on 5.4 -> 5.5 change?<o:p></o:p></p> <pre \
style="margin-left:35.4pt">Regards,<o:p></o:p></pre> <pre \
style="margin-left:35.4pt">Igor<o:p></o:p></pre> <div>
<p class="MsoNormal" style="margin-left:35.4pt">Le 22.06.2022 Ã 18:11, Karsten \
Horsmann a écrit :<o:p></o:p></p> </div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hi Igor, <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">I also use CentOS 7 with the same \
openssl version and between 1000 up to 2000 tls/wss connections. <o:p></o:p></p> \
</div> <div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Works for me. Main difference I use \
Kamailio 5.5.x<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Kind regards <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:0cm;margin-right:0cm;margin-bottom:12.0pt;margin-left:35.4pt">
Karsten Horsmann <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Igor Olhovskiy <<a \
href="mailto:igorolhovskiy@gmail.com" target="_blank">igorolhovskiy@gmail.com</a>> \
schrieb am Mi., 22. Juni 2022, 10:36:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm \
6.0pt;margin-left:4.8pt;margin-right:0cm"> <div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Hello!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Due to I still experience irregular \
Kamailio 5.4 crashes (like 1/month) related to SSL (using websockets and SIPS) I'm \
wondering, could openSSL upgrade change the situation?<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">As of now in CentOS 7 I have 1.0.2k \
version. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Does anyone have experience to fix \
crash-related to TLS problems with openSSL upgrade?<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Or maye some tuneup of TCP parameters \
can help here?My current setup is quite simple:<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">children=4<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">enable_tls=yes<br>
tcp_accept_no_cl=yes<br>
tcp_connection_lifetime=600<br>
tcp_max_connections=998976 # 1000000 - 1024, so we're leaving 1k for system \
reserve<br> tls_max_connections=998976<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Number of clients ~ 200 constantly \
connected to websocket.<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Best regards, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Igor<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" \
style="margin-left:35.4pt">__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" \
target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not reply only to the \
sender!<br> Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" \
target="_blank"> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><br>
<br>
<o:p></o:p></p>
<pre style="margin-left:35.4pt">__________________________________________________________<o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio - Users Mailing List - Non Commercial \
Discussions<o:p></o:p></pre> <pre style="margin-left:35.4pt"> * <a \
href="mailto:sr-users@lists.kamailio.org" \
target="_blank">sr-users@lists.kamailio.org</a><o:p></o:p></pre> <pre \
style="margin-left:35.4pt">Important: keep the mailing list in the recipients, do not \
reply only to the sender!<o:p></o:p></pre> <pre style="margin-left:35.4pt">Edit \
mailing list options or unsubscribe:<o:p></o:p></pre> <pre \
style="margin-left:35.4pt"> * <a \
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" \
target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
</div>
<p class="MsoNormal" \
style="margin-left:35.4pt">__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" \
target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not reply only to the \
sender!<br> Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" \
target="_blank"> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><br>
<br>
<o:p></o:p></p>
<pre style="margin-left:35.4pt">__________________________________________________________<o:p></o:p></pre>
<pre style="margin-left:35.4pt">Kamailio - Users Mailing List - Non Commercial \
Discussions<o:p></o:p></pre> <pre style="margin-left:35.4pt"> * <a \
href="mailto:sr-users@lists.kamailio.org" \
target="_blank">sr-users@lists.kamailio.org</a><o:p></o:p></pre> <pre \
style="margin-left:35.4pt">Important: keep the mailing list in the recipients, do not \
reply only to the sender!<o:p></o:p></pre> <pre style="margin-left:35.4pt">Edit \
mailing list options or unsubscribe:<o:p></o:p></pre> <pre \
style="margin-left:35.4pt"> * <a \
href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" \
target="_blank">https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></pre>
</blockquote>
<pre style="margin-left:35.4pt">-- <o:p></o:p></pre>
<pre style="margin-left:35.4pt">Daniel-Constantin Mierla -- <a \
href="http://www.asipto.com" target="_blank">www.asipto.com</a><o:p></o:p></pre> <pre \
style="margin-left:35.4pt"><a href="http://www.twitter.com/miconda" \
target="_blank">www.twitter.com/miconda</a> -- <a \
href="http://www.linkedin.com/in/miconda" \
target="_blank">www.linkedin.com/in/miconda</a><o:p></o:p></pre> <pre \
style="margin-left:35.4pt">Kamailio Advanced Training - Online: June 20-23, \
2022<o:p></o:p></pre> <pre style="margin-left:35.4pt"> * <a \
href="https://www.asipto.com/sw/kamailio-advanced-training-online/" \
target="_blank">https://www.asipto.com/sw/kamailio-advanced-training-online/</a><o:p></o:p></pre>
</div>
<p class="MsoNormal" \
style="margin-left:35.4pt">__________________________________________________________<br>
Kamailio - Users Mailing List - Non Commercial Discussions<br>
* <a href="mailto:sr-users@lists.kamailio.org" \
target="_blank">sr-users@lists.kamailio.org</a><br>
Important: keep the mailing list in the recipients, do not reply only to the \
sender!<br> Edit mailing list options or unsubscribe:<br>
* <a href="https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users" \
target="_blank"> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="margin-left:35.4pt"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Best regards, <o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-left:35.4pt">Igor<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
</body>
</html>
__________________________________________________________
Kamailio - Users Mailing List - Non Commercial Discussions
sr-users@lists.kamailio.org
Important: keep the mailing list in the recipients, do not reply only to the sender!
Edit mailing list options or unsubscribe:
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--===============0632265834==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic