[prev in list] [next in list] [prev in thread] [next in thread]
List: outages
Subject: Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?
From: "Hunt, Fred A - DOA via Outages-discussion" <outages-discussion () outages ! org>
Date: 2023-08-30 21:11:52
Message-ID: SA1PR09MB80790CD30345A596DF5E7AD38CE6A () SA1PR09MB8079 ! namprd09 ! prod ! outlook ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
[Attachment #4 (text/plain)]
The irony
[cid:image001.png@01D9DB5C.AE4916C0]
From: Outages-discussion <outages-discussion-bounces@outages.org> On Behalf Of Hunt, \
Fred A - DOA via Outages-discussion
Sent: Wednesday, August 30, 2023 3:21 PM
To: 'Pete Eisengrein' <peeip989@gmail.com>
Cc: 'outages-discussion@outages.org' <outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages-discussion] Microsoft Teams outage?
CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the \
content is safe.
Nice observation. That's the same as what I see here, after further review. I reached \
out to a contact at Microsoft earlier, but have not heard back. Am going to pass this \
along and press for some ownership of the cause.
From: Pete Eisengrein <peeip989@gmail.com<mailto:peeip989@gmail.com>>
Sent: Wednesday, August 30, 2023 2:55 PM
To: Hunt, Fred A - DOA <FredA.Hunt@wisconsin.gov<mailto:FredA.Hunt@wisconsin.gov>>
Cc: outages@voiceops.org<mailto:outages@voiceops.org>
Subject: Re: [outages] Microsoft Teams outage?
CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the \
content is safe.
I don't believe we have gotten any sort of RCA from them yet but we noted that the \
User-Agent, while calls were failing, had "v.2023.08.28.1" in it and after the \
problem cleared the User-Agent had "v.2023.06.29.3.i" in it, so it smells like a \
rollback to me.
On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein \
<peeip989@gmail.com<mailto:peeip989@gmail.com>> wrote: It is Direct Routing, but we \
weren't getting the 400 Bad Request, and for us the BYE was originating from the PSTN \
towards Microsoft due to no audio. This magically began to clear around noon eastern.
On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA \
<FredA.Hunt@wisconsin.gov<mailto:FredA.Hunt@wisconsin.gov>> wrote: Are you using \
Teams Direct Routing? This morning it does appear that Teams is not cleanly \
terminating calls. Here's an example response from Teams to a SIP BYE message sent
148249222: Aug 30 15:09:39.986: \
//12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:
Received: SIP TLS message from \
52.114.132.46:5061<http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsYuRC0nyv \
-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0LOF0ctOj \
awA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L_ZlJkKb \
dpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0Kzp2VSNM \
aEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061> \
to [REDACTED] SIP/2.0 400 Bad Request
FROM: "[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282
TO: sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0
CSEQ: 102 BYE
CALL-ID: B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]<mailto:B629C8A-467E11EE-B3F38F31-9397CD96@TeamsMADSBC.wisconsin.gov>
VIA: SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8
REASON: Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected a Protocol \
of Reason header as a token, got '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB \
C;8t\5cx96C^]BYE sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c"
CONTENT-LENGTH: 0
ALLOW: INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY
SERVER: Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10
Nothing has changed with what we are sending to MS in the SIP Reason header and I \
also see plenty of calls today that do terminate cleanly.
From: Outages <outages-bounces@outages.org<mailto:outages-bounces@outages.org>> On \
Behalf Of Pete Eisengrein via Outages
Sent: Wednesday, August 30, 2023 10:32 AM
To: outages@voiceops.org<mailto:outages@voiceops.org>
Subject: [outages] Microsoft Teams outage?
CAUTION: This email originated from outside the organization.
Do not click links or open attachments unless you recognize the sender and know the \
content is safe.
We are experiencing an issue with Microsoft this morning, where they initially \
connect but when we send a re-INVITE, Microsoft responds with a 100 Trying but then \
never updates the session and the call is left with dead air and eventually the \
caller hangs up.
Waiting on Microsoft support.....
Thanks
[Attachment #5 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"> <head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">The irony<o:p></o:p></p>
<p class="MsoNormal"><img width="378" height="154" \
style="width:3.9333in;height:1.6083in" id="Picture_x0020_1" \
src="cid:image001.png@01D9DB5C.AE4916C0"><o:p></o:p></p> <p \
class="MsoNormal"><o:p> </o:p></p> <div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion \
<outages-discussion-bounces@outages.org> <b>On Behalf Of </b>Hunt, Fred A - DOA \
via Outages-discussion<br> <b>Sent:</b> Wednesday, August 30, 2023 3:21 PM<br>
<b>To:</b> 'Pete Eisengrein' <peeip989@gmail.com><br>
<b>Cc:</b> 'outages-discussion@outages.org' \
<outages-discussion@outages.org><br> <b>Subject:</b> Re: [Outages-discussion] \
[outages-discussion] Microsoft Teams outage?<o:p></o:p></p> </div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: \
This email originated from outside the organization. </span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do \
not click links or open attachments unless you recognize the sender and know the \
content is safe.</span></strong><o:p></o:p></p> <p><o:p> </o:p></p>
<p class="MsoNormal">Nice observation. That's the same as what I see here, after \
further review. I reached out to a contact at Microsoft earlier, but have not heard \
back. Am going to pass this along and press for some ownership of the \
cause.<o:p></o:p></p> <p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Pete Eisengrein <<a \
href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>> <br>
<b>Sent:</b> Wednesday, August 30, 2023 2:55 PM<br>
<b>To:</b> Hunt, Fred A - DOA <<a \
href="mailto:FredA.Hunt@wisconsin.gov">FredA.Hunt@wisconsin.gov</a>><br> \
<b>Cc:</b> <a href="mailto:outages@voiceops.org">outages@voiceops.org</a><br> \
<b>Subject:</b> Re: [outages] Microsoft Teams outage?<o:p></o:p></p> </div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: \
This email originated from outside the organization. </span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do \
not click links or open attachments unless you recognize the sender and know the \
content is safe.</span></strong><o:p></o:p></p> <p><o:p> </o:p></p>
<div>
<p class="MsoNormal">I don't believe we have gotten any sort of RCA from them yet but \
we noted that the User-Agent, while calls were failing, had \
"v.2023.08.28.1" in it and after the problem cleared the User-Agent had \
"v.2023.06.29.3.i" in it, so it smells like a rollback to \
me.<o:p></o:p></p> </div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 1:11 PM Pete Eisengrein <<a \
href="mailto:peeip989@gmail.com">peeip989@gmail.com</a>> wrote:<o:p></o:p></p> \
</div> <blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in \
0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> \
<div> <p class="MsoNormal">It is Direct Routing, but we weren't getting the 400 Bad \
Request, and for us the BYE was originating from the PSTN towards Microsoft due to no \
audio. This magically began to clear around noon eastern.<o:p></o:p></p> </div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Wed, Aug 30, 2023 at 12:37 PM Hunt, Fred A - DOA <<a \
href="mailto:FredA.Hunt@wisconsin.gov" \
target="_blank">FredA.Hunt@wisconsin.gov</a>> wrote:<o:p></o:p></p> </div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in \
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Are \
you using Teams Direct Routing? This morning it does appear that Teams is not cleanly \
terminating calls. Here's an example response from Teams to a SIP BYE message \
sent<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">148249222: \
Aug 30 15:09:39.986: \
//12929228/0B60EED0B3ED/CUBE_VT/SIP/Msg/ccsipDisplayMsg:</span><o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">Received: SIP \
TLS message from <a href="http://secure-web.cisco.com/1kABKIdweERcFsWcUUBGyvAmisgiygsY \
uRC0nyv-GTfYKBW3pxm949BdAiAN3C_8yUAx-QnIz7eb_mjLOOVx8PB3hfdi45FUXQAxFlnTjX-YKUgyoGGr0L \
OF0ctOjawA15Yzkynjm4iLYdTxtd2743gKvLkgI05zkJNEyfJ_lbuseCBcDCRJFoDObUUxqM9h1IjiNPBrzt9L \
_ZlJkKbdpgI0H8IeAHiAcjmt_EQR3eItzV0m4jbg9k_5gwMlPG2uynes4MAhTp4B_7ko7umxwZPRnSIpLTRQ0K \
zp2VSNMaEEKRoF8no6cfRJo_H2OVLxtKcbuTGuChX8VN5Pd98IgIJyDPpvq2AjzBOpSkItfdxo/http%3A%2F%2F52.114.132.46%3A5061" \
target="_blank"> 52.114.132.46:5061</a> to [REDACTED] </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier \
New";color:black;background:yellow">SIP/2.0 400 Bad \
Request</span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">FROM: \
"[REDACTED]"sip:+[REDACTED]@[REDACTED];tag=59E32C35-1282</span><o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">TO: <a \
href="sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0">
sip:+[REDACTED]@sip.pstnhub.microsoft.com;tag=42b61721ea1d4686bf22e4b2f03384d0</a></span><o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">CSEQ: 102 \
BYE</span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">CALL-ID: <a \
href="mailto:B629C8A-467E11EE-B3F38F31-9397CD96@TeamsMADSBC.wisconsin.gov" \
target="_blank"> B629C8A-467E11EE-B3F38F31-9397CD96@[REDACTED]</a></span><o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">VIA: \
SIP/2.0/TLS [REDACTED];branch=z9hG4bK3B67395A8</span><o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier \
New";color:black;background:yellow">REASON: \
Q.850;cause=95;text="9e76dc15-53f0-4eb9-82b0-215b7e029e87;Expected a Protocol of \
Reason header as a token, got '\5cx17\5cx03\5cx03\5cx03n\5cx8BC^YB C;8t\5cx96C^]BYE \
<a href="sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c">
sip:api-du-a-usea.pstnhub.microsoft.com:443;x-i=9e76dc15-53f0-4eb9-82b0-215b7e029e87;x-c=4c4164749a1c57e5958505fe4de5d0cb/s/1/dd4c</a>"</span><o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier \
New";color:black">CONTENT-LENGTH: 0</span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">ALLOW: \
INVITE,ACK,OPTIONS,CANCEL,BYE,NOTIFY</span><o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span \
style="font-size:9.0pt;font-family:"Courier New";color:black">SERVER: \
Microsoft.PSTNHub.SIPProxy v.2023.8.28.1 i.USEA.10</span><o:p></o:p></p> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Nothing has changed with \
what we are sending to MS in the SIP Reason header and I also see plenty of calls \
today that do terminate cleanly.<o:p></o:p></p> <p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> <div \
style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> <p \
class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b> \
Outages <<a href="mailto:outages-bounces@outages.org" \
target="_blank">outages-bounces@outages.org</a>> <b>On Behalf Of </b>Pete \
Eisengrein via Outages<br> <b>Sent:</b> Wednesday, August 30, 2023 10:32 AM<br>
<b>To:</b> <a href="mailto:outages@voiceops.org" \
target="_blank">outages@voiceops.org</a><br> <b>Subject:</b> [outages] Microsoft \
Teams outage?<o:p></o:p></p> </div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
<p><strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">CAUTION: \
This email originated from outside the organization. </span></strong><br>
<strong><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:white;background:#FF6600">Do \
not click links or open attachments unless you recognize the sender and know the \
content is safe.</span></strong><o:p></o:p></p> <p> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We \
are experiencing an issue with Microsoft this morning, where they initially connect \
but when we send a re-INVITE, Microsoft responds with a 100 Trying but then never \
updates the session and the call is left with dead air and eventually the caller \
hangs up.<o:p></o:p></p> <div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
</div> <div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Waiting on Microsoft \
support.....<o:p></o:p></p> </div>
<div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p> \
</div> <div>
<p class="MsoNormal" \
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks<o:p></o:p></p> \
</div> </div>
</div>
</div>
</div>
</blockquote>
</div>
</blockquote>
</div>
</div>
</body>
</html>
["image001.png" (image/png)]
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
--===============0186378046189618944==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic