[prev in list] [next in list] [prev in thread] [next in thread]
List: outages
Subject: Re: [outages] Level 3 Detroit issues again
From: Bill Wichers <billw () waveform ! net>
Date: 2014-03-08 16:09:56
Message-ID: AE0A995A-F597-4831-A5D1-2861F4054840 () waveform ! net
[Download RAW message or body]
It appears to have been a new issue, but possibly related to the previous one. \
Level(3) reported the following this morning:
13:53 GMT<x-apple-data-detectors://1> - The IP NOC reports that protocol drops \
following a scheduled maintenance, GCR 7622319<tel:7622319>, in Detroit, MI impacted \
IP services. The issue existed on a connection between two of the routers in Detroit, \
MI. The IP NOC has disabled that connection to allow traffic to flow through \
alternate routes and eliminate the protocol drops. The IP NOC will continue \
investigating the faulty connection in an attempt to determine the root cause and \
restore the connection for service. At this time, the IP NOC reports that services \
are up and stable
Sent from my iPhone
On Mar 8, 2014, at 6:36 AM, "Mitch Patterson" \
<mitpatterson@gmail.com<mailto:mitpatterson@gmail.com>> wrote:
They were doing emergency maintance likely related to the other night
On Mar 8, 2014 6:34 AM, "Greg Smythe" \
<greg@thesmythes.org<mailto:greg@thesmythes.org>> wrote: As I type this I am seeing \
Level3 issues in the Detroit area. Been going on since about 4:30am Eastern. My co-lo \
provider had to disable peering to get us back and running.
-=-=-=-=-=-=-=-=-=-=--=-=-
Greg Smythe
From: Outages [mailto:outages-bounces@outages.org<mailto:outages-bounces@outages.org>] \
On Behalf Of Mitch Patterson
Sent: Friday, March 07, 2014 12:51 AM
To: Bill Wichers
Cc: outages@outages.org<mailto:outages@outages.org>
Subject: Re: [outages] Level 3 Detroit issues again
Just got this
*** CASCADED EXTERNAL NOTES 07-Mar-2014 05:48:37 GMT From CASE: 7621845 - Event The \
Operations Engineering team has isolated the cause of the packet loss failure. A \
change in parameters on the Detroit's network router has been initiated to restore \
normal operations.
On Fri, Mar 7, 2014 at 12:41 AM, Mitch Patterson \
<mitpatterson@gmail.com<mailto:mitpatterson@gmail.com>> wrote: Just got this
03/07/2014 05:06 AM GMT
*** CASCADED EXTERNAL NOTES 07-Mar-2014 05:06:43 GMT From CASE: 7621845 - Event
The Technical Services Center (TSC) detected a packet loss condition on a network \
router in Detroit, MI. The Operations Engineering team has been engaged on the issue \
and they are currently analyzing the packet loss condition to determine the cause of \
the failure.
On Fri, Mar 7, 2014 at 12:01 AM, Mitch Patterson \
<mitpatterson@gmail.com<mailto:mitpatterson@gmail.com>> wrote: Do you know are the \
circuits you are on legacy GLBX? from what i'm told by our guy who handles the colo \
the legacy GLBX ones are fine. but the other ones are not.
On Thu, Mar 6, 2014 at 11:59 PM, Bill Wichers \
<billw@waveform.net<mailto:billw@waveform.net>> wrote: We connect into them in \
Detroit and we don't see this. We aren't in their colo though.
-Bill
Sent from my iPhone
On Mar 6, 2014, at 11:44 PM, "Mitch Patterson" \
<mitpatterson@gmail.com<mailto:mitpatterson@gmail.com>> wrote:
> We are seeing over 20% packet loss on our circuits at their colo in Southfield. \
> They are telling me its been refereed to their IP core team as its a high level \
> issue. Anyone else seeing this? _______________________________________________
> Outages mailing list
> Outages@outages.org<mailto:Outages@outages.org>
> https://puck.nether.net/mailman/listinfo/outages
[Attachment #3 (text/html)]
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body \
dir="auto"><div style="-webkit-text-size-adjust: auto; ">It appears to have been a \
new issue, but possibly related to the previous one. Level(3) reported the following \
this morning:</div><div style="-webkit-text-size-adjust: auto; "><br></div><div><font \
color="#000000"><span style="-webkit-text-size-adjust: auto; background-color: \
rgba(255, 255, 255, 0);"><a href="x-apple-data-detectors://1" \
x-apple-data-detectors="true" x-apple-data-detectors-type="calendar-event" \
x-apple-data-detectors-result="1">13:53 GMT</a> - The IP NOC reports that \
protocol drops following a scheduled maintenance, GCR <a href="tel:7622319" \
x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" \
x-apple-data-detectors-result="2">7622319</a>, in Detroit, MI impacted IP services. \
The issue existed on a connection between two of the routers in Detroit, MI. The IP \
NOC has disabled that connection to allow traffic to flow through alternate routes \
and eliminate the protocol drops. The IP NOC will continue investigating the faulty \
connection in an attempt to determine the root cause and restore the connection for \
service. At this time, the IP NOC reports that services are up and \
stable</span></font><br><br><span style="-webkit-text-size-adjust: auto;">Sent from \
my iPhone</span></div><div style="-webkit-text-size-adjust: auto; "><br>On Mar 8, \
2014, at 6:36 AM, "Mitch Patterson" <<a \
href="mailto:mitpatterson@gmail.com">mitpatterson@gmail.com</a>> \
wrote:<br><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto; \
"><div><p dir="ltr">They were doing emergency maintance likely related to the other \
night</p> <div class="gmail_quote">On Mar 8, 2014 6:34 AM, "Greg Smythe" \
<<a href="mailto:greg@thesmythes.org">greg@thesmythes.org</a>> wrote:<br \
type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As \
I type this I am seeing Level3 issues in the Detroit area. Been going on since about \
4:30am Eastern. My co-lo provider had to disable peering to get us back and \
running.<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-=-=-=-=-=-=-=-=-=-=--=-=-<u></u><u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Greg \
Smythe<u></u><u></u></span></p> <p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span \
style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span \
style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> \
Outages [mailto:<a href="mailto:outages-bounces@outages.org" \
target="_blank">outages-bounces@outages.org</a>] <b>On Behalf Of </b>Mitch \
Patterson<br> <b>Sent:</b> Friday, March 07, 2014 12:51 AM<br>
<b>To:</b> Bill Wichers<br>
<b>Cc:</b> <a href="mailto:outages@outages.org" \
target="_blank">outages@outages.org</a><br> <b>Subject:</b> Re: [outages] Level 3 \
Detroit issues again<u></u><u></u></span></p> <p \
class="MsoNormal"><u></u> <u></u></p> <div>
<p class="MsoNormal">Just got this<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">*** CASCADED EXTERNAL NOTES 07-Mar-2014 05:48:37 GMT From CASE: \
7621845 - Event The Operations Engineering team has isolated the cause of the packet \
loss failure. A change in parameters on the Detroit's network router has been \
initiated to restore normal operations.<u></u><u></u></p> </div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Mar 7, 2014 at 12:41 AM, Mitch Patterson <<a \
href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> \
wrote:<u></u><u></u></p> <div>
<p class="MsoNormal">Just got this<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><b><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif";background:#f0f0f0">03/07/2014 \
05:06 AM GMT</span></b><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif""><br>
<br>
</span><u></u><u></u></p>
<pre style="line-height:12.95pt;background:#f0f0f0"><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif""><u></u> <u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif""><u></u> <u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif"">*** \
CASCADED EXTERNAL NOTES 07-Mar-2014 05:06:43 GMT From CASE: 7621845 - \
Event<u></u><u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span \
style="font-size:9.0pt;font-family:"Arial","sans-serif"">The \
Technical Services Center (TSC) detected a packet loss condition on a network router \
in Detroit, MI. The Operations Engineering team has been engaged on the issue \
and they are currently analyzing the packet loss condition to determine the cause of \
the failure.<u></u><u></u></span></pre>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Mar 7, 2014 at 12:01 AM, Mitch Patterson <<a \
href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> \
wrote:<u></u><u></u></p> <div>
<p class="MsoNormal">Do you know are the circuits you are on legacy GLBX? from what \
i'm told by our guy who handles the colo the legacy GLBX ones are fine. but the other \
ones are not.<u></u><u></u></p> </div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Mar 6, 2014 at 11:59 PM, Bill Wichers <<a \
href="mailto:billw@waveform.net" target="_blank">billw@waveform.net</a>> \
wrote:<u></u><u></u></p> <p class="MsoNormal">We connect into them in Detroit and we \
don't see this. We aren't in their colo though.<br> <br>
-Bill<br>
<br>
Sent from my iPhone<u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
On Mar 6, 2014, at 11:44 PM, "Mitch Patterson" <<a \
href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> \
wrote:<br> <br>
> We are seeing over 20% packet loss on our circuits at their colo in Southfield. \
They are telling me its been refereed to their IP core team as its a high level \
issue. Anyone else seeing this?<u></u><u></u></p> </div>
<p class="MsoNormal">> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</blockquote></div>
</div></blockquote></body></html>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
--===============1186993100308387727==--
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic