[prev in list] [next in list] [prev in thread] [next in thread]
List: outages
Subject: Re: [outages] Level3 issue in Dallas
From: Jason Ackley via Outages <outages () outages ! org>
Date: 2015-07-16 21:06:40
Message-ID: CAA=cXfpA7_0LuPi_chbiGg3rS0=Mj8a+ManeibZOeD9dVo-6tA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Sadly, this is a repeat offender
linecard/platform/customer-edge-device/something-or-other.
This is probably the 3rd time this has taken place for this port for us in
the past 14months?
As a side note - We reported this at 7:45 AM GMT per our ticket timestamp
when it originally took place. Not sure why it takes so long for the IP NOC
to put it together.
If there are any Level3 lurkers - may be an indication of other training or
tool problems?
In our ticket notes - they said there were 4 other ports impacted on the
linecard with down BGP sessions, so I guess I know who else is on the
linecard :) /wave
--
jason
On Thu, Jul 16, 2015 at 3:57 PM, Luke Rockwell via Outages <
outages@outages.org> wrote:
> 7/16/15 8:39:56 PM GMTThe IP NOC is working with the equipment vendor to
> isolate the root cause of the errors.7/16/15 7:54:43 PM GMTNo new
> information is currently available. Updates will be provided as they are
> received.7/16/15 6:56:49 PM GMTInvestigations are ongoing to isolate the
> issue.7/16/15 6:06:45 PM GMTThe IP NOC reports a card failing in Dallas,
> TX impacting IP services. The silent card failure was detected when the
> Technical Service Center began receiving calls from customers. The
> equipment vendor is being engaged for assistance. Please be advised that
> updates for this event will be relayed at a minimum of hourly unless
> otherwise noted.
>
> On Thu, Jul 16, 2015 at 1:51 PM, Mark Price via Outages <
> outages@outages.org> wrote:
>
>>
>> FYI, Level3 has acknowledged the issue but no ETR yet.
>>
>>
>> "The event, 9470903, reports a card failing in Dallas, TX impacting IP
>> services. The silent card failure was detected when the Technical Service
>> Center began receiving calls from customers. "
>>
>>
>>
>> On Thu, Jul 16, 2015 at 3:03 AM, Mark Price <mprice@tqhosting.com> wrote:
>>
>>>
>>> Starting at approx 06:00 UTC we had a strange issue with TCP/UDP traffic
>>> dying on Level3 in Dallas, although ICMP looked clean. For now we've
>>> stopped sending traffic to Level3 in Dallas.
>>>
>>> Anyone else seeing similar issues?
>>>
>>>
>>> -mark
>>>
>>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
[Attachment #5 (text/html)]
<div dir="ltr"><div><br></div><br><div>Sadly, this is a repeat offender \
linecard/platform/customer-edge-device/something-or-other. \
</div><div><br></div><div>This is probably the 3rd time this has taken place for this \
port for us in the past 14months?</div><div><br></div><div>As a side note - We \
reported this at 7:45 AM GMT per our ticket timestamp when it originally took place. \
Not sure why it takes so long for the IP NOC to put it \
together.</div><div><br></div><div>If there are any Level3 lurkers - may be an \
indication of other training or tool problems?</div><div><br></div><div>In our ticket \
notes - they said there were 4 other ports impacted on the linecard with down BGP \
sessions, so I guess I know who else is on the linecard :) \
/wave</div><div><br></div><div><br></div><div><br></div><div>--</div><div>jason</div><div><br></div><div><br></div><div><br></div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 16, 2015 at 3:57 PM, \
Luke Rockwell via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" \
target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><table cellspacing="1" cellpadding="2" \
width="99%" style="list-style-type:none;margin:0px;padding:0px;border-collapse:collapse;border-spacing:0px;border:1px \
solid rgb(0,0,0);color:rgb(51,51,51);font-family:Arial;font-size:12px;background:rgb(243,243,243)"><tbody \
style="list-style-type:none;margin:0px;padding:0px"><tr \
style="list-style-type:none;margin:0px;padding:0px"><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">7/16/15 \
8:39:56 PM GMT</td><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">The IP NOC \
is working with the equipment vendor to isolate the root cause of the \
errors.</td></tr><tr style="list-style-type:none;margin:0px;padding:0px"><td \
valign="top" style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">7/16/15 \
7:54:43 PM GMT</td><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">No new \
information is currently available. Updates will be provided as they are \
received.</td></tr><tr style="list-style-type:none;margin:0px;padding:0px"><td \
valign="top" style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">7/16/15 \
6:56:49 PM GMT</td><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(255,255,255)">Investigations \
are ongoing to isolate the issue.</td></tr><tr \
style="list-style-type:none;margin:0px;padding:0px"><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">7/16/15 \
6:06:45 PM GMT</td><td valign="top" \
style="list-style-type:none;padding:2px;background-color:rgb(238,238,238)">The IP NOC \
reports a card failing in Dallas, TX impacting IP services. The silent card failure \
was detected when the Technical Service Center began receiving calls from customers. \
The equipment vendor is being engaged for assistance. Please be advised that updates \
for this event will be relayed at a minimum of hourly unless otherwise \
noted.</td></tr></tbody></table><div class="gmail_extra"><br><div \
class="gmail_quote"><div><div class="h5">On Thu, Jul 16, 2015 at 1:51 PM, Mark Price \
via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" \
target="_blank">outages@outages.org</a>></span> wrote:<br></div></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div><div class="h5"><div dir="ltr"><br><div>FYI, Level3 has \
acknowledged the issue but no ETR yet.</div><div><br></div><div><br></div><div><span \
style="font-size:12.8000001907349px">"The event, 9470903, reports a card failing \
in Dallas, TX impacting IP services. The silent card failure was detected when the \
Technical Service Center began receiving calls from customers. \
"</span><br></div><div><span \
style="font-size:12.8000001907349px"><br></span></div><div><span \
style="font-size:12.8000001907349px"><br></span></div></div><div><div><div \
class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 16, 2015 at 3:03 AM, \
Mark Price <span dir="ltr"><<a href="mailto:mprice@tqhosting.com" \
target="_blank">mprice@tqhosting.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div dir="ltr"><br><div>Starting at approx 06:00 UTC we had a \
strange issue with TCP/UDP traffic dying on Level3 in Dallas, although ICMP looked \
clean. For now we've stopped sending traffic to Level3 in \
Dallas.</div><div><br></div><div>Anyone else seeing similar issues?</div><span><font \
color="#888888"><div><br></div><div><br></div><div>-mark</div><div><br></div></font></span></div>
</blockquote></div><br></div>
</div></div><br></div></div><span \
class="">_______________________________________________<br> Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br> \
<br></span></blockquote></div><br><br clear="all"><div><br></div> </div></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br> \
<br></blockquote></div><br></div>
_______________________________________________
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic