[prev in list] [next in list] [prev in thread] [next in thread]
List: outages-discussion
Subject: Re: [Outages-discussion] [outages] Second Bandwidth Outage Sunday
From: "Aryn Nakaoka 808.356.2901" <anakaoka () trinet-hi ! com>
Date: 2021-09-27 19:09:28
Message-ID: CAEMZvjg4+0+m-mHM-jKPhgizLDUccK3jqi9ncm+BMCmYTDFNDw () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/related)]
[Attachment #4 (multipart/alternative)]
Typically 3 :
https://bgp.he.net/AS12130#_graph4
- NTT
- L3
- Telia
[image: Screen Shot 2021-09-27 at 9.09.19 AM.png]
Aryn Nakaoka
anakaoka@trinet-hi.com
Direct: 808.356.2901
518 Holokahana Lane
Honolulu, Hi 96817
AlohaTone Mobile: https://youtu.be/PdUyuf0hTYY
A Better Solution https://www.trinet-hi.com/abettersolution.pdf
<https://www.trinet-hi.com/abettersolution.pdf>
Aloha Tone PBX https://www.youtube.com/watch?v=96YWPY9wCeU
CONFIDENTIALITY NOTICE: The information contained in this email and any
attachments may be privileged, confidential and protected from disclosure.
Any disclosure, distribution or copying of this email or any attachments by
persons or entities other than the intended recipient is prohibited. If you
have received this email in error, please notify the sender immediately by
replying to the message and deleting this email and any attachments from
your system. Thank you for your cooperation.
On Mon, Sep 27, 2021 at 9:04 AM Charles Sprickman <spork@bway.net> wrote:
>
>
> On Sep 27, 2021, at 2:31 PM, Jared Geiger <jared@compuwizz.net> wrote:
>
> It looks like Bandwidth dropped all their BGP peers except NTT, presumably
> to use NTT's DDoS services.
>
>
> How many do they typically have?
>
>
> As far as peering, Bandwidth offers connectivity via Megaport. Peerless
> (21808) is on a bunch of Equinix exchanges https://peeringdb.com/net/16256.
> Inteliquent (40160) is on some exchanges as well
> https://peeringdb.com/net/18236.
>
> I did try and get Peerless to join Coresite Any2East last year because
> they are in the same building but they said no.
>
> On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <
> douglas.vought@themsgctr.com> wrote:
>
>> They are. Several customers have contacted me demanding that I plug the
>> phone line back in.
>> On 9/27/2021 13:54, Drew Matthews via Outages wrote:
>>
>> i believe they are having issues again, not sure if its ddos related. but
>> they are experiencing some major issues. https://status.bandwidth.com/
>>
>> Update - Bandwidth teams are also investigating delays with Toll-Free
>> Messaging at this time along with V2 Voice, Standard Voice, 911 traffic,
>> Portal & API with intermittent service disruptions.
>> Sep 27, 13:01 EDT
>> Update - We continue to investigate this incident.
>> Sep 27, 11:57 EDT
>> Update - Bandwidth teams are also investigating delayed ports at this
>> time along with V2 Voice, Standard Voice, 911 traffic, Portal & API with
>> intermittent service disruptions.
>> Sep 27, 11:07 EDT
>> Update - We continue to investigate this incidnet.
>> Sep 27, 11:02 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice,
>> V2 Voice, 911 traffic, Portal & API with intermittent service disruption.
>> Sep 27, 10:00 EDT
>> Update - Bandwidth teams are investigating an incident impacting Voice,
>> V2 Voice, and 911 traffic with intermittent service disruption.
>> Sep 27, 09:42 EDT
>> Investigating - Bandwidth teams are investigating an incident impacting
>> our V2 Voice services. Customers may notice intermittent service
>> disruptions.
>> Sep 27, 09:12 EDT
>>
>> On Sun, Sep 26, 2021 at 3:13 PM Peter Beckman via Outages <
>> outages@outages.org> wrote:
>>
>>> Again today, on the heels of the 2h 47m DDoS-related outage yesterday,
>>> right at
>>> 11:01am US EDT (15:01:28 UTC), our monitoring alerted us to another
>>> incident at
>>> Bandwidth. The impact to us was much reduced this time, as we saw some
>>> flapping
>>> of their SIP Inbound SBCs, but around 15:20:12 UTC it was responsive
>>> enough to
>>> fall below our thresholds.
>>>
>>> The API started acting up at 14:56:00 UTC, but recovered, and then has
>>> been
>>> flapping between OK and degraded since then. As of 21:52 UTC, the API and
>>> Portal/Dashboard are either non-responsive, slow, or degraded.
>>>
>>> Inbound Voice Traffic has remaind at expected historical levels
>>> throughout.
>>> I have not checked on Termination, but it was working earlier today
>>> without
>>> issue.
>>>
>>> Latest from BW https://status.bandwidth.com/
>>>
>>> Update - Bandwidth teams have observed expected call behavior for
>>> 911
>>> calls. 911 calls are processing as designed and are fully restored.
>>> Teams
>>> continue to work on standard voice traffic.
>>>
>>> Sep 26, 17:20 EDT
>>>
>>> I question the phrase "processing as designed" which implies it is not
>>> working as the end user expects... odd.
>>>
>>> Beckman
>>>
>>> ---------------------------------------------------------------------------
>>> Peter Beckman Internet
>>> Guy
>>> beckman@angryox.com
>>> https://www.angryox.com/
>>>
>>> ---------------------------------------------------------------------------
>>> _______________________________________________
>>> Outages mailing list
>>> Outages@outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>
>> _______________________________________________
>> Outages mailing listOutages@outages.orghttps://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages-discussion mailing list
>> Outages-discussion@outages.org
>> https://puck.nether.net/mailman/listinfo/outages-discussion
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
[Attachment #7 (text/html)]
<div dir="ltr">Typically 3 : <div><br></div><div><a \
href="https://bgp.he.net/AS12130#_graph4">https://bgp.he.net/AS12130#_graph4</a></div><div><br></div><div>- \
NTT</div><div>- L3</div><div>- Telia</div><div><br></div><div><img \
src="cid:ii_ku30yjtz0" alt="Screen Shot 2021-09-27 at 9.09.19 AM.png" width="562" \
height="131"><br></div><div><br></div><div><div><div dir="ltr" \
class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div><font size="2"><br></font></div><div><font \
size="2"><br></font></div><div><font size="2">Aryn Nakaoka<br><a \
href="mailto:anakaoka@trinet-hi.com" \
target="_blank">anakaoka@trinet-hi.com</a><br>Direct: \
808.356.2901<br></font></div><div><font size="2">518 Holokahana \
Lane</font></div><div>Honolulu, Hi<font size="2"> \
96817<br></font></div><div><div><div><font size="2"><br></font></div><font \
size="2">AlohaTone Mobile: <a href="https://youtu.be/PdUyuf0hTYY" \
target="_blank">https://youtu.be/PdUyuf0hTYY</a></font></div><div><font \
size="2"><br></font></div><div><font size="2">A Better Solution <a \
href="https://www.trinet-hi.com/abettersolution.pdf" \
target="_blank">https://www.trinet-hi.com/abettersolution.pdf \
<br></a></font><div><font size="2">Aloha Tone PBX <a \
href="https://www.youtube.com/watch?v=96YWPY9wCeU" \
target="_blank">https://www.youtube.com/watch?v=96YWPY9wCeU</a><br></font></div><div><font \
size="2"><br></font></div></div><div><div><font size="2">CONFIDENTIALITY NOTICE: \
The information contained in this email and any attachments may be privileged, \
confidential and protected from disclosure. Any disclosure, distribution or copying \
of this email or any attachments by persons or entities other than the intended \
recipient is prohibited. If you have received this email in error, please notify the \
sender immediately by replying to the message and deleting this email and any \
attachments from your system. Thank you for your \
cooperation.<br></font></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div><br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 27, 2021 at 9:04 AM \
Charles Sprickman <<a href="mailto:spork@bway.net">spork@bway.net</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
style="overflow-wrap: break-word;"><br><div><br><blockquote type="cite"><div>On Sep \
27, 2021, at 2:31 PM, Jared Geiger <<a href="mailto:jared@compuwizz.net" \
target="_blank">jared@compuwizz.net</a>> wrote:</div><br><div><div dir="ltr">It \
looks like Bandwidth dropped all their BGP peers except NTT, presumably to use \
NTT's DDoS services.</div></div></blockquote><div><br></div><div>How many do they \
typically have?</div><br><blockquote type="cite"><div><div \
dir="ltr"><div><br></div><div>As far as peering, Bandwidth offers connectivity via \
Megaport. Peerless (21808) is on a bunch of Equinix exchanges <a \
href="https://peeringdb.com/net/16256" \
target="_blank">https://peeringdb.com/net/16256</a>. Inteliquent (40160) is on some \
exchanges as well <a href="https://peeringdb.com/net/18236" \
target="_blank">https://peeringdb.com/net/18236</a>. </div><div><br></div><div>I did \
try and get Peerless to join Coresite Any2East last year because they are in the same \
building but they said no.</div></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Sep 27, 2021 at 11:19 AM Douglas Vought <<a \
href="mailto:douglas.vought@themsgctr.com" \
target="_blank">douglas.vought@themsgctr.com</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">
<div><p>They are. Several customers have contacted me demanding that I
plug the phone line back in. <br>
</p>
<div>On 9/27/2021 13:54, Drew Matthews via
Outages wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">i believe they are having issues again, not sure if
its ddos related. but they are experiencing some major issues. <a \
href="https://status.bandwidth.com/" \
target="_blank">https://status.bandwidth.com/</a> <div><br>
</div>
<div>
<div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
- Bandwidth teams are also investigating delays with Toll-Free
Messaging at this time along with V2 Voice, Standard Voice,
911 traffic, Portal & API with intermittent service
disruptions.<br style="box-sizing:border-box">
<small style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outl \
ine:0px;vertical-align:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">13:01</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
-
We continue to investigate this incident.<br \
style="box-sizing:border-box"> <small \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-al \
ign:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">11:57</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
- Bandwidth teams are also investigating delayed ports at this
time along with V2 Voice, Standard Voice, 911 traffic,
Portal & API with intermittent service disruptions.<br \
style="box-sizing:border-box"> <small \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-al \
ign:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">11:07</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
-
We continue to investigate this incidnet.<br \
style="box-sizing:border-box"> <small \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-al \
ign:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">11:02</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
- Bandwidth teams are investigating an incident impacting
Voice, V2 Voice, 911 traffic, Portal & API with
intermittent service disruption.<br style="box-sizing:border-box">
<small style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outl \
ine:0px;vertical-align:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">10:00</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Update</span> \
- Bandwidth teams are investigating an incident impacting
Voice, V2 Voice, and 911 traffic with intermittent service
disruption.<br style="box-sizing:border-box">
<small style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outl \
ine:0px;vertical-align:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">09:42</var> \
EDT</small></div> <div><span \
style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-weight:700">Investigating</span> \
- Bandwidth teams are investigating an incident impacting our
V2 Voice services. Customers may notice intermittent service
disruptions.<br style="box-sizing:border-box">
<small style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outl \
ine:0px;vertical-align:baseline;background:transparent;font-size:0.875rem;line-height:1.33438rem">Sep \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">27</var>, \
<var style="box-sizing:border-box;margin:0px;padding:0px;border:0px;outline:0px;vertical-align:baseline;background:transparent;font-style:normal">09:12</var> \
EDT</small></div> </div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Sun, Sep 26, 2021 at 3:13
PM Peter Beckman via Outages <<a href="mailto:outages@outages.org" \
target="_blank">outages@outages.org</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Again today, on the \
heels of the 2h 47m DDoS-related outage yesterday, right at<br>
11:01am US EDT (15:01:28 UTC), our monitoring alerted us to
another incident at<br>
Bandwidth. The impact to us was much reduced this time, as we
saw some flapping<br>
of their SIP Inbound SBCs, but around 15:20:12 UTC it was
responsive enough to<br>
fall below our thresholds.<br>
<br>
The API started acting up at 14:56:00 UTC, but recovered, and
then has been<br>
flapping between OK and degraded since then. As of 21:52 UTC,
the API and<br>
Portal/Dashboard are either non-responsive, slow, or degraded.<br>
<br>
Inbound Voice Traffic has remaind at expected historical
levels throughout.<br>
I have not checked on Termination, but it was working earlier
today without<br>
issue.<br>
<br>
Latest from BW <a href="https://status.bandwidth.com/" rel="noreferrer" \
target="_blank">https://status.bandwidth.com/</a><br> <br>
Update - Bandwidth teams have observed expected call
behavior for 911<br>
calls. 911 calls are processing as designed and are fully
restored. Teams<br>
continue to work on standard voice traffic.<br>
<br>
Sep 26, 17:20 EDT<br>
<br>
I question the phrase "processing as designed" which implies
it is not<br>
working as the end user expects... odd.<br>
<br>
Beckman<br>
---------------------------------------------------------------------------<br>
Peter Beckman \
Internet Guy<br>
<a href="mailto:beckman@angryox.com" \
target="_blank">beckman@angryox.com</a>
<a href="https://www.angryox.com/" \
rel="noreferrer" target="_blank">https://www.angryox.com/</a><br>
---------------------------------------------------------------------------<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" \
target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br> \
</blockquote> </div>
<br>
<fieldset></fieldset>
<pre>_______________________________________________
Outages mailing list
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>
<a href="https://puck.nether.net/mailman/listinfo/outages" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages</a> </pre>
</blockquote>
</div>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" \
target="_blank">Outages-discussion@outages.org</a><br> <a \
href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br> \
</blockquote></div> _______________________________________________<br>Outages-discussion \
mailing list<br><a href="mailto:Outages-discussion@outages.org" \
target="_blank">Outages-discussion@outages.org</a><br><a \
href="https://puck.nether.net/mailman/listinfo/outages-discussion" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br></d \
iv></blockquote></div><br></div>_______________________________________________<br> \
Outages-discussion mailing list<br> <a href="mailto:Outages-discussion@outages.org" \
target="_blank">Outages-discussion@outages.org</a><br> <a \
href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br> \
</blockquote></div>
["Screen Shot 2021-09-27 at 9.09.19 AM.png" (image/png)]
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic