'Re: [Outages-discussion] [outages] IBM Cloud / NetworkLayer Outage'

[prev in list] [next in list] [prev in thread] [next in thread] 

List:       outages-discussion
Subject:    Re: [Outages-discussion] [outages] IBM Cloud / NetworkLayer Outage
From:       Andy Ringsmuth <andy () andyring ! com>
Date:       2020-06-10 15:53:31
Message-ID: 21CA2A8B-F580-4553-891C-7B7298CA3C7F () andyring ! com
[Download RAW message or body]

https://puck.nether.net/mailman/listinfo/outages-discussion

I've received messages as recently as June 7 on -discussion.

----
Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
andy@andyring.com

"Better even die free, than to live slaves." - Frederick Douglas, 1863

> On Jun 10, 2020, at 10:19 AM, Mike Castoro <mcastoro@AdriannaPapell.com> wrote:
> 
> Does anyone actually moderate outages-discuss? I've tried to join multiple times \
> over the years…. 
> Mike Castoro | VP of IT
> Adrianna Papell Group | 646-723-3885
> Chat with me in Teams
> 
> From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages
> Sent: Wednesday, June 10, 2020 10:01 AM
> To: Andy Ringsmuth <andy@andyring.com>
> Cc: Jason Kuehl via Outages <outages@outages.org>
> Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
> 
> External Email!
> 
> Didnt know what was a list.
> 
> On Wed, Jun 10, 2020 at 9:45 AM Andy Ringsmuth via Outages <outages@outages.org> \
> wrote: Could we move further discussion to outages-discuss please?
> 
> ----
> Andy Ringsmuth
> 5609 Harding Drive
> Lincoln, NE 68521-5831
> (402) 304-0083
> andy@andyring.com
> 
> "Better even die free, than to live slaves." - Frederick Douglas, 1863
> 
> > On Jun 10, 2020, at 8:22 AM, Jason Kuehl via Outages <outages@outages.org> wrote:
> > 
> > Someone didn't have proper protections in place it looks like.
> > 
> > On Wed, Jun 10, 2020 at 9:20 AM Mike Bolitho via Outages <outages@outages.org> \
> > wrote: So another BGP highjacking...
> > 
> > - Mike Bolitho
> > 
> > On Wed, Jun 10, 2020, 5:40 AM Greg Dickinson via Outages <outages@outages.org> \
> > wrote: Found this on Reddit as an initial RFO, so keep in mind the source \
> > (although r/sysadmin is usually pretty good): 
> > 
> > 
> > "A 3rd party network provider was advertising routes which resulted in our \
> > WorldWide traffic becoming severely impeded. This led to IBM Cloud clients being \
> > unable to log-in to their accounts, greatly limited internet/DC connectivity and \
> > other significant network route related impacts. Network Specialists have made \
> > adjustments to route policies to restore network access, and alleviate the \
> > impacts. The overall incident lasted from 5:55pm - 9:30pm ET. We will be \
> > providing a fully detailed Customer Incident Report/Root Cause Analysis as soon \
> > as possible" 
> > 
> > 
> > Greg Dickinson
> > 
> > Network Engineer
> > 
> > 
> > 
> > 
> > 
> > 234 Goodwin Crest Drive, Suite 500
> > 
> > Birmingham, AL  35209
> > 
> > Office: 205.917.2407  |  Mobile: 205.234.6427
> > 
> > Fax: 205.945.0515  | 
> > 
> > greg.dickinson@bryantbank.com  |  BryantBank.com
> > 
> > Unbeatable Service.  Legendary Results.
> > 
> > Facebook  |  Twitter
> > 
> > 
> > 
> > From: Outages <outages-bounces@outages.org> On Behalf Of Jason Kuehl via Outages
> > Sent: Tuesday, June 9, 2020 8:38 PM
> > To: James Brown <jbrown@easypost.com>
> > Cc: outages@outages.org
> > Subject: Re: [outages] IBM Cloud / NetworkLayer Outage
> > 
> > 
> > 
> > 
> > 
> > This Message originates from outside Bryant Bank.   Please use caution when \
> > opening this correspondence, attachments or hyperlinks (URLs).  If you have \
> > questions, please contact IT Support.  Thank you. 
> > I can't wait for this post mortem.
> > 
> > 
> > 
> > On Tue, Jun 9, 2020 at 9:21 PM James Brown via Outages <outages@outages.org> \
> > wrote: 
> > As of a few minutes ago, all US SoftLayer (IBM Cloud) locations look normal to \
> > me. 
> > 
> > 
> > On Tue, Jun 9, 2020 at 5:27 PM James Brown <jbrown@easypost.com> wrote:
> > 
> > Just wanted to share updates:
> > 
> > 
> > 
> > I got through to someone at IBM who confirmed the level of chaos over there. It \
> > looks like they've started deploying some mitigations because I'm seeing packet \
> > loss drop off but latency is now awful for all US locations. It looks like IBM \
> > has failed over all of their peering to go through their European facilities \
> > because every return packet from every POP is coming from lon03.networklayer.com \
> > or ams02.networklayer.com (via Telia). 
> > 
> > 
> > On Tue, Jun 9, 2020 at 4:12 PM James Brown <jbrown@easypost.com> wrote:
> > 
> > It looks like IBM Cloud / SoftLayer / NetworkLayer is having a global outage \
> > since about 15:04 Pacific time affecting routing to AWS, Level 3 / CenturyLink, \
> > and a bunch of other providers. Has anyone been able to get more details? 
> > 
> > 
> > --
> > 
> > James Brown
> > 
> > Network Engineer
> > 
> > 
> > 
> > 
> > 
> > --
> > 
> > James Brown
> > 
> > Engineer
> > 
> > 
> > 
> > 
> > 
> > --
> > 
> > James Brown
> > 
> > Engineer
> > 
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> > 
> > 
> > 
> > 
> > 
> > --
> > 
> > Sincerely,
> > 
> > Jason W Kuehl
> > Cell 920-419-8983
> > jason.w.kuehl@gmail.com
> > 
> > 
> > 
> > NOTICE: This electronic mail message and any files transmitted with it are \
> > intended exclusively for the individual or entity to which it is addressed. The \
> > message, together with any attachment, may contain confidential and/or privileged \
> > information. Any unauthorized review, use, print, save, copy, disclosure or \
> > distribution is strictly prohibited. If you have received this message in error, \
> > please immediately advise the sender by reply email and delete copies.  Thank \
> > you. _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> > 
> > 
> > -- 
> > Sincerely,
> > 
> > Jason W Kuehl
> > Cell 920-419-8983
> > jason.w.kuehl@gmail.com
> > _______________________________________________
> > Outages mailing list
> > Outages@outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> 
> _______________________________________________
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
> 
> 
> -- 
> Sincerely,
> 
> Jason W Kuehl
> Cell 920-419-8983
> jason.w.kuehl@gmail.com

_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion

[prev in list] [next in list] [prev in thread] [next in thread] 