[prev in list] [next in list] [prev in thread] [next in thread]
List: outages-discussion
Subject: Re: [Outages-discussion] [outages] Problem with credit card machine processing? "Datawire"
From: <frnkblk () iname ! com>
Date: 2018-08-05 5:06:52
Message-ID: 000e01d42c7a$20afc750$620f55f0$ () iname ! com
[Download RAW message or body]
This is a multipart message in MIME format.
[Attachment #2 (multipart/alternative)]
This is a multipart message in MIME format.
Thanks for sharing – hopefully DNSsec is being turned on soon for their zones. \
That, in combination with making sure the terminals use at least one validating \
DNSsec resolver (if not their ISPs resolver, a dynamic list of other resolvers), \
would have minimized the number of terminals from contacting the wrong data center.
Frank
From: Chris <chris@vnworks.net>
Sent: Saturday, August 4, 2018 11:25 PM
To: frnkblk@iname.com
Cc: outages-discussion@outages.org
Subject: Re: [Outages-discussion] [outages] Problem with credit card machine \
processing? "Datawire"
Yup.
There's nothing like a little passively gathered evidence...
c/o Farsight's passive DNS service.
;; bailiwick: datawire.net <http://datawire.net> .
;; count: 4
;; first seen: 2018-07-10 23:44:11 -0000
;; last seen: 2018-07-13 00:55:39 -0000
vxn.datawire.net <http://vxn.datawire.net> . IN A 45.227.252.17
;; bailiwick: datawire.net <http://datawire.net> .
;; count: 5
;; first seen: 2018-07-10 23:44:12 -0000
;; last seen: 2018-07-13 00:57:51 -0000
vxn1.datawire.net <http://vxn1.datawire.net> . IN A 45.227.252.17
;; bailiwick: datawire.net <http://datawire.net> .
;; count: 2
;; first seen: 2018-07-10 23:44:12 -0000
;; last seen: 2018-07-10 23:44:12 -0000
vxn2.datawire.net <http://vxn2.datawire.net> . IN A 45.227.252.17
;; bailiwick: datawire.net <http://datawire.net> .
;; count: 6
;; first seen: 2018-07-10 23:44:13 -0000
;; last seen: 2018-07-13 00:56:07 -0000
vxn3.datawire.net <http://vxn3.datawire.net> . IN A 45.227.252.17
;; bailiwick: datawire.net <http://datawire.net> .
;; count: 9
;; first seen: 2018-07-10 23:44:14 -0000
;; last seen: 2018-07-13 00:49:06 -0000
vxn4.datawire.net <http://vxn4.datawire.net> . IN A 45.227.252.17
prod.ssl53.com <http://prod.ssl53.com> . IN A 45.227.252.17
vxn.datawire.net <http://vxn.datawire.net> . IN A 45.227.252.17
vxn1.datawire.net <http://vxn1.datawire.net> . IN A 45.227.252.17
vxn2.datawire.net <http://vxn2.datawire.net> . IN A 45.227.252.17
vxn3.datawire.net <http://vxn3.datawire.net> . IN A 45.227.252.17
vxn4.datawire.net <http://vxn4.datawire.net> . IN A 45.227.252.17
;;; Returned 6 RRs in 0.02 seconds.
;;; DNSDB
Data wire is First Data, however, that outlier above....
;; bailiwick: ssl53.com <http://ssl53.com> .
;; count: 17
;; first seen: 2018-07-13 00:48:09 -0000
;; last seen: 2018-07-13 01:00:26 -0000
prod.ssl53.com <http://prod.ssl53.com> . IN A 45.227.252.17
= Vantiv, LLC, isn't that Worldpay?
Hmmm.
On 5 Aug 2018, at 11:56, <frnkblk@iname.com <mailto:frnkblk@iname.com> > \
<frnkblk@iname.com <mailto:frnkblk@iname.com> > wrote:
Looks like Datawire did sweep it under the rug – here's a Dyn blog written by Doug \
Madory about how the IP address space for Datawire's nameservers were hijacked for a \
short time:
https://dyn.com/blog/bgp-dns-hijacks-target-payment-systems/
The July 10 incident would be Tuesday afternoon/early evening in the U.S.
Now its' very clear why the payment processors wanted ISPs to flush Datawire's host \
entries.
Frank
From: Outages-discussion <outages-discussion-bounces@outages.org \
<mailto:outages-discussion-bounces@outages.org> > On Behalf Of Frank \
Bulk
Sent: Tuesday, July 17, 2018 3:41 PM
To: outages-discussion@outages.org <mailto:outages-discussion@outages.org>
Subject: Re: [Outages-discussion] [outages] Problem with credit card machine \
processing? "Datawire"
I had assumed that the VPS provider was their DR solution. =)
Frank
From: Randy McAnally <rsm@fast-serv.com <mailto:rsm@fast-serv.com> >
Sent: Tuesday, July 17, 2018 3:07 PM
To: Frank Bulk <frnkblk@iname.com <mailto:frnkblk@iname.com> >
Cc: outages-discussion@outages.org <mailto:outages-discussion@outages.org>
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
45.227.252.17 + high TTL + ukraine VPS provider
did first data just sweep this under the rug?
On 07/16/2018 12:52 pm, Frank Bulk via Outages wrote:
Just received this afternoon:
==================
Support Team,
You have several business customers being affected by an ongoing issue. In order to \
resolve this, First Data is requesting that you clear the cache on all DNS servers \
being used to support them. We propagated a correction over 16 hours ago and know \
that Google DNS and others are translating correctly. Would you please help us assist \
your customers?
The correct resolutions are:
vxn.datawire.net <http://vxn.datawire.net/> 216.220.36.75
vxn1.datawire.net <http://vxn1.datawire.net/> 205.167.140.10
vxn2.datawire.net <http://vxn2.datawire.net/> 64.243.142.36
vxn3.datawire.net <http://vxn3.datawire.net/> 206.112.91.167
vxn4.datawire.net <http://vxn4.datawire.net/> 63.240.199.76
If you are resolving it as anything starting with 45.x.x.x, it is incorrect. Please \
feel free to compare to the Google DNS resolution for confirmation.
Please either reply all or call First Data's Network Operations at 888-377-8726 \
Option 3.
<snip>
First Data, 240 North Roosevelt Av
Chandler, Arizona 85226
==================
That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too long – \
if they had been short, like they are now at 300 seconds, the issue would mostly have \
cleared up.
From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On \
Behalf Of frnkblk--- via Outages
Sent: Friday, July 13, 2018 9:56 PM
To: 'Luke Guillory' <lguillory@reservetele.com <mailto:lguillory@reservetele.com> >; \
jayson@peakinter.net <mailto:jayson@peakinter.net> ; outages@outages.org \
<mailto:outages@outages.org>
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
Yes, we learned of issues late Wednesday morning after receiving reports from two and \
then three business customers. Indications suggest the issue started Tuesday \
evening. One local Dairy Queen and another 20 minutes away couldn't accept credit \
cards on Wednesday.
The request to preform DNS flushes of vxn.datawire.net <http://vxn.datawire.net> \
came to us Thursday afternoon from two of three customers who (eventually) called \
their credit card partners/processors. So we flushed our (ISP) caches and then \
encouraged those customers to power cycle their router and then their credit card \
machines, but that wasn't 100% successful for them, either. At that point we \
directed them back to their credit card partners/processors. It was interesting to \
see DNS resolution for vxn.datawire.net <http://vxn.datawire.net> pointing to a \
mixture of 216.220.36.75 (vxn.datawire.net <http://vxn.datawire.net> ) and \
45.227.252.17 (hosting-by.net4web.org <http://hosting-by.net4web.org> ). Maybe it's \
normal that they have multiple, but on Wednesday it was just 216.220.36.75. The TTL \
for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 300 \
seconds) and had a bad SSL certificate for https://vxn.datawire.net. I suspect they \
moved some operations to another data center, but made a mistake with TTL.
All told we probably heard from six or seven different businesses.
More here:
https://twitter.com/ExecPro/status/1016860164983611392
https://status.cayan.com/issues/5b45477e8dc35afae9000fe6
https://status.cayan.com/issues/5b4546508dc35a5975000fdc
https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7
https://status.cayan.com/issues/5b478b918dc35aff310030c9
https://twitter.com/TriphenTech/status/1016852856408690693
https://twitter.com/C_Forrest/status/1017819893704593410
https://twitter.com/Vicinity_7/status/1017800989347401728
https://twitter.com/pokehbar/status/1017796090052128769
https://twitter.com/glyngh/status/1017790958610493440
https://twitter.com/tallbaby21/status/1017121159526133760
https://twitter.com/devin_ledude/status/1017451556000522241
https://status.cayan.com/issues/5b478ba38dc35a3da80030d9
Frank
# whob 216.220.36.75
IP: 216.220.36.75
Origin-AS: 12188
Prefix: 216.220.32.0/20
AS-Path: 18106 6939 12188
AS-Org-Name: Q9 Networks Inc.
Org-Name: Q9 Networks Inc.
Net-Name: Q9-NET1
Cache-Date: 1531374425
Latitude: 43.508330
Longitude: -79.883333
City: Milton
Region: Ontario
Country: Canada
Country-Code: CA
# whob 45.227.252.17
IP: 45.227.252.17
Origin-AS: 58271
Prefix: 45.227.252.0/24
AS-Path: 34224 12389 44125 201765 48882 58271
AS-Org-Name: VSERVER-AS
Org-Name: This network range is not fully allocated to APNIC.
Net-Name: IANA-NETBLOCK-45
Cache-Date: 1531374425
Latitude: 0.000000
Longitude: 0.000000
City: NULL
Region: NULL
Country: NULL
Country-Code: NULL
From: Outages <outages-bounces@outages.org <mailto:outages-bounces@outages.org> > On \
Behalf Of Luke Guillory via Outages
Sent: Friday, July 13, 2018 9:18 PM
To: jayson@peakinter.net <mailto:jayson@peakinter.net> ; outages@outages.org \
<mailto:outages@outages.org>
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"
We had a customer call saying they needed is to clear dns cache because they couldn't \
process CCs.
One of my guys read about the large outage so when it came in we knew it wasn't \
anything to do with us.
Sent from my iPhone
On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages <outages@outages.org \
<mailto:outages@outages.org> > wrote:
Our folks have spent the better part of a day chasing an issue with a customer that \
had issues processing cards from their physical in-store terminal. That turned into \
2, 3, and a handful more.
We finally got info that all of these impacted terminals connect to a company \
"Datawire" who went down last night at 1800 and came back up at 0800 this morning \
(unknown TZ). They continued to point to us as the issue until just a short while \
ago when some person at this Datawire admitted a large portion of the country may \
still be down.
Anyone else seeing anything like this? Perhaps it could save you chasing your tail \
as well.
Perhaps better for a discussions-list conversation, but... seriously... a credit card \
processing firm that has an outage like this? Hmm...
Jayson
Peak Internet
Luke Guillory
Vice President – Technology and Innovation
<http://www.rtconline.com>
Tel:
985.536.1212
Fax:
985.536.0300
Email:
lguillory@reservetele.com <mailto:lguillory@reservetele.com>
Web:
www.rtconline.com <http://www.rtconline.com>
Reserve Telecommunications
100 RTC Dr
Reserve, LA 70084
Disclaimer:
The information transmitted, including attachments, is intended only for the \
person(s) or entity to which it is addressed and may contain confidential and/or \
privileged material which should not disseminate, distribute or be copied. Please \
notify Luke Guillory immediately by e-mail if you have received this e-mail by \
mistake and delete this e-mail from your system. E-mail transmission cannot be \
guaranteed to be secure or error-free as information could be intercepted, corrupted, \
lost, destroyed, arrive late or incomplete, or contain viruses. Luke Guillory \
therefore does not accept liability for any errors or omissions in the contents of \
this message, which arise as a result of e-mail transmission.
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages@outages.org <mailto:Outages@outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org <mailto:Outages-discussion@outages.org>
https://puck.nether.net/mailman/listinfo/outages-discussion
[Attachment #5 (text/html)]
<html xmlns:v="urn:schemas-microsoft-com:vml" \
xmlns:o="urn:schemas-microsoft-com:office:office" \
xmlns:w="urn:schemas-microsoft-com:office:word" \
xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" \
xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type \
content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 \
(filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);} o\:* \
{behavior:url(#default#VML);} w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
p.imprintuniqueid, li.imprintuniqueid, div.imprintuniqueid
{mso-style-name:imprintuniqueid;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div \
class=WordSection1><p class=MsoNormal>Thanks for sharing – hopefully DNSsec is \
being turned on soon for their zones. That, in combination with making sure the \
terminals use at least one validating DNSsec resolver (if not their ISPs resolver, a \
dynamic list of other resolvers), would have minimized the number of terminals from \
contacting the wrong data center.<o:p></o:p></p><p \
class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank <o:p></o:p></p><p \
class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid \
#E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Chris \
<chris@vnworks.net> <br><b>Sent:</b> Saturday, August 4, 2018 11:25 \
PM<br><b>To:</b> frnkblk@iname.com<br><b>Cc:</b> \
outages-discussion@outages.org<br><b>Subject:</b> Re: [Outages-discussion] [outages] \
Problem with credit card machine processing? \
"Datawire"<o:p></o:p></p></div></div><p \
class=MsoNormal><o:p> </o:p></p><div><p \
class=MsoNormal>Yup. <o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>There's nothing \
like a little passively gathered evidence...<o:p></o:p></p></div><div><p \
class=MsoNormal>c/o Farsight's passive DNS service.<o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://datawire.net">datawire.net</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 4<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-10 23:44:11 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-13 00:55:39 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn.datawire.net">vxn.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://datawire.net">datawire.net</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 5<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-10 23:44:12 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-13 00:57:51 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn1.datawire.net">vxn1.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://datawire.net">datawire.net</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 2<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-10 23:44:12 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-10 23:44:12 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn2.datawire.net">vxn2.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://datawire.net">datawire.net</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 6<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-10 23:44:13 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-13 00:56:07 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn3.datawire.net">vxn3.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://datawire.net">datawire.net</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 9<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-10 23:44:14 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-13 00:49:06 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn4.datawire.net">vxn4.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal><a \
href="http://prod.ssl53.com">prod.ssl53.com</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn.datawire.net">vxn.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn1.datawire.net">vxn1.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn2.datawire.net">vxn2.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn3.datawire.net">vxn3.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://vxn4.datawire.net">vxn4.datawire.net</a>. IN A \
45.227.252.17<o:p></o:p></p></div><div><p class=MsoNormal>;;; Returned 6 RRs in 0.02 \
seconds.<o:p></o:p></p></div><div><p class=MsoNormal>;;; \
DNSDB<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Data wire \
is First Data, however, that outlier \
above.... <o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><div><p class=MsoNormal>;; \
bailiwick: <a href="http://ssl53.com">ssl53.com</a>.<o:p></o:p></p></div><div><p \
class=MsoNormal>;; count: 17<o:p></o:p></p></div><div><p \
class=MsoNormal>;; first seen: 2018-07-13 00:48:09 -0000<o:p></o:p></p></div><div><p \
class=MsoNormal>;; last seen: 2018-07-13 01:00:26 \
-0000<o:p></o:p></p></div><div><p class=MsoNormal><a \
href="http://prod.ssl53.com">prod.ssl53.com</a>. IN A \
45.227.252.17<o:p></o:p></p></div></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><div><p \
class=MsoNormal>= Vantiv, LLC, isn't that \
Worldpay?<o:p></o:p></p></div><div><p \
class=MsoNormal><br><br><o:p></o:p></p></div><div><p \
class=MsoNormal>Hmmm. <o:p></o:p></p></div><div><p \
class=MsoNormal><o:p> </o:p></p></div><p class=MsoNormal \
style='margin-bottom:12.0pt'>On 5 Aug 2018, at 11:56, <<a \
href="mailto:frnkblk@iname.com">frnkblk@iname.com</a>> <<a \
href="mailto:frnkblk@iname.com">frnkblk@iname.com</a>> \
wrote:<o:p></o:p></p></div><blockquote \
style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>Looks like \
Datawire did sweep it under the rug – here's a Dyn blog written by Doug Madory \
about how the IP address space for Datawire's nameservers were hijacked for a short \
time:<o:p></o:p></p><p class=MsoNormal><a \
href="https://dyn.com/blog/bgp-dns-hijacks-target-payment-systems/">https://dyn.com/blog/bgp-dns-hijacks-target-payment-systems/</a><o:p></o:p></p><p \
class=MsoNormal>The July 10 incident would be Tuesday afternoon/early evening in the \
U.S.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Now \
its' very clear why the payment processors wanted ISPs to flush Datawire's host \
entries.<o:p></o:p></p><p class=MsoNormal> <o:p></o:p></p><p \
class=MsoNormal>Frank <o:p></o:p></p><p \
class=MsoNormal> <o:p></o:p></p><div><div style='border:none;border-top:solid \
#E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> \
Outages-discussion <<a \
href="mailto:outages-discussion-bounces@outages.org">outages-discussion-bounces@outages.org</a>> \
<b>On Behalf Of </b>Frank Bulk<br><b>Sent:</b> Tuesday, July 17, 2018 3:41 \
PM<br><b>To:</b> <a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a><br><b>Subject:</b> \
Re: [Outages-discussion] [outages] Problem with credit card machine processing? \
"Datawire"<o:p></o:p></p></div></div><p \
class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>I had assumed that the VPS \
provider was their DR solution. =)<o:p></o:p></p><p \
class=MsoNormal> <o:p></o:p></p><p class=MsoNormal>Frank <o:p></o:p></p><p \
class=MsoNormal> <o:p></o:p></p><div><div style='border:none;border-top:solid \
#E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Randy \
McAnally <<a href="mailto:rsm@fast-serv.com">rsm@fast-serv.com</a>> \
<br><b>Sent:</b> Tuesday, July 17, 2018 3:07 PM<br><b>To:</b> Frank Bulk <<a \
href="mailto:frnkblk@iname.com">frnkblk@iname.com</a>><br><b>Cc:</b> <a \
href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a><br><b>Subject:</b> \
Re: [outages] Problem with credit card machine processing? \
"Datawire"<o:p></o:p></p></div></div><p \
class=MsoNormal> <o:p></o:p></p><p><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>45.227.252.17 + high TTL + \
ukraine VPS provider</span><o:p></o:p></p><p><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>did first data just sweep \
this under the rug?</span><o:p></o:p></p><p><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> </span><o:p></o:p></p><p><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>On 07/16/2018 12:52 pm, \
Frank Bulk via Outages wrote:</span><o:p></o:p></p><blockquote \
style='border:none;border-left:solid #1010FF 1.5pt;padding:0in 0in 0in \
5.0pt;margin-left:0in;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'><div><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>Just received this \
afternoon:</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>==================</span><o:p></o:p></p><p><span \
style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>Support \
Team,</span><o:p></o:p></p><p><span \
style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>You have \
several business customers being affected by an ongoing issue. In order to resolve \
this, First Data is requesting that you clear the cache on all DNS servers being used \
to support them. We propagated a correction over 16 hours ago and know that Google \
DNS and others are translating correctly. Would you please help us assist your \
customers?</span><o:p></o:p></p><p><span \
style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>The correct \
resolutions are:<br><a href="http://vxn.datawire.net/">vxn.datawire.net</a> \
216.220.36.75<br><a href="http://vxn1.datawire.net/">vxn1.datawire.net</a> \
205.167.140.10<br><a href="http://vxn2.datawire.net/">vxn2.datawire.net</a> \
64.243.142.36<br><a href="http://vxn3.datawire.net/">vxn3.datawire.net</a> \
206.112.91.167<br><a href="http://vxn4.datawire.net/">vxn4.datawire.net</a> \
63.240.199.76</span><o:p></o:p></p><p><span \
style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>If you are \
resolving it as anything starting with 45.x.x.x, it is incorrect. Please feel free to \
compare to the Google DNS resolution for confirmation.</span><o:p></o:p></p><p><span \
style='font-size:10.5pt;font-family:"Arial",sans-serif;color:#333333'>Please either \
reply all or call First Data's Network Operations at 888-377-8726 Option \
3.</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif;color:#004165'><snip></span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif;color:#1F497D'><br>First \
Data, <em><span style='font-family:"Verdana",sans-serif'>240 North Roosevelt \
Av</span></em></span><em><span \
style='font-size:8.0pt;font-family:"Arial",sans-serif;color:#1F497D'> </span></em><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;margin-bottom:12.0pt'><em><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif;color:#1F497D'>Chandler, \
Arizona 85226</span></em><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif;color:#1F497D'><br><br><br></span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>==================</span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> </span><o:p></o:p></p><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>That kind of confirms that \
the TTL for the 45.x.x.x record(s) were a bit too long – if they had been short, \
like they are now at 300 seconds, the issue would mostly have cleared \
up.</span><o:p></o:p></p><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> </span><o:p></o:p></p><div><div \
style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p \
class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><strong><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'>From:</span></strong><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> Outages <<a \
href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>> \
<strong><span style='font-family:"Verdana",sans-serif'>On Behalf Of \
</span></strong>frnkblk--- via Outages<br><strong><span \
style='font-family:"Verdana",sans-serif'>Sent:</span></strong> Friday, July 13, 2018 \
9:56 PM<br><strong><span style='font-family:"Verdana",sans-serif'>To:</span></strong> \
'Luke Guillory' <<a \
href="mailto:lguillory@reservetele.com">lguillory@reservetele.com</a>>; <a \
href="mailto:jayson@peakinter.net">jayson@peakinter.net</a>; <a \
href="mailto:outages@outages.org">outages@outages.org</a><br><strong><span \
style='font-family:"Verdana",sans-serif'>Subject:</span></strong> Re: [outages] \
Problem with credit card machine processing? \
"Datawire"</span><o:p></o:p></p></div></div><p class=MsoNormal \
style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span \
style='font-size:10.0pt;font-family:"Verdana",sans-serif'> </span><o:p></o:p></p><p \
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic