[prev in list] [next in list] [prev in thread] [next in thread]
List: outages-discussion
Subject: Re: [Outages-discussion] =?utf-8?q?=5BEXTERNAL=5D_Re=3A_Question_abo?=
From: Zach Camara <zach.j.camara () gmail ! com>
Date: 2020-06-29 11:04:42
Message-ID: CAMfmsaEJb0QpAA9nQA9MTrSrvtxPiUK7hk2r0EryTT8isC3sjA () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
When I dealt with this a few employers ago it meant running some reports
from the proxy logs hunting who had the most traffic to google and one of
the top 10 had a RealPlayer browser plugin that was misbehaving as Damian
noted is a popular reason.
That or someone playing around with scripting requests :)
Zach
On Mon, Jun 29, 2020 at 1:25 AM Chapman, Brad (NBCUniversal) <
Brad.Chapman@nbcuni.com> wrote:
> It's a network that has no direct route to the Internet and all web
> traffic **must** traverse the proxy.
>
>
>
> -Brad
>
>
>
> *From: *Damian Menscher <damian@google.com>
> *Date: *Sunday, June 28, 2020 at 10:05 PM
> *To: *"Chapman, Brad (NBCUniversal)" <Brad.Chapman@nbcuni.com>
> *Cc: *"outages-discussion@outages.org" <outages-discussion@outages.org>
> *Subject: *Re: [EXTERNAL] Re: [Outages-discussion] Question about the
> Google "sorry" page...
>
>
>
> Two cautions regarding proxies:
>
> - as mentioned before, make sure you don't have an open proxy, which
> might be abused
>
> - if you're proxying only some traffic (eg, for content filtering, etc),
> then be sure all Google traffic gets proxied out the same IP. We sometimes
> see weirdness when some requests go through the proxy, but other requests
> go directly from the (home) IP. This can cause problems, for example the
> captcha exemption may fail due to the IP mis-match.
>
>
>
> Damian
>
>
>
> On Sun, Jun 28, 2020 at 9:55 PM Chapman, Brad (NBCUniversal) <
> Brad.Chapman@nbcuni.com> wrote:
>
> Interesting; thanks.
>
>
>
> Would you expect to see this behavior in an environment where a proxy
> server is used to funnel traffic to the Internet and clients have to use a
> PAC file or WPAD?
>
> —Sent from my iPhone
>
>
>
> On Jun 28, 2020, at 9:34 PM, Damian Menscher <damian@google.com> wrote:
>
> Blocking occurs when automated searching is detected, not simply due to
> the total volume of requests from a single IP. As such, there is no option
> for an exception.
>
>
>
> To "solve" this, we recommend you minimize the number of users sharing an
> IP. The easiest method is with IPv6, since then each user can have their
> own /64 (our abuse systems don't look deeper than that). If you're stuck
> with IPv4, separate your corporate-managed machines from the guest wifi
> (which is harder to control), and try to give different groups of users
> their own NAT IP (by building or floor, etc). That way when there's a
> problem you'll have fewer users impacted, and a smaller list of suspects.
>
>
>
> If you want to start digging into the reasons why your IP might have been
> blocked, the most common reasons for getting blocked (mostly for websearch)
> include (in no particular order):
>
> - malware that proxies abuse for criminals
>
> - browser extensions that automate searching
>
> - misconfigured browsers that have anomalous behavior
>
> - corporate proxies that are open for abuse
>
> - users installing "P2P VPN" software, which is also abused
>
>
>
> Damian
>
> --
>
> Damian Menscher :: Security Reliability Engineer :: Google :: AS15169
>
>
>
> On Sun, Jun 28, 2020 at 4:57 PM Chapman, Brad (NBCUniversal) <
> Brad.Chapman@nbcuni.com> wrote:
>
> Greetings Outages-Discussion,
>
> I hope you are all having a pleasant Sunday afternoon / evening with no P1
> / SevA / 4-alarm fires caused by a violation of Read-only Friday.
>
> Given the number of sysadmins and telecom / network engineers on this
> list, I am guessing that we have seen (or been asked to explain) the Google
> "Sorry" page.
>
> Occasionally, our company gets a burst of calls about this issue, until
> the lockout expires on Google's side. We manage >50,000 computers so even
> short lockouts can generate dozens of calls.
>
> Has anyone ever approached Google's NOC team to request an exemption from
> the Sorry page for their busiest external IP addresses? Or, if not a
> blanket exemption, to request an increase in the threshold before it is
> tripped?
>
> Hope you're all staying safe.
>
> Cheers,
> Brad Chapman
> NBCUniversal
>
> —Sent from my iPhone
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
> <https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages-discuss \
> ion__;!!PIZeeW5wscynRQ!-T5SokgIYLbWPeqRO4boP4fHxQbHaOHVW5G6FNDQ4sI2cVgFNtCDeAvOwaP5eN4PNg$>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion@outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
[Attachment #5 (text/html)]
<div><div dir="auto">When I dealt with this a few employers ago it meant running some \
reports from the proxy logs hunting who had the most traffic to google and one of the \
top 10 had a RealPlayer browser plugin that was misbehaving as Damian noted is a \
popular reason.</div><div dir="auto"><br></div><div dir="auto">That or someone \
playing around with scripting requests :)</div><div dir="auto"><br></div><div \
dir="auto"><br></div><div dir="auto">Zach</div></div><div \
dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">On Mon, Jun 29, 2020 at 1:25 AM Chapman, Brad (NBCUniversal) \
<<a href="mailto:Brad.Chapman@nbcuni.com">Brad.Chapman@nbcuni.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_7225003523452863785WordSection1">
<p class="MsoNormal">It's a network that has no direct route to the Internet and all \
web traffic *<b>must</b>* traverse the proxy. </p></div></div><div lang="EN-US" \
link="blue" vlink="purple"><div class="m_7225003523452863785WordSection1"><p \
class="MsoNormal"><u></u><u></u></p> <p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-Brad<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;padding:3pt 0in \
0in;border-top-color:rgb(181,196,223)"> <p class="MsoNormal"><b><span \
style="font-size:12pt;color:black">From: </span></b><span \
style="font-size:12pt;color:black">Damian Menscher <<a \
href="mailto:damian@google.com" target="_blank">damian@google.com</a>><br> \
<b>Date: </b>Sunday, June 28, 2020 at 10:05 PM<br> <b>To: </b>"Chapman, Brad \
(NBCUniversal)" <<a href="mailto:Brad.Chapman@nbcuni.com" \
target="_blank">Brad.Chapman@nbcuni.com</a>><br> <b>Cc: </b>"<a \
href="mailto:outages-discussion@outages.org" \
target="_blank">outages-discussion@outages.org</a>" <<a \
href="mailto:outages-discussion@outages.org" \
target="_blank">outages-discussion@outages.org</a>><br> <b>Subject: </b>Re: \
[EXTERNAL] Re: [Outages-discussion] Question about the Google "sorry" \
page...<u></u><u></u></span></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Two cautions regarding proxies:<u></u><u></u></p>
<div>
<p class="MsoNormal"> - as mentioned before, make sure you don't have an open \
proxy, which might be abused<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> - if you're proxying only some traffic (eg, for content \
filtering, etc), then be sure all Google traffic gets proxied out the same IP. We \
sometimes see weirdness when some requests go through the proxy, but other requests \
go directly from the (home) IP. This can cause problems, for example the captcha \
exemption may fail due to the IP mis-match.<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Damian<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jun 28, 2020 at 9:55 PM Chapman, Brad (NBCUniversal) \
<<a href="mailto:Brad.Chapman@nbcuni.com" \
target="_blank">Brad.Chapman@nbcuni.com</a>> wrote:<u></u><u></u></p> </div>
<blockquote style="border-style:none none none \
solid;border-left-width:1pt;padding:0in 0in 0in \
6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)"> <div>
<p class="MsoNormal">Interesting; thanks. <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">Would you expect to see this behavior \
in an environment where a proxy server is used to funnel traffic to the Internet and \
clients have to use a PAC file or WPAD?<u></u><u></u></p> <div>
<p class="MsoNormal">—Sent from my iPhone<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal" style="margin-bottom:12pt">On Jun 28, 2020, at 9:34 PM, Damian \
Menscher <<a href="mailto:damian@google.com" \
target="_blank">damian@google.com</a>> wrote:<u></u><u></u></p> </blockquote>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<p class="MsoNormal">Blocking occurs when automated searching is detected, not simply \
due to the total volume of requests from a single IP. As such, there is no option \
for an exception.<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">To "solve" this, we recommend you minimize the number \
of users sharing an IP. The easiest method is with IPv6, since then each user can \
have their own /64 (our abuse systems don't look deeper than that). If \
you're stuck with IPv4, separate your corporate-managed machines from the guest \
wifi (which is harder to control), and try to give different groups of users their \
own NAT IP (by building or floor, etc). That way when there's a problem \
you'll have fewer users impacted, and a smaller list of \
suspects.<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you want to start digging into the reasons why your IP might \
have been blocked, the most common reasons for getting blocked (mostly for websearch) \
include (in no particular order):<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> - malware that proxies abuse for criminals<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - browser extensions that automate \
searching<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> - misconfigured browsers that have anomalous \
behavior<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"> - corporate proxies that are open for abuse<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - users installing "P2P VPN" software, which is \
also abused<u></u><u></u></p> </div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Damian<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Damian Menscher :: Security Reliability Engineer :: Google :: \
AS15169<u></u><u></u></p> </div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jun 28, 2020 at 4:57 PM Chapman, Brad (NBCUniversal) \
<<a href="mailto:Brad.Chapman@nbcuni.com" \
target="_blank">Brad.Chapman@nbcuni.com</a>> wrote:<u></u><u></u></p> </div>
<blockquote style="border-style:none none none \
solid;border-left-width:1pt;padding:0in 0in 0in \
6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)"> <p \
class="MsoNormal">Greetings Outages-Discussion,<br> <br>
I hope you are all having a pleasant Sunday afternoon / evening with no P1 / SevA / \
4-alarm fires caused by a violation of Read-only Friday. <br>
<br>
Given the number of sysadmins and telecom / network engineers on this list, I am \
guessing that we have seen (or been asked to explain) the Google "Sorry" page.<br> \
<br> Occasionally, our company gets a burst of calls about this issue, until the \
lockout expires on Google's side. We manage >50,000 computers so even short \
lockouts can generate dozens of calls. <br>
<br>
Has anyone ever approached Google's NOC team to request an exemption from the Sorry \
page for their busiest external IP addresses? Or, if not a blanket exemption, to \
request an increase in the threshold before it is tripped?<br> <br>
Hope you're all staying safe.<br>
<br>
Cheers,<br>
Brad Chapman<br>
NBCUniversal<br>
<br>
—Sent from my iPhone<br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" \
target="_blank">Outages-discussion@outages.org</a><br> <a \
href="https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages-disc \
ussion__;!!PIZeeW5wscynRQ!-T5SokgIYLbWPeqRO4boP4fHxQbHaOHVW5G6FNDQ4sI2cVgFNtCDeAvOwaP5eN4PNg$" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" \
target="_blank">Outages-discussion@outages.org</a><br> <a \
href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" \
target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br> \
</blockquote></div></div>
_______________________________________________
Outages-discussion mailing list
Outages-discussion@outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic