[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-frameworks-devel
Subject:    Re: OCS Providers File - High Numbers of Requests
From:       Aleix Pol <aleixpol () kde ! org>
Date:       2021-09-26 18:03:20
Message-ID: CACcA1Ro8V1EoW3kj3iafmaq=cFpjW4AeA2TS7aGjCQNwSK3qmg () mail ! gmail ! com
[Download RAW message or body]

On Fri, Sep 24, 2021 at 8:26 PM Ben Cooksley <bcooksley@kde.org> wrote:
>
> On Sat, Sep 25, 2021 at 12:34 AM Aleix Pol <aleixpol@kde.org> wrote:
>>
>> On Fri, Sep 24, 2021 at 1:32 AM Nicol=C3=A1s Alvarez
>> <nicolas.alvarez@gmail.com> wrote:
>> >
>> > El jue, 23 de sep. de 2021 a la(s) 19:31, Aleix Pol (aleixpol@kde.org)=
 escribi=C3=B3:
>> > >
>> > > On Thu, Sep 23, 2021 at 10:12 PM Nicol=C3=A1s Alvarez
>> > > <nicolas.alvarez@gmail.com> wrote:
>> > > >
>> > > > El jue, 23 de sep. de 2021 a la(s) 08:55, Aleix Pol (aleixpol@kde.=
org) escribi=C3=B3:
>> > > > >
>> > > > > On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley <bcooksley@kde.org=
> wrote:
>> > > > > >
>> > > > > > Hi all,
>> > > > > >
>> > > > > > It has recently come to our attention that the number of queri=
es being handled for the endpoint https://autoconfig.kde.org/ocs/providers.=
xml on a day to day basis has gotten to the point where it is causing issue=
s with server responsiveness to other traffic. This is perhaps best summari=
sed by the following:
>> > > > > >
>> > > > > > root@nicoda /var/log/apache2 # ls -lah ...
>> > > > > > -rw-r----- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log=
.1
>> > > > > > -rw-r----- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.l=
og.1
>> > > > > > -rw-r----- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
>> > > > > >
>> > > > > > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | =
wc -l
>> > > > > > 4,222,343
>> > > > > >
>> > > > > > Based on those numbers we're looking at 48-49 requests per sec=
ond (on average - peaks are much higher by many magnitudes), which seems ex=
tremely excessive given that this file is only supposed to be retrieved by =
KDE software when GHNS functionality is triggered. That is supported by the=
 substantial size difference it has over networkcheck.kde.org - which is us=
ed by plasma-nm and NetworkManager (on Neon) to check for whether they have=
 a working internet connection - which i'd expect to be the site receiving =
the most traffic.
>> > > > > >
>> > > > > > As such, I therefore suspect we have bug(s) in software that m=
akes use of GHNS functionality.
>> > > > > >
>> > > > > > It would therefore be appreciated if we could please review th=
e software in question to determine whether it is operating correctly. Give=
n that it usually runs in the background on user systems, i'd especially ap=
preciate it if a detailed review could be conducted on Discover and other s=
oftware that conducts package management operations or assists in managing =
updates.
>> > > > > >
>> > > > > > Unfortunately all these applications submit a fairly useless u=
ser agent (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any f=
urther information. If we could get information on the software that is ori=
ginating the request added to the user agent to assist in investigating the=
se issues in the future that would be extremely helpful.
>> > > > > >
>> > > > > > Thanks,
>> > > > > > Ben
>> > > > >
>> > > > > That's correct. Discover fetches them at startup. It's necessary=
 to be
>> > > > > able to check if there are updates on KNS-provided resources.
>> > > > >
>> > > > > Incidentally,  I was looking into this yesterday incidentally. W=
e
>> > > > > could see if caching is broken somehow. A request will still be =
needed
>> > > > > though to check if the cache is out of date.
>> > > >
>> > > > Caching seems to be working, since the vast majority of the reques=
ts
>> > > > are returning 304 Not Modified.
>> > > >
>> > > > However in *many* cases I see a single IP making multiple requests=
 in
>> > > > the same second, and doing it again the next minute. Here's one IP
>> > > > address picked randomly:
>> > > >
>> > > > [22/Sep/2021:06:25:41 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:25:41 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:25:41 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:25:41 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:27:57 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:27:58 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:27:58 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:32 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:32 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:32 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:32 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:59 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:59 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:59 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:28:59 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:30:11 +0000] "GET /ocs/providers.xml HTTP/1.1" 200
>> > > > [22/Sep/2021:06:30:11 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:30:11 +0000] "GET /ocs/providers.xml HTTP/1.1" 200
>> > > > [22/Sep/2021:06:30:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:30:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:30:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:30:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:19 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:19 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:19 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:19 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > > [22/Sep/2021:06:31:38 +0000] "GET /ocs/providers.xml HTTP/1.1" 304
>> > > >
>> > > > This continues for hours. And it's not an isolated case; again, th=
e IP
>> > > > I searched for was a random one from the log.
>> > > >
>> > > > There are 120 IP addresses that *each* made more than 10,000 reque=
sts
>> > > > in a 24h period.
>> > > >
>> > > > I tried a few GHNS things on my own system and I couldn't reproduc=
e it...
>> > > >
>> > > > --
>> > > > Nicol=C3=A1s
>> > >
>> > > Is there a possibilty that we can get an UserAgent of something like
>> > > that to see who's asking?
>> >
>> > Yes please :P
>> >
>> > (all requests to this file currently have "Mozilla/5.0" as the
>> > User-Agent, I very much welcome modifying the code so that it sends
>> > something useful instead)
>> >
>> > --
>> > Nicol=C3=A1s
>>
>> This is for KNS:
>> https://invent.kde.org/frameworks/knewstuff/-/merge_requests/142
>>
>> Attica already sets the UserAgent.
>
>
> Thanks for getting all of these changes through.
> Hopefully in the coming months we'll start to see the impact of these cha=
nges.
>
> In the meantime, we've switched autoconfig.kde.org to our CDN provider, w=
hich should enable our systems to continue to handle the current load we're=
 experiencing.

Ok, thank you. Please reach out as soon as you see where these
requests are coming from.

Aleix
[prev in list] [next in list] [prev in thread] [next in thread]