[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kde-devel
Subject:    Re: OCS Providers File - High Numbers of Requests
From:       Aleix Pol <aleixpol () kde ! org>
Date:       2021-09-23 13:40:44
Message-ID: CACcA1RrHVZyU6scqD8k3gogdKFwKh9uYcgfAhNvXRNQa66shGg () mail ! gmail ! com
[Download RAW message or body]

On Thu, Sep 23, 2021 at 1:54 PM Aleix Pol <aleixpol@kde.org> wrote:
>
> On Thu, Sep 23, 2021 at 11:52 AM Ben Cooksley <bcooksley@kde.org> wrote:
> >
> > Hi all,
> >
> > It has recently come to our attention that the number of queries being =
handled for the endpoint https://autoconfig.kde.org/ocs/providers.xml on a =
day to day basis has gotten to the point where it is causing issues with se=
rver responsiveness to other traffic. This is perhaps best summarised by th=
e following:
> >
> > root@nicoda /var/log/apache2 # ls -lah ...
> > -rw-r----- 1 root adm 458M Sep 23 06:25 autoconfig.kde.org.log.1
> > -rw-r----- 1 root adm 381M Sep 23 06:25 networkcheck.kde.org.log.1
> > -rw-r----- 1 root adm 143M Sep 23 06:25 www.kde.org.log.1
> >
> > root@nicoda /var/log/apache2 # cat autoconfig.kde.org.log.1 | wc -l
> > 4,222,343
> >
> > Based on those numbers we're looking at 48-49 requests per second (on a=
verage - peaks are much higher by many magnitudes), which seems extremely e=
xcessive given that this file is only supposed to be retrieved by KDE softw=
are when GHNS functionality is triggered. That is supported by the substant=
ial size difference it has over networkcheck.kde.org - which is used by pla=
sma-nm and NetworkManager (on Neon) to check for whether they have a workin=
g internet connection - which i'd expect to be the site receiving the most =
traffic.
> >
> > As such, I therefore suspect we have bug(s) in software that makes use =
of GHNS functionality.
> >
> > It would therefore be appreciated if we could please review the softwar=
e in question to determine whether it is operating correctly. Given that it=
 usually runs in the background on user systems, i'd especially appreciate =
it if a detailed review could be conducted on Discover and other software t=
hat conducts package management operations or assists in managing updates.
> >
> > Unfortunately all these applications submit a fairly useless user agent=
 (Mozilla/5.0) so it is impossible for Sysadmin to ascertain any further in=
formation. If we could get information on the software that is originating =
the request added to the user agent to assist in investigating these issues=
 in the future that would be extremely helpful.
> >
> > Thanks,
> > Ben
>
> That's correct. Discover fetches them at startup. It's necessary to be
> able to check if there are updates on KNS-provided resources.
>
> Incidentally,  I was looking into this yesterday incidentally. We
> could see if caching is broken somehow. A request will still be needed
> though to check if the cache is out of date.
>
> Or we can prefer the cached version of the file for a few days and
> assume they don't change that much.
>
> I've seen some of them point at
> "https://download.kde.org/ocs/providers.xml", I don't know if this is
> any better.
>
> Aleix

https://invent.kde.org/frameworks/attica/-/merge_requests/15/
https://invent.kde.org/frameworks/knewstuff/-/merge_requests/141
https://invent.kde.org/plasma/discover/-/merge_requests/165

This should lessen the problem from Discover's side. Still that one
providers.xml request at startup remains.

Aleix
[prev in list] [next in list] [prev in thread] [next in thread]