[prev in list] [next in list] [prev in thread] [next in thread]
List: pywikipediabot-users
Subject: Re: [pywikibot] Throttling for minutes?
From: Jean-Frﾃｩdﾃｩric <jeanfrederic.wiki () gmail ! com>
Date: 2020-02-01 23:41:54
Message-ID: CADCCoom3hEdLHqLrK0wL361ewEm-z9Q2LLTwOQ3HMcRmNYdqeg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Le ven. 31 janv. 2020 ﾃ 22:11, Strainu <strainu10@gmail.com> a ﾃｩcrit :
> Jean-Fred, if you want to confirm whether the two are related you can try
> displaying the site which gets throttled. In my case it was always
> wikidata.
Oh, right, forgot to mention: that particular tool is only interacting with
Wikidata (at least for now). (I have added Pastes with the log lines to the
Phab ticket too).
>
> Strainu
>
> Pe vineri, 31 ianuarie 2020, Jean-Frﾃｩdﾃｩric <jeanfrederic.wiki@gmail.com>
> a scris:
>
>> Hi,
>>
>> I encountered what I believe is the same issue as others. This happens on
>> my integraality tool with entirely read-only operations, without any hits
>> on the Query service.
>>
>> I filed <https://phabricator.wikimedia.org/T244030> with my
>> investigations and the tracing showing how Pywikibot spent over 25 minutes
>> doing a page.templatesWithParams().
>>
>> Hope that helps,
>>
>> -- Jean-Fred
>>
>> Le ven. 31 janv. 2020 ﾃ 18:32, AntiCompositeNumber <
>> anticompositenumber@gmail.com> a ﾃｩcrit :
>>
>>> https://phabricator.wikimedia.org/T243701
>>>
>>> On Fri, Jan 31, 2020 at 10:01 AM Albert Villanova del Moral
>>> <albertvillanova@gmail.com> wrote:
>>> >
>>> > Hello,
>>> >
>>> > Have you finally created a ticket for this issue? It is very annoying:
>>> sometimes, more than 30 minutes "sleeping".
>>> >
>>> > Regards,
>>> > Albert.
>>> >
>>> > Albert VILLANOVA DEL MORAL
>>> > Phone: +33 6 24 20 42 38
>>> > Email: albert.villanova@gmail.com
>>> >
>>> >
>>> > Missatge de Amir Sarabadani <ladsgroup@gmail.com> del dia ds., 25 de
>>> gen. 2020 a les 23:55:
>>> >>
>>> >> I would recommend creating a ticket against search platform team as
>>> this is due to WDQS nodes being behind Wikidata's production.
>>> >>
>>> >> On Sat, Jan 25, 2020 at 11:27 PM Strainu <strainu10@gmail.com> wrote:
>>> >>>
>>> >>> I can confirm the throttling is caused by Wikidata. Not sure why I
>>> >>> didn't see it before, as I was using that api query that you gave
>>> >>> above. Are there some (known) issues currently or should I log a bug
>>> >>> on this?
>>> >>>
>>> >>> Strainu
>>> >>>
>>> >>> ﾃ始 dum., 26 ian. 2020 la 00:01, Amir Sarabadani <ladsgroup@gmail.com>
>>> a scris:
>>> >>> >
>>> >>> > It should be accessible in
>>> https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?orgId=1&refresh=1m&fullscreen&panelId=22
>>> but I'm not sure why other types of max lag is not showing up.
>>> >>> >
>>> >>> > You can also check maxlag like this:
>>> https://www.wikidata.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1
>>> >>> >
>>> >>> > On Sat, Jan 25, 2020 at 10:53 PM Strainu <strainu10@gmail.com>
>>> wrote:
>>> >>> >>
>>> >>> >> Actually, now that you mention it, it might be because of
>>> wikidata -
>>> >>> >> I read stuff from there in all affected scripts. If I get
>>> throttled
>>> >>> >> there 5s for each element, this might account for the long
>>> delays. Let
>>> >>> >> me see if I can print some debug data. Any suggestion on what I
>>> should
>>> >>> >> look for regarding Wikidata lag?
>>> >>> >>
>>> >>> >> ﾃ始 sﾃ｢m., 25 ian. 2020 la 23:48, Strainu <strainu10@gmail.com> a
>>> scris:
>>> >>> >> >
>>> >>> >> > All sites. Currently I also encounter other issues (connection
>>> >>> >> > timeouts), but the throttling has been going on for days (at
>>> least
>>> >>> >> > since 1/20).
>>> >>> >> >
>>> >>> >> > ﾃ始 sﾃ｢m., 25 ian. 2020 la 23:46, Amir Sarabadani <
>>> ladsgroup@gmail.com> a scris:
>>> >>> >> > >
>>> >>> >> > > Is it Wikidata only it's all sites? Wikidata uses another set
>>> of measures for maxlag, like lag between dispatching to other wiki and lag
>>> to WDQS.
>>> >>> >> > >
>>> >>> >> > > HTH
>>> >>> >> > >
>>> >>> >> > > On Sat, Jan 25, 2020 at 10:40 PM Strainu <strainu10@gmail.com>
>>> wrote:
>>> >>> >> > >>
>>> >>> >> > >> Hi,
>>> >>> >> > >>
>>> >>> >> > >> In the last few days my bots get throttled for minutes and
>>> minutes (up
>>> >>> >> > >> to 10, on the worse instances) on basically all sites I use
>>> (ro.wp,
>>> >>> >> > >> commons, wikidata). I knew about write throttle and I use
>>> the default
>>> >>> >> > >> setting there, but it seems that now I get throttled on read
>>> >>> >> > >> operations as well. I use pwb from git, currently synced at
>>> >>> >> > >> 0924e1655ef9e61c86e98884226b18d8250da368.
>>> >>> >> > >>
>>> >>> >> > >> I checked the server lag and it is 0 most times I see the
>>> throttling.
>>> >>> >> > >> Any idea why do I get throttled for such long periods of
>>> time?
>>> >>> >> > >>
>>> >>> >> > >> Thanks,
>>> >>> >> > >> Strainu
>>> >>> >> > >>
>>> >>> >> > >> _______________________________________________
>>> >>> >> > >> pywikibot mailing list
>>> >>> >> > >> pywikibot@lists.wikimedia.org
>>> >>> >> > >> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > >
>>> >>> >> > > --
>>> >>> >> > > Amir (he/him)
>>> >>> >> > >
>>> >>> >> > > _______________________________________________
>>> >>> >> > > pywikibot mailing list
>>> >>> >> > > pywikibot@lists.wikimedia.org
>>> >>> >> > > https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >>> >>
>>> >>> >> _______________________________________________
>>> >>> >> pywikibot mailing list
>>> >>> >> pywikibot@lists.wikimedia.org
>>> >>> >> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >>> >
>>> >>> >
>>> >>> >
>>> >>> > --
>>> >>> > Amir (he/him)
>>> >>> >
>>> >>> > _______________________________________________
>>> >>> > pywikibot mailing list
>>> >>> > pywikibot@lists.wikimedia.org
>>> >>> > https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >>>
>>> >>> _______________________________________________
>>> >>> pywikibot mailing list
>>> >>> pywikibot@lists.wikimedia.org
>>> >>> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> Amir (he/him)
>>> >>
>>> >> _______________________________________________
>>> >> pywikibot mailing list
>>> >> pywikibot@lists.wikimedia.org
>>> >> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>> >
>>> > _______________________________________________
>>> > pywikibot mailing list
>>> > pywikibot@lists.wikimedia.org
>>> > https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>>
>>> _______________________________________________
>>> pywikibot mailing list
>>> pywikibot@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>>>
>>
>>
>> --
>> Jean-Frﾃｩdﾃｩric
>>
> _______________________________________________
> pywikibot mailing list
> pywikibot@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/pywikibot
>
--
Jean-Frﾃｩdﾃｩric
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">Le ven. 31 janv. 2020 ﾃ 22:11, Strainu <<a \
href="mailto:strainu10@gmail.com">strainu10@gmail.com</a>> a ﾃｩcrit \
:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Jean-Fred, if you want \
to confirm whether the two are related you can try displaying the site which gets \
throttled. In my case it was always wikidata. </blockquote><div><br></div><div>Oh, \
right, forgot to mention: that particular tool is only interacting with Wikidata (at \
least for now). (I have added Pastes with the log lines to the Phab ticket \
too).<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div><br></div><div>Strainu <br><br>Pe vineri, 31 \
ianuarie 2020, Jean-Frﾃｩdﾃｩric <<a href="mailto:jeanfrederic.wiki@gmail.com" \
target="_blank">jeanfrederic.wiki@gmail.com</a>> a scris:<br><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi,</div><div><br></div><div>I \
encountered what I believe is the same issue as others. This happens on my \
integraality tool with entirely read-only operations, without any hits on the Query \
service.</div><div><br></div><div>I filed <<a \
href="https://phabricator.wikimedia.org/T244030" \
target="_blank">https://phabricator.wikimedia.org/T244030</a>> with my \
investigations and the tracing showing how Pywikibot spent over 25 minutes doing a \
<tt>page.templatesWithParams()</tt>.</div><div><br></div><div>Hope that \
helps,</div><div><br></div><div>-- Jean-Fred<br></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">Le ven. 31 janv. 2020 ﾃ \
18:32, AntiCompositeNumber <<a href="mailto:anticompositenumber@gmail.com" \
target="_blank">anticompositenumber@gmail.com</a>> a ﾃｩcrit \
:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><a \
href="https://phabricator.wikimedia.org/T243701" rel="noreferrer" \
target="_blank">https://phabricator.wikimedia.org/T243701</a><br> <br>
On Fri, Jan 31, 2020 at 10:01 AM Albert Villanova del Moral<br>
<<a href="mailto:albertvillanova@gmail.com" \
target="_blank">albertvillanova@gmail.com</a>> wrote:<br> ><br>
> Hello,<br>
><br>
> Have you finally created a ticket for this issue? It is very annoying: \
sometimes, more than 30 minutes "sleeping".<br> ><br>
> Regards,<br>
> Albert.<br>
><br>
> Albert VILLANOVA DEL MORAL<br>
> Phone: +33 6 24 20 42 38<br>
> Email: <a href="mailto:albert.villanova@gmail.com" \
target="_blank">albert.villanova@gmail.com</a><br> ><br>
><br>
> Missatge de Amir Sarabadani <<a href="mailto:ladsgroup@gmail.com" \
target="_blank">ladsgroup@gmail.com</a>> del dia ds., 25 de gen. 2020 a les \
23:55:<br> >><br>
>> I would recommend creating a ticket against search platform team as this is \
due to WDQS nodes being behind Wikidata's production.<br> >><br>
>> On Sat, Jan 25, 2020 at 11:27 PM Strainu <<a \
href="mailto:strainu10@gmail.com" target="_blank">strainu10@gmail.com</a>> \
wrote:<br> >>><br>
>>> I can confirm the throttling is caused by Wikidata. Not sure why I<br>
>>> didn't see it before, as I was using that api query that you \
gave<br> >>> above. Are there some (known) issues currently or should I log \
a bug<br> >>> on this?<br>
>>><br>
>>> Strainu<br>
>>><br>
>>> ﾃ始 dum., 26 ian. 2020 la 00:01, Amir Sarabadani <<a \
href="mailto:ladsgroup@gmail.com" target="_blank">ladsgroup@gmail.com</a>> a \
scris:<br> >>> ><br>
>>> > It should be accessible in <a \
href="https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?orgId=1&refresh=1m&fullscreen&panelId=22" \
rel="noreferrer" target="_blank">https://grafana.wikimedia.org/d/000000156/wikidata-dispatch?orgId=1&refresh=1m&fullscreen&panelId=22</a> \
but I'm not sure why other types of max lag is not showing up.<br> >>> \
><br> >>> > You can also check maxlag like this: <a \
href="https://www.wikidata.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1" \
rel="noreferrer" target="_blank">https://www.wikidata.org/w/api.php?action=query&titles=MediaWiki&format=json&maxlag=-1</a><br>
>>> ><br>
>>> > On Sat, Jan 25, 2020 at 10:53 PM Strainu <<a \
href="mailto:strainu10@gmail.com" target="_blank">strainu10@gmail.com</a>> \
wrote:<br> >>> >><br>
>>> >> Actually, now that you mention it, it might be because of \
wikidata -<br> >>> >> I read stuff from there in all affected scripts. \
If I get throttled<br> >>> >> there 5s for each element, this might \
account for the long delays. Let<br> >>> >> me see if I can print some \
debug data. Any suggestion on what I should<br> >>> >> look for \
regarding Wikidata lag?<br> >>> >><br>
>>> >> ﾃ始 sﾃ｢m., 25 ian. 2020 la 23:48, Strainu <<a \
href="mailto:strainu10@gmail.com" target="_blank">strainu10@gmail.com</a>> a \
scris:<br> >>> >> ><br>
>>> >> > All sites. Currently I also encounter other issues \
(connection<br> >>> >> > timeouts), but the throttling has been \
going on for days (at least<br> >>> >> > since 1/20).<br>
>>> >> ><br>
>>> >> > ﾃ始 sﾃ｢m., 25 ian. 2020 la 23:46, Amir Sarabadani <<a \
href="mailto:ladsgroup@gmail.com" target="_blank">ladsgroup@gmail.com</a>> a \
scris:<br> >>> >> > ><br>
>>> >> > > Is it Wikidata only it's all sites? Wikidata uses \
another set of measures for maxlag, like lag between dispatching to other wiki and \
lag to WDQS.<br> >>> >> > ><br>
>>> >> > > HTH<br>
>>> >> > ><br>
>>> >> > > On Sat, Jan 25, 2020 at 10:40 PM Strainu <<a \
href="mailto:strainu10@gmail.com" target="_blank">strainu10@gmail.com</a>> \
wrote:<br> >>> >> > >><br>
>>> >> > >> Hi,<br>
>>> >> > >><br>
>>> >> > >> In the last few days my bots get throttled for \
minutes and minutes (up<br> >>> >> > >> to 10, on the worse \
instances) on basically all sites I use (ro.wp,<br> >>> >> > \
>> commons, wikidata). I knew about write throttle and I use the default<br> \
>>> >> > >> setting there, but it seems that now I get \
throttled on read<br> >>> >> > >> operations as well. I use \
pwb from git, currently synced at<br> >>> >> > >> \
0924e1655ef9e61c86e98884226b18d8250da368.<br> >>> >> > >><br>
>>> >> > >> I checked the server lag and it is 0 most times I \
see the throttling.<br> >>> >> > >> Any idea why do I get \
throttled for such long periods of time?<br> >>> >> > >><br>
>>> >> > >> Thanks,<br>
>>> >> > >> Strainu<br>
>>> >> > >><br>
>>> >> > >> \
_______________________________________________<br> >>> >> > \
>> pywikibot mailing list<br> >>> >> > >> <a \
href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >>> >> > \
>> <a href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" \
rel="noreferrer" target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br>
>>> >> > ><br>
>>> >> > ><br>
>>> >> > ><br>
>>> >> > > --<br>
>>> >> > > Amir (he/him)<br>
>>> >> > ><br>
>>> >> > > _______________________________________________<br>
>>> >> > > pywikibot mailing list<br>
>>> >> > > <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >>> >> > > \
<a href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
>>> >><br> >>> >> \
_______________________________________________<br> >>> >> pywikibot \
mailing list<br> >>> >> <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >>> >> <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
>>> ><br> >>> ><br>
>>> ><br>
>>> > --<br>
>>> > Amir (he/him)<br>
>>> ><br>
>>> > _______________________________________________<br>
>>> > pywikibot mailing list<br>
>>> > <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >>> > <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
>>><br> >>> _______________________________________________<br>
>>> pywikibot mailing list<br>
>>> <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >>> <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
>><br> >><br>
>><br>
>> --<br>
>> Amir (he/him)<br>
>><br>
>> _______________________________________________<br>
>> pywikibot mailing list<br>
>> <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> >> <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
><br> > _______________________________________________<br>
> pywikibot mailing list<br>
> <a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> > <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> <br>
_______________________________________________<br>
pywikibot mailing list<br>
<a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
</blockquote></div><br clear="all"><br>-- <br><div \
dir="ltr">Jean-Frﾃｩdﾃｩric<br></div></div> </blockquote></div>
_______________________________________________<br>
pywikibot mailing list<br>
<a href="mailto:pywikibot@lists.wikimedia.org" \
target="_blank">pywikibot@lists.wikimedia.org</a><br> <a \
href="https://lists.wikimedia.org/mailman/listinfo/pywikibot" rel="noreferrer" \
target="_blank">https://lists.wikimedia.org/mailman/listinfo/pywikibot</a><br> \
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" \
class="gmail_signature">Jean-Frﾃｩdﾃｩric<br></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
pywikibot mailing list
pywikibot@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/pywikibot
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic