[prev in list] [next in list] [prev in thread] [next in thread] 

List:       wikitech-l
Subject:    Re: [Wikitech-l] job queue
From:       bawolff <bawolff+wn () gmail ! com>
Date:       2013-03-25 17:36:02
Message-ID: CA+oo+DVfB2h35fXdSMVfiiszimK7q=AhbUTFRizGfXuf_JhM_Q () mail ! gmail ! com
[Download RAW message or body]

On Mon, Mar 25, 2013 at 1:34 PM, dan entous <d_entous@yahoo.com> wrote:
> context
> -------
> i'm working on a mediawiki extension, \
> http://www.mediawiki.org/wiki/Extension:GWToolset, which has as one of its goals, \
> the ability to upload media files to a wiki. the extension, among other tasks, will \
> process an xml file that has a list of urls to media files and upload those media \
> files to the wiki. our ideal goal is to have this extension run on \
> http://commons.wikimedia.org/. 
> 
> job queue goals
> ---------------
> 1. setup the processing of the xml file as a job queue job
> 2. each media file upload to be setup as a job queue job
> 
> 
> current implementation
> ----------------------
> i have been able to achieve goal 2 and will sort out goal 1 shortly.
> 
> 
> issues/questions
> ----------------
> 1. each of these jobs can take several seconds to complete. i have noticed in my \
> local wiki that each of these jobs is picked up with each wiki visit and slows down \
> the response of the wiki by however many seconds the job takes to run, a sleep in \
> the job shows that if the job takes 15 seconds to run the wiki will be slowed down \
> by that amount of time; i don't want this to happen on my local wiki or on commons. \
>  a. are jobs on commons run as part of each wiki visit?
> b. is there a cron job that takes care of the job queue on commons instead of using \
> each wiki visit? c. if not, is there a way to indicate that the job should only be \
> run as a cron job and not with a wiki visit? 
> 2. if there's no solution to running the job with each wiki visit and slowing down \
> the site, what other suggestions are there on processing the xml file and \
> individual media file uploads? 
> 
> thanks in advance!
> dan
> _______________________________________________
> Wikitech-l mailing list
> Wikitech-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Given your use-case, it seems like making a maintenance script that
reads the xml file and imports images might make more sense. Mass
uploads are rare enough events that I imagine going through the
process of somebody running the maintinance script would not be
prohibitive (But that's just a guess).

> a. are jobs on commons run as part of each wiki visit?
> b. is there a cron job that takes care of the job queue on commons instead of using \
> each wiki visit?

No, commons uses runJobs.php (I believe. At the very least jobs are
not run from webrequests). I doubt cron is used, I imagine there are
enough jobs that there is a web server (probably more than one)
dedicated solely to dealing with the job queue at all times.

--bawolff

p.s. As an aside, looking for this source code I noticed its developed
on github "until it is ready for full review". You do realize if you
wanted to you could put the code in Wikimedia's gerrit and ignore the
review features until you're ready for the full review.

_______________________________________________
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l


[prev in list] [next in list] [prev in thread] [next in thread]