[prev in list] [next in list] [prev in thread] [next in thread] 

List:       mediawiki-l
Subject:    Re: [MediaWiki-l] Load balancing web servers and multimedia content
From:       Justin Lloyd <jclbugz () gmail ! com>
Date:       2014-10-28 8:33:45
Message-ID: CAHuiOCBkJOKF00y0yXTQ6FJwzB6KyR7HW=ZCe55qNkfEQn9fUg () mail ! gmail ! com
[Download RAW message or body]

Hmm, interesting. This will take time for me to digest and I need to get to
bed. However, I will say that adding an S3 backend is beyond me since I'm a
sysadmin, not a developer, and I haven't done any significant PHP work in
about 12 years.

Also, unless I'm missing something or being dense (it is late here), rsync
simply wouldn't work since the upload directories are constantly being
accessed and files written through one web server could easily be
immediately accessed afterwards through another web server, and since there
are four web servers (and possibly more or even less if I were to add AWS
Auto Scaling into the mix), so keeping them all identical when writes could
go through any of them would be pretty much impossible.

Also, we've seen problems in the past with the master-write / replica-read
structure due to issues with MySQL 5.5 (since we're running Ubuntu 12.04
currently), such as temp tables getting created on the replica but then
writes to those tables would fail due to the replica being marked
read-only, so for now we have all reads and writes going to the master with
the replica just being there for backups (via cron jobs running Percona's
Innobackupex several times a day). We'd likely move from our current
physical MySQL servers to AWS RDS, so perhaps that would eliminate the past
issues we've encountered...

Thanks for the links and suggestions. I'll investigate them tomorrow.

Justin


On Tue, Oct 28, 2014 at 1:01 AM, Jeremy Baron <jeremy@tuxmachine.com> wrote:

> On Oct 28, 2014 2:56 AM, "Justin Lloyd" <jclbugz@gmail.com> wrote:
> > Again, perhaps there's a better way to architect a resilient set of wikis
> > that would simplify this design, and I'm open to all suggestions, so far
> > what I have is the best I've come up with in the time I've managed these
> > wikis.
>
> You should take a look at how WMF handles this. The wiki farm stuff (aka
> hetdeploy/mwmultiversion) is documented at
> https://wikitech.wikimedia.org/wiki/Heterogeneous_deployment (maybe up to
> date because edited recently)
>
> also:
> *
>
> https://github.com/wikimedia/operations-mediawiki-config/blob/master/multiversion/MWMultiVersion.php
> *
>
> https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/InitialiseSettings.php
> *
>
> https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/CommonSettings.php
>
> (and see other files in that dir too)
>
> WMF essentially runs its own s3. (called swift. An openstack project) I
> don't see why you couldn't use s3 in a similar way. might be overkill for
> you to run your own swift cluster. Rackspace has a public swift cluster you
> could use too.
>
> It doesn't look like there's currently an s3 filebackend but you're welcome
> to add one.
>
> * https://github.com/wikimedia/mediawiki/tree/master/includes/filebackend
> *
>
> https://github.com/wikimedia/operations-mediawiki-config/blob/master/wmf-config/filebackend.php
> * https://wikitech.wikimedia.org/wiki/Media_storage
> * our InitialiseSettings.php: wgUploadPath
>
> In the short term you could load balance reads with a more frequent rsync
> and then it's only spof for writes and rendering file description pages. I
> think. (wgUploadPath points to load balancer for a cluster of machines that
> all have a local copy of all images.)
>
> -Jeremy
> _______________________________________________
> MediaWiki-l mailing list
> To unsubscribe, go to:
> https://lists.wikimedia.org/mailman/listinfo/mediawiki-l
>
_______________________________________________
MediaWiki-l mailing list
To unsubscribe, go to:
https://lists.wikimedia.org/mailman/listinfo/mediawiki-l

[prev in list] [next in list] [prev in thread] [next in thread]