[prev in list] [next in list] [prev in thread] [next in thread]
List: wikitech-l
Subject: [Wikitech-l] Re: Infrastructure diagrams
From: Selena Deckelmann <sdeckelmann () wikimedia ! org>
Date: 2022-10-24 23:21:51
Message-ID: CA+tSke38it4-UAXqCWk0=a9qY7yoK-D4_Lu9VQ8crQp_SMDUAQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Thanks so much, Timo! Having these snapshots of our systems and
infrastructure, and also capturing how it evolves over time, is invaluable.
-selena
On Mon, Oct 24, 2022 at 6:51 AM Krinkle <krinkle@fastmail.com> wrote:
> I've done a major update to a number of diagrams on Wikitech.
>
> Usually, I don't mention an update here, but I'm highlighting it now as
> it's been a while since we mentioned them on-list and the community and
> foundation have grown a lot so some of these may be new to you.
>
> Given how much has changed in recent changes, I also included a changelog
> and a link to where in the docs you'd normally discover this diagram
> on-wiki:
>
> *== 1. File:Wikipedia_webrequest_2022.png
> <https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png>
> (Updated) ==*
>
> This is a highly simplified diagram, covering the general shape of our
> stack through the example of a typical Wikipedia webrequest.
>
> Previous:
> https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flow_2020.png
> New:
> https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_2022.png
> Documentation: wikitech:MediaWiki_at_WMF
> <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF> and
> wikitech:Caching_overview
> <https://wikitech.wikimedia.org/wiki/Caching_overview>.
> Notable changes:
> * Change edge TLS termination ("HTTPS") from ats-tls to HAProxy. I wrote a
> "Caching overview § History
> <https://wikitech.wikimedia.org/wiki/Caching_overview>" section.
> * Change appserver TLS from Nginx- to Envoy.
> * Add new MainStash DB.
> * Include storage ExternalStore DB, ParserCache DB, and Swift media.
> * Include services Shellbox, Mathoid, and Kask.
>
> *== 2. File:WMF_infrastructure_2022.png
> <https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png>
> (Updated) ==*
>
> This is a continous attempt at an overview of tier-1/user-facing
> infrastructure. It will likely never be complete from all POV, but.. it is
> more accurate and complete than it has been. Thanks to all that contributed
> by entertaining my many questions over the years.
>
> Previous (2016 by Elukey):
> https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png
> New:
> https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.png
> Documentation: wikitech:Wikimedia_infrastructure
> <https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure> and
> wikitech:Purged <https://wikitech.wikimedia.org/wiki/Purged>
> Notable changes:
> * Add new Drmrs data center in Marseille, France.
> * Add new services: purged.go, EventStreams, Thumbor, mcrouter, Envoy,
> etcd.
> * Add new distinction for Multi-DC between primary and secondary data
> center.
> * Change sessionstore from Redis to Kask/Cassandra.
> * Change jobqueue from Redis to EventGate/Kafka.
> * Include distinct MediaWiki server roles and clusters.
> * Include high-level MediaWiki platform components.
> * Include example flow for "JobQueue job" and "CDN purge".
>
> *== 3. File:MediaWiki_infrastructure_2022.png
> <https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png>
> (New) ==*
>
> Similar to WMF Infra diagram, but more abstract around DC and services,
> and more detailed within the platform. Including more core services, and
> recognising extensions as their own layer.
>
> New:
> https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure_2022.png
> Documentation: wikitech:MediaWiki_at_WMF
> <https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF>
>
> *== 4. File:Wikipedia_Memcached_flow_2022.png
> <https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png>
> (Updated)*
>
> Previous:
> https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow_2020.png
> New:
> https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow_2022.png
> Documentation: wikitech:Memcached_for_MediaWiki
> <https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki>
> Notable changes:
> * Include the three tiers of ParserCache.
> * Add WANCache legend to explain different keytypes you may encounter on
> the network.
> * Add full name of the mcrouter-with-onhost-tier service for greppability.
> * Add new WRStats service (T310662
> <https://phabricator.wikimedia.org/T310662>). This was part of Multi-DC
> work
> <https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki>
> to reduce primary DB writes and (not bi-di replicated) Redis use in
> AbuseFilter. This service also replaces the old "User ping limiter" in core
> and is now able to serve both use cases.
> * Remove "on-host: soon" labels. Adopting on-host memc for WANCache was
> considered not worth the added runtime complexity (T264604
> <https://phabricator.wikimedia.org/T264604>). Note that SRE's work on
> adding 10G network links for memcached hosts, and the addition of
> mcrouter-managed gutter pools take care of the general usecase that we were
> exploring on-host for. We kept it for ParserCache however (T244340
> <https://phabricator.wikimedia.org/T244340>.
>
> *== Edit link ==*
>
> As before, each diagram file page has an "Edit" link in the description
> that takes you directly to the open-source Diagrams.net web app (loading
> file read-only from Google Drive). You can fork by using "Save as" in the
> web app. See also: wikitech:Performance/Runbook/diagrams.net_conventions
> <https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions>
>
> -- Timo
>
>
> _______________________________________________
> Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
> To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
> https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
>
[Attachment #5 (text/html)]
<div dir="ltr"><div>Thanks so much, Timo! Having these snapshots of our systems and \
infrastructure, and also capturing how it evolves over time, is \
invaluable.</div><div><br></div><div>-selena<br></div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 24, 2022 at 6:51 AM \
Krinkle <<a href="mailto:krinkle@fastmail.com">krinkle@fastmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div \
class="msg7591804222657934669"><u></u><div><div dir="ltr"><div>I've done a major \
update to a number of diagrams on Wikitech.<br></div><div><br></div><div>Usually, I \
don't mention an update here, but I'm highlighting it now as it's been a \
while since we mentioned them on-list and the community and foundation have grown a \
lot so some of these may be new to you.<br></div><div><br></div><div>Given how much \
has changed in recent changes, I also included a changelog and a link to where in the \
docs you'd normally discover this diagram \
on-wiki:<br></div><div><br></div><div><b>== 1. <a \
href="https://wikitech.wikimedia.org/wiki/File:Wikipedia_webrequest_2022.png" \
target="_blank">File:Wikipedia_webrequest_2022.png</a> (Updated) \
==</b><br></div><div><br></div><div>This is a highly simplified diagram, covering the \
general shape of our stack through the example of a typical Wikipedia \
webrequest.<br></div><div><b></b><br></div><div>Previous: <a \
href="https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flow_2020.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/b/b3/Wikipedia_webrequest_flow_2020.png</a><br></div><div>New: \
<a href="https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_2022.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/4/4d/Wikipedia_webrequest_2022.png</a><br></div><div>Documentation: \
<a href="https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF" \
target="_blank">wikitech:MediaWiki_at_WMF</a> and <a \
href="https://wikitech.wikimedia.org/wiki/Caching_overview" \
target="_blank">wikitech:Caching_overview</a>.<br></div><div><div>Notable \
changes:<br></div><div>* Change edge TLS termination ("HTTPS") from ats-tls \
to HAProxy. I wrote a "<a \
href="https://wikitech.wikimedia.org/wiki/Caching_overview" target="_blank">Caching \
overview § History</a>" section.<span><br>* Change appserver TLS from Nginx- to \
Envoy.<br>* Add new MainStash DB.<br>* Include storage ExternalStore DB, ParserCache \
DB, and Swift media.<br>* Include services Shellbox, Mathoid, and \
Kask.</span></div></div><div><div><br></div><div><b>== 2. <a \
href="https://wikitech.wikimedia.org/wiki/File:WMF_infrastructure_2022.png" \
target="_blank">File:WMF_infrastructure_2022.png</a> <b>(Updated) \
==</b></b><br></div><div><br></div><div>This is a continous attempt at an overview of \
tier-1/user-facing infrastructure. It will likely never be complete from all POV, \
but.. it is more accurate and complete than it has been. Thanks to all that \
contributed by entertaining my many questions over the \
years.<br></div><div><b><b></b></b><br></div><div><div>Previous (2016 by Elukey): <a \
href="https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/labs/4/4d/Infrastructure_overview.png</a><br></div><div>New: \
<a href="https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/4/48/WMF_infrastructure_2022.png</a><br></div></div><div>Documentation: \
<a href="https://wikitech.wikimedia.org/wiki/Wikimedia_infrastructure" \
target="_blank">wikitech:Wikimedia_infrastructure</a> and <a \
href="https://wikitech.wikimedia.org/wiki/Purged" \
target="_blank">wikitech:Purged</a><br></div><div><div>Notable changes:<span><br>* \
Add new Drmrs data center in Marseille, France.<br>* Add new services: purged.go, \
EventStreams, Thumbor, mcrouter, Envoy, etcd.<br></span>* Add new distinction for \
Multi-DC between primary and secondary data center.</div><div>* Change sessionstore \
from Redis to Kask/Cassandra.<span><br>* Change jobqueue from Redis to \
EventGate/Kafka.<br></span>* Include distinct MediaWiki server roles and \
clusters.</div><div>* Include high-level MediaWiki platform \
components.<br></div><div>* Include example flow for "JobQueue job" and \
"CDN purge".<br></div></div><div><br></div></div><div><b>== 3. <a \
href="https://wikitech.wikimedia.org/wiki/File:MediaWiki_infrastructure_2022.png" \
target="_blank">File:MediaWiki_infrastructure_2022.png</a> (New) \
==</b><br></div><div><br></div></div><div>Similar to WMF Infra diagram, but more \
abstract around DC and services, and more detailed within the platform. Including \
more core services, and recognising extensions as their own layer.<br></div><div \
dir="ltr"><div><b></b><br></div><div><div>New: <a \
href="https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure_2022.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/e/ee/MediaWiki_infrastructure_2022.png</a><br></div><div>Documentation: \
<a href="https://wikitech.wikimedia.org/wiki/MediaWiki_at_WMF" \
target="_blank">wikitech:MediaWiki_at_WMF</a><br></div><div><br></div></div><div><b>== \
4. <a href="https://wikitech.wikimedia.org/wiki/File:Wikipedia_Memcached_flow_2022.png" \
target="_blank">File:Wikipedia_Memcached_flow_2022.png</a> \
<b>(Updated)</b></b><br></div><div><b><b></b></b><br></div><div>Previous: <a \
href="https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow_2020.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/d/db/Wikipedia_Memcached_flow_2020.png</a><br></div><div>New: \
<a href="https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow_2022.png" \
target="_blank">https://upload.wikimedia.org/wikipedia/commons/4/45/Wikipedia_Memcached_flow_2022.png</a><br></div><div>Documentation: \
<a href="https://wikitech.wikimedia.org/wiki/Memcached_for_MediaWiki" \
target="_blank">wikitech:Memcached_for_MediaWiki</a><br></div><div>Notable \
changes:<br></div><div>* Include the three tiers of ParserCache.<br></div><div>* Add \
WANCache legend to explain different keytypes you may encounter on the \
network.<br></div><div>* Add full name of the mcrouter-with-onhost-tier service for \
greppability.<br></div><div><div>* Add new WRStats service (<a \
href="https://phabricator.wikimedia.org/T310662" target="_blank">T310662</a>). This \
was part of <a href="https://www.mediawiki.org/wiki/Wikimedia_Performance_Team/Multi-DC_MediaWiki" \
target="_blank">Multi-DC work</a> to reduce primary DB writes and (not bi-di \
replicated) Redis use in AbuseFilter. This service also replaces the old "User \
ping limiter" in core and is now able to serve both use \
cases.<br></div></div><div>* Remove "on-host: soon" labels. Adopting \
on-host memc for WANCache was considered not worth the added runtime complexity (<a \
href="https://phabricator.wikimedia.org/T264604" target="_blank">T264604</a>). Note \
that SRE's work on adding 10G network links for memcached hosts, and the \
addition of mcrouter-managed gutter pools take care of the general usecase that we \
were exploring on-host for. We kept it for ParserCache however (<a \
href="https://phabricator.wikimedia.org/T244340" \
target="_blank">T244340</a>.<br></div><div><br></div></div><div><b>== Edit link \
==</b><br></div><div dir="ltr"><div><br></div><div>As before, each \
<span>diagram</span> file page has an "Edit" link in the description that \
takes you directly to the open-source <span>Diagrams</span>.net web app (loading file \
read-only from Google Drive). You can fork by using "Save as" in the web \
app. See also: <a href="https://wikitech.wikimedia.org/wiki/Performance/Runbook/diagrams.net_conventions" \
target="_blank">wikitech:Performance/Runbook/<span>diagrams</span>.net_conventions</a><br></div></div><span \
style="color:rgb(136,136,136)"></span><div><br></div><div>-- \
Timo<br></div><div><br></div><div><br></div></div>_______________________________________________<br>
Wikitech-l mailing list -- <a href="mailto:wikitech-l@lists.wikimedia.org" \
target="_blank">wikitech-l@lists.wikimedia.org</a><br> To unsubscribe send an email \
to <a href="mailto:wikitech-l-leave@lists.wikimedia.org" \
target="_blank">wikitech-l-leave@lists.wikimedia.org</a><br> <a \
href="https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/" \
rel="noreferrer" target="_blank">https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/</a></div></blockquote></div>
_______________________________________________
Wikitech-l mailing list -- wikitech-l@lists.wikimedia.org
To unsubscribe send an email to wikitech-l-leave@lists.wikimedia.org
https://lists.wikimedia.org/postorius/lists/wikitech-l.lists.wikimedia.org/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic