[prev in list] [next in list] [prev in thread] [next in thread]
List: vdsm-devel
Subject: =?utf-8?q?=5Bovirt-devel=5D?= Fwd: Copr outage - details
From: Sandro Bonazzola <sbonazzo () redhat ! com>
Date: 2020-02-20 9:58:07
Message-ID: CAPQRNTnVGPfYkADVKGHxX32tNQOLJ8fXABWByOpc9_TubHCG6Q () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
FYI
---------- Forwarded message ---------
Da: Miroslav Suchý <msuchy@redhat.com>
Date: gio 20 feb 2020 alle ore 10:28
Subject: Copr outage - details
To: Development discussions related to Fedora <devel@lists.fedoraproject.org
>
tl;dr: On Sunday 23rd February, there will be Copr outage. It will last the
whole day.
PPC64LE builder and chroots will be deactivated. The PPC64LE builders
should be back in a matter of weeks.
Hi.
As previously announced, Fedora's infrastructure is moving to a different
datacenter. For some servers, the move is
trivial. Copr servers are different. Copr build system consists of four
servers, plus four staging servers. Eight TB of
repos, four TB of dist-git, and several small volumes.
The original plan was to move to Washington D.C. to IAD2 datacenter by
June. Copr is running in Fedora OpenStack, and
this cloud has to be evacuated by the beginning of March to free an IP
range.
The plan was to move Copr to new hardware (thanks to Red Hat) and later
move this HW to the new datacenter. That would
mean two outages, where the second one lasted at least 15 days (!).
We were looking for another option and we found it. We are going to move
Copr to Amazon AWS, shutdown old VM on Fedora
Cloud. Move the new HW to IAD2 datacenter and then move Copr from AWS to
new HW in IAD2 - FYI, the final destination is
still subject to change. This still means two outages, but they should be
just a few hours. And web server with DNF
repositories should be available all the time.
The second outage, will happen in May or June.
Here is a detailed schedule. We are going to update this table during
migration. You can watch the progress during
migration:
https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHPK_pocuf4/edit?usp=sharing
Here is a short abstract:
* we are doing constant rsync to the new location
* we spin up staging and production instances in the new location
* on Sunday morning we stop frontend and therefore accepting new jobs. The
backend with DNF repos will still be
operational.
* we do final rsync (~6 hours)
* around 13:00 UTC we switch DNS to the new location
* we then enable all services
* once we confirm that everything is operational, the outage will be over
There are several caveats:
* After we enable services on Sunday 13:00 UTC you may see some failures.
Be asured that we will swiftly address them.
* Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We
are going to deactivate those chroots just
before the migration. After a few weeks, we should get it back. ETA is
unknown. The worst-case scenario is in June 2020.
We will be aiming to bring it back as soon as possible.
* Any small issue can easily change the schedule by hours. E.g., just
simple 'chown -R' on backend runs ~4 hours.
There are going to be three Copr engineers and one fedora-infrastructure
member available whole Sunday. If you
experienced a problem, do not hesitate to contact us. We are on
#fedora-buildsys on Freenode.
The link to the outage ticket is:
https://pagure.io/fedora-infrastructure/issue/8668
--
Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno,
#fedora-buildsys
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
--
Sandro Bonazzola
MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
Red Hat EMEA <https://www.redhat.com/>
sbonazzo@redhat.com
<https://www.redhat.com/>*Red Hat respects your work life balance.
Therefore there is no need to answer this email out of your office hours.
<https://mojo.redhat.com/docs/DOC-1199578>*
[Attachment #5 (text/html)]
<div dir="ltr">FYI<br><br><div class="gmail_quote"><div dir="ltr" \
class="gmail_attr">---------- Forwarded message ---------<br>Da: <strong \
class="gmail_sendername" dir="auto">Miroslav Suchý</strong> <span dir="auto"><<a \
href="mailto:msuchy@redhat.com">msuchy@redhat.com</a>></span><br>Date: gio 20 feb \
2020 alle ore 10:28<br>Subject: Copr outage - details<br>To: Development discussions \
related to Fedora <<a \
href="mailto:devel@lists.fedoraproject.org">devel@lists.fedoraproject.org</a>><br></div><br><br>tl;dr: \
On Sunday 23rd February, there will be Copr outage. It will last the whole day.<br> \
PPC64LE builder and chroots will be deactivated. The PPC64LE builders should be back \
in a matter of weeks.<br> <br>
Hi.<br>
As previously announced, Fedora's infrastructure is moving to a different \
datacenter. For some servers, the move is<br> trivial. Copr servers are different. \
Copr build system consists of four servers, plus four staging servers. Eight TB \
of<br> repos, four TB of dist-git, and several small volumes.<br>
<br>
The original plan was to move to Washington D.C. to IAD2 datacenter by June. Copr \
is running in Fedora OpenStack, and<br> this cloud has to be evacuated by the \
beginning of March to free an IP range.<br> The plan was to move Copr to new hardware \
(thanks to Red Hat) and later move this HW to the new datacenter. That would<br> \
mean two outages, where the second one lasted at least 15 days (!).<br> <br>
We were looking for another option and we found it. We are going to move Copr to \
Amazon AWS, shutdown old VM on Fedora<br> Cloud. Move the new HW to IAD2 datacenter \
and then move Copr from AWS to new HW in IAD2 - FYI, the final destination is<br> \
still subject to change. This still means two outages, but they should be just a few \
hours. And web server with DNF<br> repositories should be available all the time.<br>
The second outage, will happen in May or June.<br>
<br>
Here is a detailed schedule. We are going to update this table during migration. You \
can watch the progress during<br> migration:<br>
<br>
<a href="https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHPK_pocuf4/edit?usp=sharing" \
rel="noreferrer" target="_blank">https://docs.google.com/spreadsheets/d/1jrCgdhseZwi91CTRlo9Y5DNwfl9VHoZfjHPK_pocuf4/edit?usp=sharing</a><br>
<br>
Here is a short abstract:<br>
<br>
* we are doing constant rsync to the new location<br>
* we spin up staging and production instances in the new location<br>
* on Sunday morning we stop frontend and therefore accepting new jobs. The backend \
with DNF repos will still be<br> operational.<br>
* we do final rsync (~6 hours)<br>
* around 13:00 UTC we switch DNS to the new location<br>
* we then enable all services<br>
* once we confirm that everything is operational, the outage will be over<br>
<br>
There are several caveats:<br>
<br>
* After we enable services on Sunday 13:00 UTC you may see some failures. Be asured \
that we will swiftly address them.<br>
* Once we get out of Fedora Cloud, we lost access to PPC64LE builders. We are going \
to deactivate those chroots just<br> before the migration. After a few weeks, we \
should get it back. ETA is unknown. The worst-case scenario is in June 2020.<br> We \
will be aiming to bring it back as soon as possible.<br>
* Any small issue can easily change the schedule by hours. E.g., just simple \
'chown -R' on backend runs ~4 hours.<br> <br>
There are going to be three Copr engineers and one fedora-infrastructure member \
available whole Sunday. If you<br> experienced a problem, do not hesitate to contact \
us. We are on #fedora-buildsys on Freenode.<br> <br>
The link to the outage ticket is:<br>
<a href="https://pagure.io/fedora-infrastructure/issue/8668" rel="noreferrer" \
target="_blank">https://pagure.io/fedora-infrastructure/issue/8668</a><br> <br>
-- <br>
Miroslav Suchy, RHCARed Hat, Associate Manager ABRT/Copr, #brno, #fedora-buildsys<br>
_______________________________________________<br>
devel mailing list -- <a href="mailto:devel@lists.fedoraproject.org" \
target="_blank">devel@lists.fedoraproject.org</a><br> To unsubscribe send an email to \
<a href="mailto:devel-leave@lists.fedoraproject.org" \
target="_blank">devel-leave@lists.fedoraproject.org</a><br> Fedora Code of Conduct: \
<a href="https://docs.fedoraproject.org/en-US/project/code-of-conduct/" \
rel="noreferrer" target="_blank">https://docs.fedoraproject.org/en-US/project/code-of-conduct/</a><br>
List Guidelines: <a href="https://fedoraproject.org/wiki/Mailing_list_guidelines" \
rel="noreferrer" target="_blank">https://fedoraproject.org/wiki/Mailing_list_guidelines</a><br>
List Archives: <a href="https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org" \
rel="noreferrer" target="_blank">https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org</a><br>
</div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" \
data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div \
dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div \
dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p \
style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:capitalize"><span>Sandro</span> \
<span>Bonazzola</span><span \
style="text-transform:uppercase;color:rgb(170,170,170);margin:0px"></span></p><p \
style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:12px;margin:0px;text-transform:capitalize"><span>MANAGER, \
SOFTWARE ENGINEERING, EMEA R&D RHV</span></p><p \
style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;margin:0px 0px \
4px;font-size:12px"><a href="https://www.redhat.com/" \
style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat \
<span>EMEA</span></a></p><div \
style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;font-size:medium;margin-bottom:4px"></div><p \
style="color:rgb(0,0,0);font-family:RedHatText,sans-serif;margin:0px;font-size:12px"><span \
style="margin:0px;padding:0px"><a href="mailto:sbonazzo@redhat.com" \
style="color:rgb(0,0,0);margin:0px" target="_blank">sbonazzo@redhat.com</a> \
</span></p><div style="margin-top:12px"><table border="0"><tbody><tr><td \
width="100px"><a href="https://www.redhat.com/" target="_blank"><font color="#000000" \
face="RedHatText, sans-serif" size="3"><img \
src="https://marketing-outfit-prod-images.s3-us-west-2.amazonaws.com/f5445ae0c9ddafd5b2f1836854d7416a/Logo-RedHat-Email.png" \
width="90" height="auto"></font></a></td></tr></tbody></table><font color="#000000" \
face="arial, sans-serif" size="1"><b><a \
href="https://mojo.redhat.com/docs/DOC-1199578" target="_blank">Red Hat respects your \
work life balance. Therefore there is no need to answer this email out of your office \
hours.</a></b></font><br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
[Attachment #6 (text/plain)]
_______________________________________________
Devel mailing list -- devel@ovirt.org
To unsubscribe send an email to devel-leave@ovirt.org
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct: https://www.ovirt.org/community/about/community-guidelines/
List Archives: https://lists.ovirt.org/archives/list/devel@ovirt.org/message/O3GIVRLOPE33UA4CQNJ76RI27WSHLK3Z/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic