[prev in list] [next in list] [prev in thread] [next in thread] 

List:       gentoo-dev
Subject:    [gentoo-dev] [FYI] Gentoo Service instability June 28 - 29
From:       Alec Warner <antarus () gentoo ! org>
Date:       2021-06-29 3:12:31
Message-ID: CAAr7Pr8P_fCtVbFVA=V2TKewkpb-FiE8P3_kFyukA2bn_ZvRqg () mail ! gmail ! com
[Download RAW message or body]

Hi,

Many of you reported that various infra-owned services were not functioning
properly these past two days. We had some package upgrades roll out that
were bad, and a subset of services were non-functional while we worked
through the incident.

(a) The upgrades did not appear to cause any data loss, but service
availability was reduced (500s / 502s for some HTTP services.)
(b) Some hosts may have had problems running some commands. E.g. I know
that 'grep' didn't work for a while. If you observed errors related to
missing 'GLIBC_2.33' symbols then this was likely related.
(c) Many replicated services (including infra-status) rely on gitweb to
take updates from git; and gitweb was not up, so they were unable to
receive updates.
(d) More impacted notes will follow in a postmortem that will come later
this week for this incident.

Thanks to all of you who reported problems and apologies for the service
disruptions.

-A

[Attachment #3 (text/html)]

<div dir="ltr">Hi,<div><br></div><div>Many of you reported that various infra-owned \
services were not functioning properly these past two days. We had some package \
upgrades roll out that were bad, and a subset of services were non-functional while \
we worked through the incident.</div><div><br></div><div>(a) The upgrades did not \
appear  to cause any data loss, but service availability was reduced (500s / 502s for \
some HTTP services.)</div>(b) Some hosts may have had problems running some commands. \
E.g. I know that &#39;grep&#39; didn&#39;t work for a while. If you observed errors \
related to missing &#39;GLIBC_2.33&#39; symbols then this was likely related.<div>(c) \
Many replicated services (including infra-status) rely on gitweb to take updates from \
git; and gitweb was not up, so they were unable to receive updates.</div><div>(d) \
More impacted notes will follow in a postmortem that will come later this week for \
this incident.<div><br></div><div>Thanks to all of you who reported problems and \
apologies for the service disruptions.</div></div><div><br></div><div>-A</div></div>



[prev in list] [next in list] [prev in thread] [next in thread]