[prev in list] [next in list] [prev in thread] [next in thread]
List: centos-devel
Subject: [CentOS-devel] [EXT] Re: https://blog.centos.org/2020/12/future-is-centos-stream/
From: JD Maloney <jdphotography7 () gmail ! com>
Date: 2020-12-12 18:26:17
Message-ID: CAL+Ow5mOHdRwrqT5DscR4t3cXkOa3qBEvLny-LmnuYteREC2kQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
>On 10/12/2020 16.50, Rich Bowen wrote:
>> >> >>* On 12/9/20 12:32 PM, David Hrb=C3=A1=C4=8D wrote:
*>>>* I don't use CentOS Stream, I use RHEL. I use RHEL to develop soft=
ware
*>>>* for RHEL and compatible OS clones, including CentOS. If Stream
*>>>* retains
*>>>* binary compatibility, and specifically kernel ABI compatibility, =
then
*>>>* the users of the software packages we develop can continue to use
*>>>* them.
*>>>* If not, they can't. Simple as that. So please don't push rolling
*>>>* kernel
*>>>* updates to Stream that break the kernel ABI.
*>>>>>>>>>* Rolling kernel updates are going to kill all the traditional HP=
C
*>>>* clusters. Almost 25% of the TOP 500 HPC clusters run CentOS. See
*>>>* https://www.top500.org/statistics/list/
<https://www.top500.org/statistics/list/>
*>>>* <https://www.top500.org/statistics/list/
<https://www.top500.org/statistics/list/>>
*>> >>* I was under the impression that practically all of those run custom
*>>* kernels anyway, right?
*
> Obviously in no way generally valid: In the last ~5 years I have been
> involved in / responsible for the administration of two systems that
> have been listed in the TOP500 (both lower half of TOP500, but top field
> of Green 500 at their respective time). Both systems are running CentOS
> 7 with vanilla kernel. We only add external 3rd party kernel modules as
> required.
> There have been plans/conversations/ideas about updating the newer
> system to CentOS Linux 8 soon. However, with the changed perspective of
> CentOS, this plan is now dead as well. Depending on how CentOS Stream
> works out (i.e., kernel ABI compatibility to current RHEL minor
> release), switching to CentOS Stream might be an option.
Same here, almost a decade in HPC and have admin'd multiple systems in
the Top 100, attended SC conferences regularly, as well as other
gatherings in the HPC community, all the systems I know of personally,
and I know of a few in the Top 10; dozens in the Top 100, currently
run vanilla kernels. While I can't speak for all of the vast amount
of Supercomputing centers, from all my experience I don't know of many
places that run custom kernels.
>> Can you elaborate on what you mean by rolling kernel updates? If you
>> mean wholesale kernel rebases (e.g. kernel-5.8 -> kernel-5.9), that's
>> not what Stream will be doing.
>> Stream will include more frequent updates of the *RHEL* kernel, where
>> we take the RHEL kABI whitelist very seriously. It is literally the
>> RHEL kernel. In the context of HPC deployments, they could use Stream
>> and choose when to update and reboot those machines based on the
>> kernel updates that come out, just as the would with CentOS Linux.
This isn't possible, I'll use an example from IBM as maybe that'll
help illustrate it better. When RHEL 8.3 was released (for example
but same applies to 7.8, 7.9, 8.1, 8.2, etc.), we couldn't just
upgrade to it as the kernel changes break the ability to build the
kernel module for IBM Spectrum Scale -- the cluster's main shared file
system (eg. running mmbuildgpl would fail, you would have to downgrade
the kernel to the prior .x release's version to get it to run); and
the Lustre project is in the same boat; btw these two file systems
service the VAST majority of HPC systems around the world. We also
have to build Mellanox (Nvidia Networking) OFED against the kernel and
use their drivers that are released with RHEL updates. Upgrading the
kernel to something ahead of the latest official RHEL release (eg. the
current, what will be the 8.4 kernel, in Stream) would break both
cluster networking and storage pretty much every time, it has every .x
--> .(x+1) release I've ever seen over the past 4 years, which would
render the entire cluster unusable. Thus I just do not see a way as
you guys describe it where Stream could ever be a viable option as we
need the kernel to freeze, have vendors get their drivers and software
to work with it, THEN deploy it. What is in Stream will always be too
far ahead of what the vendors have available and built/tested against.
We could use yum/dnf to lock the kernel versions (and some other
supporting packages) but that could have other implications and adds a
lot of fragility as one is essentially trying to un-stream a stream
release.
This move has alienated the HPC community from CentOS, and the naivety
that RH folks are showing about the HPC ecosystem is even more
saddening. HPC users that are on CentOS 7 will likely get stuck there
for a bit, while they re-evaluate offerings and look for where to jump
to next, which is sad as 8 has/had a lot of nice things. Those who
have already jumped to CentOS 8 are now in a very tricky spot, which
can not be understated. Red Hat's lack of plan to go along with this
announcement is mind boggling, something I'd expect from some Silicon
Valley startup, not a seasoned industry player. A "hang tight we'll
get back to you on a new offering in 1H 2021" (and if you don't
like/can't afford our offer congrats you now have 6 months to lift and
shift your entire environment and the dozens to hundreds of end-user
applications you support) is an insult to folks who run these systems.
I completely get the benefits of CentOS Stream for RHEL development,
it makes a lot of sense and for that I think Stream is a great thing.
Also likely a good thing for the hyperscalers who likely run most, if
not all, their stuff in user space and have large development budgets
to support their operations. Though I've yet to see an answer from RH,
about why both CentOS proper and Stream couldn't continue as they were
since they aren't mutually exclusive. We're left to assume it's about
extracting more money from users and (an attempt at) forcing more
community/vendor involvement further upstream.
JD
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="ltr"><div dir="ltr"><pre \
style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">>On \
10/12/2020 16.50, Rich Bowen wrote: >><i style="box-sizing:border-box">
</i>>><i style="box-sizing:border-box">
</i>>><i style="box-sizing:border-box"> On 12/9/20 12:32 PM, David Hrbáč \
wrote: </i>>>><i style="box-sizing:border-box"> I don't use \
CentOS Stream, I use RHEL. I use RHEL to develop software </i>>>><i \
style="box-sizing:border-box"> for RHEL and compatible OS clones, including \
CentOS. If Stream </i>>>><i style="box-sizing:border-box"> retains
</i>>>><i style="box-sizing:border-box"> binary compatibility, and \
specifically kernel ABI compatibility, then </i>>>><i \
style="box-sizing:border-box"> the users of the software packages we develop \
can continue to use </i>>>><i style="box-sizing:border-box"> them.
</i>>>><i style="box-sizing:border-box"> If not, they can't. \
Simple as that. So please don't push rolling </i>>>><i \
style="box-sizing:border-box"> kernel </i>>>><i \
style="box-sizing:border-box"> updates to Stream that break the kernel ABI. \
</i>>>><i style="box-sizing:border-box"> </i>>>><i \
style="box-sizing:border-box"> </i>>>><i style="box-sizing:border-box"> \
Rolling kernel updates are going to kill all the traditional HPC </i>>>><i \
style="box-sizing:border-box"> clusters. Almost 25% of the TOP 500 HPC clusters run \
CentOS. See </i>>>><i style="box-sizing:border-box"> <a \
href="https://www.top500.org/statistics/list/" \
style="box-sizing:border-box;color:rgb(0,123,255);text-decoration:none">https://www.top500.org/statistics/list/</a> \
</i>>>><i style="box-sizing:border-box"> <<a \
href="https://www.top500.org/statistics/list/" \
style="box-sizing:border-box;color:rgb(0,123,255);text-decoration:none">https://www.top500.org/statistics/list/</a>>
</i>>><i style="box-sizing:border-box">
</i>>><i style="box-sizing:border-box"> I was under the impression that \
practically all of those run custom </i>>><i style="box-sizing:border-box"> \
kernels anyway, right? </i>
> Obviously in no way generally valid: In the last ~5 years I have been
> involved in / responsible for the administration of two systems that
> have been listed in the TOP500 (both lower half of TOP500, but top field
> of Green 500 at their respective time). Both systems are running CentOS
> 7 with vanilla kernel. We only add external 3rd party kernel modules as
> required.
> There have been plans/conversations/ideas about updating the newer
> system to CentOS Linux 8 soon. However, with the changed perspective of
> CentOS, this plan is now dead as well. Depending on how CentOS Stream
> works out (i.e., kernel ABI compatibility to current RHEL minor
> release), switching to CentOS Stream might be an option.
</pre><pre style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">Same \
here, almost a decade in HPC and have admin'd multiple systems in the Top 100, \
attended SC conferences regularly, as well as other gatherings in the HPC community, \
all the systems I know of personally, and I know of a few in the Top 10; dozens in \
the Top 100, currently run vanilla kernels. While I can't speak for all of the \
vast amount of Supercomputing centers, from all my experience I don't know of \
many places that run custom kernels.</pre></div><div dir="ltr"><br></div><div \
dir="ltr"><pre style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">>> \
Can you elaborate on what you mean by rolling kernel updates? If you >> mean \
wholesale kernel rebases (e.g. kernel-5.8 -> kernel-5.9), that's >> not \
what Stream will be doing.
>> Stream will include more frequent updates of the *RHEL* kernel, where
>> we take the RHEL kABI whitelist very seriously. It is literally the
>> RHEL kernel. In the context of HPC deployments, they could use Stream
>> and choose when to update and reboot those machines based on the
>> kernel updates that come out, just as the would with CentOS Linux.</pre><pre \
style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">This \
isn't possible, I'll use an example from IBM as maybe that'll help \
illustrate it better. When RHEL 8.3 was released (for example but same applies to \
7.8, 7.9, 8.1, 8.2, etc.), we couldn't just upgrade to it as the kernel changes \
break the ability to build the kernel module for IBM Spectrum Scale -- the \
cluster's main shared file system (eg. running mmbuildgpl would fail, you would \
have to downgrade the kernel to the prior .x release's version to get it to run); \
and the Lustre project is in the same boat; btw these two file systems service the \
VAST majority of HPC systems around the world. We also have to build Mellanox \
(Nvidia Networking) OFED against the kernel and use their drivers that are released \
with RHEL updates. Upgrading the kernel to something ahead of the latest official \
RHEL release (eg. the current, what will be the 8.4 kernel, in Stream) would break \
both cluster networking and storage pretty much every time, it has every .x --> \
.(x+1) release I've ever seen over the past 4 years, which would render the \
entire cluster unusable. Thus I just do not see a way as you guys describe it where \
Stream could ever be a viable option as we need the kernel to freeze, have vendors \
get their drivers and software to work with it, THEN deploy it. What is in Stream \
will always be too far ahead of what the vendors have available and built/tested \
against. We could use yum/dnf to lock the kernel versions (and some other supporting \
packages) but that could have other implications and adds a lot of fragility as one \
is essentially trying to un-stream a stream release. </pre><pre \
style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">This \
move has alienated the HPC community from CentOS, and the naivety that RH folks are \
showing about the HPC ecosystem is even more saddening. HPC users that are on CentOS \
7 will likely get stuck there for a bit, while they re-evaluate offerings and look \
for where to jump to next, which is sad as 8 has/had a lot of nice things. Those who \
have already jumped to CentOS 8 are now in a very tricky spot, which can not be \
understated. Red Hat's lack of plan to go along with this announcement is mind \
boggling, something I'd expect from some Silicon Valley startup, not a seasoned \
industry player. A "hang tight we'll get back to you on a new offering in \
1H 2021" (and if you don't like/can't afford our offer congrats you now \
have 6 months to lift and shift your entire environment and the dozens to hundreds of \
end-user applications you support) is an insult to folks who run these systems. \
</pre><pre style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;color:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">I \
completely get the benefits of CentOS Stream for RHEL development, it makes a lot of \
sense and for that I think Stream is a great thing. Also likely a good thing for the \
hyperscalers who likely run most, if not all, their stuff in user space and have \
large development budgets to support their operations. Though I've yet to see an \
answer from RH, about why both CentOS proper and Stream couldn't continue as they \
were since they aren't mutually exclusive. We're left to assume it's \
about extracting more money from users and (an attempt at) forcing more \
community/vendor involvement further upstream.</pre><pre \
style="box-sizing:border-box;font-family:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation \
Mono","Courier \
New",monospace;font-size:14px;margin-top:0px;margin-bottom:1rem;overflow:auto;col \
or:rgb(33,37,41);word-wrap:break-word;white-space:pre-wrap">JD</pre></div></div></div>
_______________________________________________
CentOS-devel mailing list
CentOS-devel@centos.org
https://lists.centos.org/mailman/listinfo/centos-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic