[prev in list] [next in list] [prev in thread] [next in thread]
List: python-distutils-sig
Subject: [Distutils] Idea: perennial manylinux tag
From: Nathaniel Smith <njs () pobox ! com>
Date: 2018-11-30 8:09:52
Message-ID: CAPJVwBnyKSkbnL7sVGO-Hm9z0f3VfsLuxVN5MXezxtDJw5smAQ () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi all,
The manylinux1 -> manylinux2010 transition has turned out to be very
difficult. Timeline so far:
March 2017: CentOS 5 went EOL
April 2018: PEP 517 accepted
May 2018: support for manylinux2010 lands in warehouse
November 2018: support lands in auditwheel, and pip master
December 2018: 21 months after CentOS 5 EOL, wwee still don't have an
official build environment, or support in a pip release
We'll get through this, but it's been super painful and maybe we can change
things somehow so it will suck less next time.
We don't have anything like this pain on Windows or macOS. We never have to
update pip, warehouse, etc., after those OSes hit EOLs. Why not?
On Windows, we have just two tags: "win32" and "win_amd64". These are
defined to mean something like "this wheel will run on any recent-ish
Windows system". So the meaning of the tag actually changes over time: it
used to be that if a wheel said it ran on win32, then that meant it would
work on winxp, but since winxp hit EOL people started uploading "win32"
wheels that don't work on winxp, and that's worked fine.
On macOS, the tags look like "macosx_10_9_x86_64". So here we have the OS
version embedded in the tag. This means that we do occasionally switch
which tags we're using, kind of like how manylinux1 -> manylinux2010 is
intended to work. But, unlike for the manylinux tags, defining a new macosx
tag is totally trivial: every time a new OS version is released, the tag
springs into existence without any human intervention. Warehouse already
accepts uploads with this tag; pip already knows which systems can install
wheels with this tag, etc.
Can we take any inspiration from this for manylinux?
We could do the Windows thing, and have a plain "manylinux" tag that means
"any recent-ish glibc-based Linux". Today it would be defined to be "any
distro newer than CentOS 6". When CentOS 6 goes out of service, we could
tweak the definition to be "any distro newer than CentOS 7". Most parts of
the toolchain wouldn't need to be updated, though, because the tag wouldn't
change, and by assumption, enforcement wouldn't really be needed, because
the only people who could break would be ones running on unsupported
platforms. Just like happens on Windows.
We could do the macOS thing, and have a "manylinux_${glibc version}" tag
that means "this package works on any Linux using glibc newer than ${glibc
version}". We're already using this as our heuristic to handle the current
manylinux profiles, so e.g. manylinux1 is effectively equivalent to
manylinux_2_5, and manylinux2010 will be equivalent to manylinux_2_12. That
way we'd define the manylinux tags once, get support into pip and warehouse
and auditwheel once, and then in the future the only thing that would have
to change to support new distro releases or new architectures would be to
set up a proper build environment.
What do y'all think?
-n
[Attachment #5 (text/html)]
<div dir="ltr"><div dir="auto">Hi all,<div dir="auto"><br></div><div dir="auto">The \
manylinux1 -> manylinux2010 transition has turned out to be very difficult. \
Timeline so far:</div><div dir="auto"><br></div><div dir="auto">March 2017: CentOS 5 \
went EOL</div><div dir="auto">April 2018: PEP 517 accepted</div><div dir="auto">May \
2018: support for manylinux2010 lands in warehouse</div></div><div>November 2018: \
support lands in auditwheel, and pip master<br></div><div>December 2018: 21 months \
after CentOS 5 EOL, wwee still don't have an official build environment, or \
support in a pip release<br></div><div dir="auto"><br><div dir="auto">We'll get \
through this, but it's been super painful and maybe we can change things somehow \
so it will suck less next time.</div><div dir="auto"><br></div><div dir="auto">We \
don't have anything like this pain on Windows or macOS. We never have to update \
pip, warehouse, etc., after those OSes hit EOLs. Why not?</div><div \
dir="auto"><br></div><div dir="auto">On Windows, we have just two tags: \
"win32" and "win_amd64". These are defined to mean something like \
"this wheel will run on any recent-ish Windows system". So the meaning of \
the tag actually changes over time: it used to be that if a wheel said it ran on \
win32, then that meant it would work on winxp, but since winxp hit EOL people started \
uploading "win32" wheels that don't work on winxp, and that's \
worked fine.</div><div dir="auto"><br></div><div dir="auto">On macOS, the tags look \
like "macosx_10_9_x86_64". So here we have the OS version embedded in the \
tag. This means that we do occasionally switch which tags we're using, kind of \
like how manylinux1 -> manylinux2010 is intended to work. But, unlike for the \
manylinux tags, defining a new macosx tag is totally trivial: every time a new OS \
version is released, the tag springs into existence without any human intervention. \
Warehouse already accepts uploads with this tag; pip already knows which systems can \
install wheels with this tag, etc.</div><div dir="auto"><br></div><div>Can we take \
any inspiration from this for manylinux?</div><div><br></div><div>We could do the \
Windows thing, and have a plain "manylinux" tag that means "any \
recent-ish glibc-based Linux". Today it would be defined to be "any distro \
newer than CentOS 6". When CentOS 6 goes out of service, we could tweak the \
definition to be "any distro newer than CentOS 7". Most parts of the \
toolchain wouldn't need to be updated, though, because the tag wouldn't \
change, and by assumption, enforcement wouldn't really be needed, because the \
only people who could break would be ones running on unsupported platforms. Just like \
happens on Windows.</div><div><br></div><div>We could do the macOS thing, and have a \
"manylinux_${glibc version}" tag that means "this package works on any \
Linux using glibc newer than ${glibc version}". We're already using this as \
our heuristic to handle the current manylinux profiles, so e.g. manylinux1 is \
effectively equivalent to manylinux_2_5, and manylinux2010 will be equivalent to \
manylinux_2_12. That way we'd define the manylinux tags once, get support into \
pip and warehouse and auditwheel once, and then in the future the only thing that \
would have to change to support new distro releases or new architectures would be to \
set up a proper build environment.<br></div><div><br></div><div>What do y'all \
think?<br></div><div><br></div><div>-n<br></div></div> </div>
--
Distutils-SIG mailing list -- distutils-sig@python.org
To unsubscribe send an email to distutils-sig-leave@python.org
https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
Message archived at https://mail.python.org/archives/list/distutils-sig@python.org/message/6AFS4HKX6PVAS76EQNI7JNTGZZRHQ6SQ/
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic