[prev in list] [next in list] [prev in thread] [next in thread] 

List:       perl5-porters
Subject:    PTS2023 - Multiple Indices Discussion
From:       "Paul \"LeoNerd\" Evans" <leonerd () leonerd ! org ! uk>
Date:       2023-04-30 15:04:43
Message-ID: 20230430160428.3ada413c () shy ! leonerd ! org ! uk
[Download RAW message or body]

(A writeup of a discussion we held at PTS2023)

## The Problem To Be Solved

**Primary usecase**: Authors who want to use new perl syntax/features
in new versions of their modules, without breaking "older perl".
Ideally, allow older perl versions to install prior versions of said
module.

*For example*: Example::Module version 2.5 works on all perl versions.
Author wants to use some new features that appear in 5.36. So, author
bumps version number up to 3.000, sets required perl version to 5.036,
uploads to PAUSE. Ideally, users on older perl versions will still see
version 2.5 if they try to install it.

Secondary usecase: Authors who, having uploaded a newer version of a
module that now has a later minimum perl version requirement, wish to
still release bugfixes for older perl versions. These could be
described as *non-monotonic* releases; releases whose version number is
not a monotonic increase on the previous version.

*For example*: The above author now wishes to fix a bug in
Example::Module version 2.5, so uploads version 2.6. This becomes
available for the older perl versions, while still leaving 3.0
available for those on perl 5.36 or later.

## A Proposed Solution

The overall solution we arrive at is not perfect. It has shortcomings
and limitations. We simply provide it as a "reasonable effort" that
makes things easier for authors to update modules, while also making it
easier for users of older perl versions to still have access to working
modules. We don't claim it solves all problems for all people; it just
has to be better than the current state of affairs.

To support this, we'd need some way to maintain a packages index per
perl version, rather than simply having just one. CPAN client can then
by some mechanism use a more appropriate index tailored to that perl
version.

It is considered infeasible to get PAUSE alone to solve this. We can't
just generate multiple 02packages-VER.txt files, for example.

HAARG is already working on a feature in metacpan that could help this.
The idea is that individual users can publish named "packages
overlays"; a set of changes to make to the actual package index. Each
overlay creates a virtual CPAN mirror that in effect has a modified
index. This would allow an index per perl version to be created, by
means of overlays. These could all live under some new user account
created for this purpose. It doesn't have to be one "official" place.

I (Paul) am unsure exactly what the end-user client process would be
when installing modules through this method, but I imagine in some way
or other a user would configure their CPAN client to use some
alternative mirror/URL/something. Already this feature would be useful
for a bunch of other use-cases besides this one, so it seems a
worth-while thing to create anyway.

This now reduces the problem to a question of how to create those
alternative indexes, now that the above process will solve how to
distribute them to users.

We imagine "some process" (yet to be determined where this runs, by
whom, who maintains it, etc...) that either regularly scans the
"primary" index or by some other means is informed of new distribution
uploads. For each new upload it looks at the required perl version
declared by that module's META.yml. It then enters information about
that distribution into the overlay of that perl version and all later
versions, leaving untouched the prior entries in overlays relating to
older perls.

Thus, over time, the overlay associated with each perl version will
only refer to package distributions that declare via their own metadata
that they work with that version of perl.

Accepting that "we know this isn't perfect", the following observations
can be made. These primarily come from the fact that the index overlays
are a set of changes on top of the official package index.

 * Because in the above description the per-perl-version overlay lists
   are generated by some external process, there will be a delay
   between a new module appearing in the official index, and any
   modifications appearing in or being hidden by the per-version
   overlays. There therefore exists a small window of time after a
   package upload where the overlays might not yet be correct.

 * Package overlays, and the process described above to create them,
   cannot "hide" modules from the index. There is no way as written to
   make a module disappear from an older-perl index. This means that
   new modules that appear in new versions of existing distributions
   would appear in older perl indexes. This doesn't really impact the
   primary use-case for this solution, but it might cause user
   confusion. If it is thought necessary to fix this, a more extensible
   format for package overlays that allows modules to be removed could
   be designed.

One notable advantage of this arrangement is that the majority of
module authors do not have to do anything special. We can construct
indexes out of existing data and maintain them going forward, entirely
by inspecting the minimum perl version declaration that modules already
provide.

## Supporting Non-Monotonic Releases

The above process does not yet solve the secondary use-case; that of
allowing non-monotonic bugfix releases. We discussed a way to solve
this one, that involves more opt-in from the module author to provide
extra data for the per-version indexers to use.

This part of the discussion was much less precisely specified, and
details can be filled in later. In summary: The suggestion was that the
"latest" module meta for a package can (direcrly or indirectly) provide
a mapping from perl versions to versions of its own package, that says
which other version numbers to apply to the per-perl index.

While it *could* be stored directly, that means that in the secondary
use-case described above, the module author would upload an
Example-Module-2.6.tar.gz module release, but then would have to update
the metadata in the module by also releasing an
Example-Module-3.001.tar.gz; a release that might otherwise contain no
changes except to that metadata.

Better then, for the module to indirectly provide this mapping, by
means perhaps of naming another file in some format yet to be
described, that would in that module author's directory on PAUSE.
Additionally, PAUSE would have to allow that file to be replaced
directly

## Summary of Next Actions

 * Create the concept of per-user named "package overlays" on
   (meta)cpan, allowing users to publish alternate indexes

 * Build a process by which a specific (virtual) user can publish
   alternate indexes based on older perl versions

 * Reconstruct some initial overlay indexes, containing "what these
   overlays would have been" had this process always existed

 * *Optionally*: Update `CPAN.pm`, `cpanminus`, et.al. to automatically
   use the older-perl alternate index when appropriate. Or at least,
   offer guidance to users on how they can easily use this.

 * Extension: Define the format and process by which authors can
   declare non-monotonic version releases for older perls

   + This may require changes to PAUSE, to permit users to replace
     certain files of some name pattern with newer content at the same
     name

-- 
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk      |  https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/  |  https://www.tindie.com/stores/leonerd/
[prev in list] [next in list] [prev in thread] [next in thread]