[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fedora-devel-list
Subject:    Re: Small rant: installer environment size
From:       Zbigniew =?utf-8?Q?J=C4=99drzejewski-Szmek?= <zbyszek () in ! waw ! pl>
Date:       2022-12-10 11:59:50
Message-ID: Y5R0tv6/cnWIS0kP () kawka3 ! in ! waw ! pl
[Download RAW message or body]

On Fri, Dec 09, 2022 at 03:08:19PM -0700, Chris Murphy wrote:
> 
> 
> On Fri, Dec 9, 2022, at 7:30 AM, Ray Strode wrote:
> > Hi,
> > 
> > On Thu, Dec 8, 2022 at 2:55 PM Adam Williamson
> > <adamwill@fedoraproject.org> wrote:
> > > This is the direction Daniel was thinking down. I'm waiting for someone
> > > with more expertise to reply, but I suspect the reply is going to be
> > > along the lines of "yes, we *can* do that, but it's somewhat tricky
> > > work that involves thinking about lots of paths that aren't obvious,
> > > and somebody would need to dedicate their time to working on that".
> > Presumably we could package the firmware separately and just unpack it
> > into place from a udev rule when the hardware is detected?
> > 
> > But first, do we actually know this is a problem?
> > I think you're saying squashfs loads the whole decompressed image into
> > memory, but my expectation prior to your mail was that it performs I/O
> > on the usb stick (with a cache in between).  If my intuition was right
> > and files only hit ram when accessed, then it seems like this is
> > pretty much not an issue, right?
> 
> From a certain point of view there's a potential inefficiency with squashfs reads \
> in that there's a minimum block size that it needs to read in order decompress its \
> 128 KiB block. It's possible quite a lot of what's decompressed isn't (immediately) \
> needed. But it's still a random access file system. It's not necessary to read the \
> whole image into RAM. 
> Repo metadata is the big hit for netinstall because it's downloaded into /tmp which \
> is tmpfs. And DVD already has repomd on it, and only downloads more if you enable \
> some other repo. Live doesn't need repomd. 
> So initially netinstaller uses less memory up until the the Anaconda language \
> selection screen appears, at which point it starts background downloading repomd. \
> It quickly catches up to, and surpasses, Live media memory consumption.

We *could* do something about repo metadata: only install the "main" metadata,
and not the "filepath" metadata. This would reduce the metadata size by ~80%.
It'd also have huge benefits for speed: on small dnf operations a significant
portion of time is spent just loading metadata.

We had some discussions on this a few years ago, but this never went anywhere.
(I did a quick search, but can't find the ticket now.)  But the idea was that
dnf would load "main" metadata by default, and then load "filepath" metadata
if needed and restart the transaction. Our Packaging Guidelines forbid filepath
dependencies (except for a small set of directories which are in the "main" part),
so normal rpm transactions don't need the "filepath" metadata. It is only
needed for non-confirming rpms and when the user does something like
'dnf install /some/strange/path'.

If we did the split, we'd gain nice speedup on typical dnf invocations, and
smaller memory usage, and things would be better in general. A hero who
groks dnf and the underlying libraries and can tackle this would be needed.

Zbyszek
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
 Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue


[prev in list] [next in list] [prev in thread] [next in thread]