[prev in list] [next in list] [prev in thread] [next in thread] 

List:       fedora-devel-list
Subject:    F34 Change: Reduce installation media size by improving the compression ratio of SquashFS filesystem
From:       Ben Cotton <bcotton () redhat ! com>
Date:       2020-08-27 15:13:26
Message-ID: CA+voJeUMd-JOs6mwgEBBK92zqyhbRa7b4OUtYYG=8Oj7V5H_0A () mail ! gmail ! com
[Download RAW message or body]

https://fedoraproject.org/wiki/Changes/OptimizeSquashFS

== Summary ==
Improve compression ratio of SquashFS filesystem on the installation media.

== Owner ==
* Name: [[User:bkhomuts|Bohdan Khomutskyi]]
* Email: bkhomuts@redhat.com


== Detailed Description ==
As of Fedora 31, the LiveOS/squashfs.img file on the installation
image, is compressed with default block size of mksquashfs. The
default block size is 128k. Additionaly, lorax sets BCJ filter
depending on the architecture. Those parameters can be adjusted which
will lead to a better compression ratio and/or reduction of the CPU
usage at build time.


This is simple to achieve. Recently, Lorax has gotten support for
adjusting the compression options for mksquashfs via the configuration
file. The file should be altered as following:
<pre>
[compression]
bcj = no
args = -b 1M -Xdict-size 1M -no-recovery
</pre>
Where -b 1M and -Xdict-size 1M are block and dictionary sizes
respectively; bcj -- branch-call-jump filter.

Based on the results above, I'd suggest selecting the following
''optimal configuration'': XZ algorithm, with block size of 1MiB and
without BCJ filter (plain xz -b 1M, without -Xbcj x86).
On the right, you can see the impact of the compression algorithms on
installation time.

As can be seen from the picture on the right hand side, selecting
'plain xz -b 1M configuration' has minimal impact on the installation
time and CPU usage during the installation. The compression will
result in +6.51% or, in real terms, +24.94s additional installation
time, compared to the savings of 142 MiB on the installation media,
refer to the documentation section to download the ISO images. This
increase in installation time will be compensated by the change in the
installer: https://github.com/rhinstaller/anaconda/pull/2292
I noticed, that even with maximum compression, CPU is not fully
utilized during installation.


== Benefit to Fedora ==
* Reduction of the installation media size and the cost of storing and
distributing Fedora.
* Reduction of the CPU usage at build time. Depending on which
compression parameters chosen.

== Scope ==
* Proposal owners:
The build environment should have support for adjusting the Lorax
configuration file and -squashfs-only parameter. Lorax is a program
that produces the LiveOS/squashfs.img file on the installation media.

One of the ways to enable such customization is to introduce support
in Pungi to pass -c option to Lorax.

* Other developers:
The pungi utility should support passing the custom configuration file
location to the Lorax utility. This option should apply during
buildInstall phase of pungi.
* Release engineering: [https://pagure.io/releng/issue/9127]
<!-- Does this feature require coordination with release engineering
(e.g. changes to installer image generation or update package
delivery)?  Is a mass rebuild required?  include a link to the releng
issue.
The issue is required to be filed prior to feature submission, to
ensure that someone is on board to do any process development work and
testing, and that all changes make it into the pipeline; a bullet
point in a change is not sufficient communication -->
* Policies and guidelines: Not required.
* Trademark approval: N/A (not needed for this Change)


== Upgrade/compatibility impact ==
N/A (not a System Wide Change)

== How To Test ==
<pre>
mkdir -p /mnt/new /mnt/old
sudo mount -o loop,ro FedoraInstallationOld.iso
sudo mount -o loop,ro FedoraInstallationNew.iso
ls -l /mnt/{new,old}/LiveOS/squashfs.img
</pre>
And then calculate the size difference.

== User Experience ==
* Decreasing the installation image size will reduce cost of mirroring
and storing Fedora installation images.
* Decreasing the installation image size will reduce the download time.
* Increasing the block size on the current configuration with EXT4
file system, should increase latency while accessing the EXT4
filesystem. The exact impact is to be evaluated.
* The impact of latency will be reduced, if the plain SquashFS option
is be choosen.

== Dependencies ==
Pungi, a utility that builds the compose, should include new
functionality mentioned above.
Alternatively, the /etc/lorax/lorax.conf should be altered in the
environment where Lorax is running.

== Contingency Plan ==
N/A

== Documentation ==
https://pagure.io/releng/issue/9127.<br/>
mksquashfs(1)<br/>
lorax(1)<br/>
https://docs.pagure.org/pungi<br/>
Select ISOs can be downloaded at https://khomutsky.com/fedora-dvd/<br/>
Spreadsheet pictured above [[File:Comparison Table SquashFS.ods]]
[https://github.com/rhinstaller/anaconda/pull/2292 Multi-core
decompression of SquashFS merge request]


-- 
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

[prev in list] [next in list] [prev in thread] [next in thread]