[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ubuntu-devel
Subject:    zstd compression for packages
From:       Julian Andres Klode <julian.klode () canonical ! com>
Date:       2018-03-12 10:06:11
Message-ID: 20180312100611.6ewl2fh57cw4amfo () jak-x230
[Download RAW message or body]

Hey folks,

We had a coding day in Foundations last week and Balint and Julian added support for \
zstd compression to dpkg [1] and apt [2].

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664
[2] https://salsa.debian.org/apt-team/apt/merge_requests/8

Zstd is a compression algorithm developed by Facebook that offers far 
higher decompression speeds than xz or even gzip (at roughly constant 
speed and memory usage across all levels), while offering 19 compression 
levels ranging from roughly comparable to gzip in size (but much faster) 
to 19, which is roughly comparable to xz -6:

In our configuration, we run zstd at level 19. For bionic main amd64, 
this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB. 
Installs speed up by about 10%, or, if eatmydata is involved, by up to 
40% - user time generally by about 50%.

Our implementations for apt and dpkg support multiple frames as used by 
pzstd, so packages can be compressed and decompressed in parallel 
eventually.

We are considering requesting a FFe for that - the features are not 
invasive, and it allows us to turn it on by default in 18.10.

Thanks,
Balint and Julian

Raw Measurements
===============
All measurements where performed on a cloud instance of bionic, in a basic bionic \
schroot with overlay, on an ssd.

Kernel install (eatmydata, perf report, time spent in compression)
---------------------------------------------------------------------------------------
                
Before:  54.79%  liblzma.so.5.2.2
After:  11.04%  libzstd.so.1.3.3

Kernel install (eatmydata)
----------------------------------

12.49user 3.04system 0:12.57elapsed 123%CPU (0avgtext+0avgdata 68720maxresident)k
0inputs+1056712outputs (0major+159306minor)pagefaults 0swaps

5.60user 2.33system 0:07.07elapsed 112%CPU (0avgtext+0avgdata 81388maxresident)k      \
 0inputs+1108720outputs (0major+171171minor)pagefaults 0swaps                 

firefox
--------
8.80user 3.57system 0:37.17elapsed 33%CPU (0avgtext+0avgdata 25260maxresident)k
8inputs+548024outputs (0major+376614minor)pagefaults 0swaps

4.52user 3.30system 0:33.14elapsed 23%CPU (0avgtext+0avgdata 25152maxresident)k
0inputs+544560outputs (0major+386394minor)pagefaults 0swaps

firefox eatmydata
-----------------------
8.79user 2.87system 0:12.43elapsed 93%CPU (0avgtext+0avgdata 25416maxresident)k
0inputs+548016outputs (0major+384193minor)pagefaults 0swaps
4.24user 2.57system 0:08.54elapsed 79%CPU (0avgtext+0avgdata 25280maxresident)k
0inputs+544584outputs (0major+392117minor)pagefaults 0swaps

libreoffice
-------------
22.51user 7.65system 1:28.34elapsed 34%CPU (0avgtext+0avgdata 64856maxresident)k
0inputs+1376160outputs (0major+1018794minor)pagefaults 0swaps

11.34user 6.66system 1:18.04elapsed 23%CPU (0avgtext+0avgdata 64676maxresident)k
16inputs+1370112outputs (0major+1024989minor)pagefaults 0swaps

libreoffice eatmydata
----------------------------
22.41user 6.82system 0:27.45elapsed 106%CPU (0avgtext+0avgdata 64772maxresident)k
0inputs+1376160outputs (0major+1035581minor)pagefaults 0swaps

10.86user 5.78system 0:17.70elapsed 94%CPU (0avgtext+0avgdata 64800maxresident)k
0inputs+1370112outputs (0major+1043637minor)pagefaults 0swaps

-- 
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer                              i speak de, en

-- 
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: \
https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel


[prev in list] [next in list] [prev in thread] [next in thread]