[prev in list] [next in list] [prev in thread] [next in thread]
List: ubuntu-devel
Subject: Re: zstd compression for packages
From: Daniel Axtens <daniel.axtens () canonical ! com>
Date: 2018-03-12 13:11:42
Message-ID: CAKjEEsG7X15wo6Hrtda7_rJFamTJVhF_hjNzMy9RMq+R13fjkg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
Hi,
I looked into compression algorithms a bit in a previous role, and to be
honest I'm quite surprised to see zstd proposed for package storage. zstd,
according to its own github repo, is "targeting real-time compression
scenarios". It's not really designed to be run at its maximum compression
level, it's designed to really quickly compress data coming off the wire -
things like compressing log files being streamed to a central server, or I
guess writing random data to btrfs where speed is absolutely an issue.
Is speed of decompression a big user concern relative to file size? I admit
that I am biased - as an Australian and with the crummy internet that my
location entails, I'd save much more time if the file was 6% smaller and
took 10% longer to decompress than the other way around.
Did you consider Google's Brotli?
Regards,
Daniel
On Mon, Mar 12, 2018 at 9:58 PM, Julian Andres Klode <
julian.klode@canonical.com> wrote:
> On Mon, Mar 12, 2018 at 11:06:11AM +0100, Julian Andres Klode wrote:
> > Hey folks,
> >
> > We had a coding day in Foundations last week and Balint and Julian added
> support for zstd compression to dpkg [1] and apt [2].
> >
> > [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664
> > [2] https://salsa.debian.org/apt-team/apt/merge_requests/8
> >
> > Zstd is a compression algorithm developed by Facebook that offers far
> > higher decompression speeds than xz or even gzip (at roughly constant
> > speed and memory usage across all levels), while offering 19 compression
> > levels ranging from roughly comparable to gzip in size (but much faster)
> > to 19, which is roughly comparable to xz -6:
> >
> > In our configuration, we run zstd at level 19. For bionic main amd64,
> > this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.
> > Installs speed up by about 10%, or, if eatmydata is involved, by up to
> > 40% - user time generally by about 50%.
> >
> > Our implementations for apt and dpkg support multiple frames as used by
> > pzstd, so packages can be compressed and decompressed in parallel
> > eventually.
>
> More links:
>
> PPA: https://launchpad.net/~canonical-foundations/+
> archive/ubuntu/zstd-archive
> APT merge request: https://salsa.debian.org/apt-team/apt/merge_requests/8
> dpkg patches: https://bugs.debian.org/892664
>
> I'd also like to talk a bit more about libzstd itself: The package is
> currently in universe, but btrfs recently gained support for zstd,
> so we already have a copy in the kernel and we need to MIR it anyway
> for btrfs-progs.
>
> --
> debian developer - deb.li/jak | jak-linux.org - free software dev
> ubuntu core developer i speak de, en
>
> --
> ubuntu-devel mailing list
> ubuntu-devel@lists.ubuntu.com
> Modify settings or unsubscribe at: https://lists.ubuntu.com/
> mailman/listinfo/ubuntu-devel
>
[Attachment #5 (text/html)]
<div dir="ltr">Hi,<div><br></div><div>I looked into compression algorithms a bit in a \
previous role, and to be honest I'm quite surprised to see zstd proposed for \
package storage. zstd, according to its own github repo, is "targeting real-time \
compression scenarios". It's not really designed to be run at its maximum \
compression level, it's designed to really quickly compress data coming off the \
wire - things like compressing log files being streamed to a central server, or I \
guess writing random data to btrfs where speed is absolutely an \
issue.</div><div><br></div><div>Is speed of decompression a big user concern relative \
to file size? I admit that I am biased - as an Australian and with the crummy \
internet that my location entails, I'd save much more time if the file was 6% \
smaller and took 10% longer to decompress than the other way \
around.</div><div><br></div><div>Did you consider Google's \
Brotli?</div><div><br></div><div>Regards,</div><div>Daniel</div></div><div \
class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 12, 2018 at 9:58 PM, \
Julian Andres Klode <span dir="ltr"><<a href="mailto:julian.klode@canonical.com" \
target="_blank">julian.klode@canonical.com</a>></span> wrote:<br><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><span class="">On Mon, Mar 12, 2018 at 11:06:11AM +0100, \
Julian Andres Klode wrote:<br> > Hey folks,<br>
><br>
> We had a coding day in Foundations last week and Balint and Julian added support \
for zstd compression to dpkg [1] and apt [2].<br> ><br>
> [1] <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664" \
rel="noreferrer" target="_blank">https://bugs.debian.org/cgi-<wbr>bin/bugreport.cgi?bug=892664</a><br>
> [2] <a href="https://salsa.debian.org/apt-team/apt/merge_requests/8" \
rel="noreferrer" target="_blank">https://salsa.debian.org/apt-<wbr>team/apt/merge_requests/8</a><br>
><br>
> Zstd is a compression algorithm developed by Facebook that offers far<br>
> higher decompression speeds than xz or even gzip (at roughly constant<br>
> speed and memory usage across all levels), while offering 19 compression<br>
> levels ranging from roughly comparable to gzip in size (but much faster)<br>
> to 19, which is roughly comparable to xz -6:<br>
><br>
> In our configuration, we run zstd at level 19. For bionic main amd64,<br>
> this causes a size increase of about 6%, from roughly 5.6 to 5.9 GB.<br>
> Installs speed up by about 10%, or, if eatmydata is involved, by up to<br>
> 40% - user time generally by about 50%.<br>
><br>
> Our implementations for apt and dpkg support multiple frames as used by<br>
> pzstd, so packages can be compressed and decompressed in parallel<br>
> eventually.<br>
<br>
</span>More links:<br>
<br>
PPA: <a \
href="https://launchpad.net/~canonical-foundations/+archive/ubuntu/zstd-archive" \
rel="noreferrer" target="_blank">https://launchpad.net/~<wbr>canonical-foundations/+<wbr>archive/ubuntu/zstd-archive</a><br>
APT merge request: <a href="https://salsa.debian.org/apt-team/apt/merge_requests/8" \
rel="noreferrer" target="_blank">https://salsa.debian.org/apt-<wbr>team/apt/merge_requests/8</a><br>
dpkg patches: <a href="https://bugs.debian.org/892664" rel="noreferrer" \
target="_blank">https://bugs.debian.org/892664</a><br> <br>
I'd also like to talk a bit more about libzstd itself: The package is<br>
currently in universe, but btrfs recently gained support for zstd,<br>
so we already have a copy in the kernel and we need to MIR it anyway<br>
for btrfs-progs.<br>
<div class="HOEnZb"><div class="h5"><br>
--<br>
debian developer - <a href="http://deb.li/jak" rel="noreferrer" \
target="_blank">deb.li/jak</a> | <a href="http://jak-linux.org" rel="noreferrer" \
target="_blank">jak-linux.org</a> - free software dev<br> ubuntu core developer \
i speak de, en<br> <br>
--<br>
ubuntu-devel mailing list<br>
<a href="mailto:ubuntu-devel@lists.ubuntu.com">ubuntu-devel@lists.ubuntu.com</a><br>
Modify settings or unsubscribe at: <a \
href="https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel" rel="noreferrer" \
target="_blank">https://lists.ubuntu.com/<wbr>mailman/listinfo/ubuntu-devel</a><br> \
</div></div></blockquote></div><br></div>
[Attachment #6 (text/plain)]
--
ubuntu-devel mailing list
ubuntu-devel@lists.ubuntu.com
Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic