[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: UDSEntry compression ideas
From: Mark <markg85 () gmail ! com>
Date: 2013-09-17 12:49:46
Message-ID: CAPd6JnHL94nkdETk56XBRg6aUqfOeqR63PKBtMHfP=Tuq48Dmw () mail ! gmail ! com
[Download RAW message or body]
On Tue, Sep 17, 2013 at 1:58 PM, Todd <toddrjen@gmail.com> wrote:
>
> On Sep 17, 2013 12:58 AM, "Frank Reininghaus" <frank78ac@googlemail.com>
> wrote:
> >
> > Hi Mark,
> >
> > I'm too tired for a longer reply already, but here is a quick idea
> > that I just had and tried immediately.
> >
> > 2013/9/16 Mark:
> > > Hi All,
> > >
> > > please read this as a brainstorm post with some ideas. I have no code
> > > working in this area yet.
> > >
> > > Lazy loading is awesome, but if you can compress the data enough that
> you
> > > still have all details without lazy loading is even better i think. For
> > > example, doing a directory listing on any given folder gives at least
> the
> > > following UDSEntry values:
> > > - Device ID
> > > - Inode
> > > - User
> > > - Group
> > >
> > > In the "experimental" case where you list the folder contents that has
> just
> > > a bunch of files created by one user you can say that at least the
> above 4
> > > properties are the same for all files. However, the experimental
> situation
> > > certainly doesn't work on "real data" where you can have files from
> multiple
> > > users and folders as well.
> >
> > Well, I guess that most people have quite a few folders where the
> > "User" and "Group" are the same for all files ;-)
> >
> > The easiest "compression" that I could think of was to force the
> > UDSEntries to implicitly share all their QStrings for one field if
> > they are all equal:
> >
> > http://paste.kde.org/p52a24b49/
> >
> > Reduces the memory consumption in Details View for 100,000 items from
> > 160.5 MB to 147 MB here. The patch is a bit hackish though, and it
> > might have a small performance penalty. Maybe it's worth investigating
> > that further though.
> >
> > I guess any more sophisticated "compression" approach will have to be
> > implemented in frameworks only.
> >
> > Cheers,
> > Frank
>
> Do we need the UID and GID? Can we just store whether the file is
> readable, writable, and executable? That would be three bools rather than
> two ints.
>
I don't know, but that would be a saving on both the client side as well as
the slave side..
Perhaps we need to invite David to chime in and give his view on this?
Added him to the CC.
[Attachment #3 (text/html)]
<div dir="ltr">On Tue, Sep 17, 2013 at 1:58 PM, Todd <span dir="ltr"><<a \
href="mailto:toddrjen@gmail.com" target="_blank">toddrjen@gmail.com</a>></span> \
wrote:<br><div class="gmail_extra"><div class="gmail_quote"><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5"><p dir="ltr"><br>
On Sep 17, 2013 12:58 AM, "Frank Reininghaus" <<a \
href="mailto:frank78ac@googlemail.com" \
target="_blank">frank78ac@googlemail.com</a>> wrote:<br> ><br>
> Hi Mark,<br>
><br>
> I'm too tired for a longer reply already, but here is a quick idea<br>
> that I just had and tried immediately.<br>
><br>
> 2013/9/16 Mark:<br>
> > Hi All,<br>
> ><br>
> > please read this as a brainstorm post with some ideas. I have no code<br>
> > working in this area yet.<br>
> ><br>
> > Lazy loading is awesome, but if you can compress the data enough that \
you<br> > > still have all details without lazy loading is even better i think. \
For<br> > > example, doing a directory listing on any given folder gives at \
least the<br> > > following UDSEntry values:<br>
> > - Device ID<br>
> > - Inode<br>
> > - User<br>
> > - Group<br>
> ><br>
> > In the "experimental" case where you list the folder contents \
that has just<br> > > a bunch of files created by one user you can say that at \
least the above 4<br> > > properties are the same for all files. However, the \
experimental situation<br> > > certainly doesn't work on "real \
data" where you can have files from multiple<br> > > users and folders as \
well.<br> ><br>
> Well, I guess that most people have quite a few folders where the<br>
> "User" and "Group" are the same for all files ;-)<br>
><br>
> The easiest "compression" that I could think of was to force the<br>
> UDSEntries to implicitly share all their QStrings for one field if<br>
> they are all equal:<br>
><br>
> <a href="http://paste.kde.org/p52a24b49/" \
target="_blank">http://paste.kde.org/p52a24b49/</a><br> ><br>
> Reduces the memory consumption in Details View for 100,000 items from<br>
> 160.5 MB to 147 MB here. The patch is a bit hackish though, and it<br>
> might have a small performance penalty. Maybe it's worth investigating<br>
> that further though.<br>
><br>
> I guess any more sophisticated "compression" approach will have to \
be<br> > implemented in frameworks only.<br>
><br>
> Cheers,<br>
> Frank</p>
</div></div><p dir="ltr">Do we need the UID and GID? Can we just store whether the \
file is readable, writable, and executable? That would be three bools rather than two \
ints.<br> </p>
</blockquote></div>I don't know, but that would be a saving on both the client \
side as well as the slave side..</div><div class="gmail_extra">Perhaps we need to \
invite David to chime in and give his view on this? Added him to the CC.</div>
</div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic