[prev in list] [next in list] [prev in thread] [next in thread]
List: kfm-devel
Subject: Re: UDSEntry compression ideas
From: Todd <toddrjen () gmail ! com>
Date: 2013-09-17 11:58:25
Message-ID: CAFpSVpJJwOF47V8gA7G90Mk2zV=zOkZsn+18OPpuQrXz2oDBkw () mail ! gmail ! com
[Download RAW message or body]
On Sep 17, 2013 12:58 AM, "Frank Reininghaus" <frank78ac@googlemail.com>
wrote:
>
> Hi Mark,
>
> I'm too tired for a longer reply already, but here is a quick idea
> that I just had and tried immediately.
>
> 2013/9/16 Mark:
> > Hi All,
> >
> > please read this as a brainstorm post with some ideas. I have no code
> > working in this area yet.
> >
> > Lazy loading is awesome, but if you can compress the data enough that
you
> > still have all details without lazy loading is even better i think. For
> > example, doing a directory listing on any given folder gives at least
the
> > following UDSEntry values:
> > - Device ID
> > - Inode
> > - User
> > - Group
> >
> > In the "experimental" case where you list the folder contents that has
just
> > a bunch of files created by one user you can say that at least the
above 4
> > properties are the same for all files. However, the experimental
situation
> > certainly doesn't work on "real data" where you can have files from
multiple
> > users and folders as well.
>
> Well, I guess that most people have quite a few folders where the
> "User" and "Group" are the same for all files ;-)
>
> The easiest "compression" that I could think of was to force the
> UDSEntries to implicitly share all their QStrings for one field if
> they are all equal:
>
> http://paste.kde.org/p52a24b49/
>
> Reduces the memory consumption in Details View for 100,000 items from
> 160.5 MB to 147 MB here. The patch is a bit hackish though, and it
> might have a small performance penalty. Maybe it's worth investigating
> that further though.
>
> I guess any more sophisticated "compression" approach will have to be
> implemented in frameworks only.
>
> Cheers,
> Frank
Do we need the UID and GID? Can we just store whether the file is
readable, writable, and executable? That would be three bools rather than
two ints.
[Attachment #3 (text/html)]
<p dir="ltr"><br>
On Sep 17, 2013 12:58 AM, "Frank Reininghaus" <<a \
href="mailto:frank78ac@googlemail.com">frank78ac@googlemail.com</a>> wrote:<br> \
><br> > Hi Mark,<br>
><br>
> I'm too tired for a longer reply already, but here is a quick idea<br>
> that I just had and tried immediately.<br>
><br>
> 2013/9/16 Mark:<br>
> > Hi All,<br>
> ><br>
> > please read this as a brainstorm post with some ideas. I have no code<br>
> > working in this area yet.<br>
> ><br>
> > Lazy loading is awesome, but if you can compress the data enough that \
you<br> > > still have all details without lazy loading is even better i think. \
For<br> > > example, doing a directory listing on any given folder gives at \
least the<br> > > following UDSEntry values:<br>
> > - Device ID<br>
> > - Inode<br>
> > - User<br>
> > - Group<br>
> ><br>
> > In the "experimental" case where you list the folder contents \
that has just<br> > > a bunch of files created by one user you can say that at \
least the above 4<br> > > properties are the same for all files. However, the \
experimental situation<br> > > certainly doesn't work on "real \
data" where you can have files from multiple<br> > > users and folders as \
well.<br> ><br>
> Well, I guess that most people have quite a few folders where the<br>
> "User" and "Group" are the same for all files ;-)<br>
><br>
> The easiest "compression" that I could think of was to force the<br>
> UDSEntries to implicitly share all their QStrings for one field if<br>
> they are all equal:<br>
><br>
> <a href="http://paste.kde.org/p52a24b49/">http://paste.kde.org/p52a24b49/</a><br>
><br>
> Reduces the memory consumption in Details View for 100,000 items from<br>
> 160.5 MB to 147 MB here. The patch is a bit hackish though, and it<br>
> might have a small performance penalty. Maybe it's worth investigating<br>
> that further though.<br>
><br>
> I guess any more sophisticated "compression" approach will have to \
be<br> > implemented in frameworks only.<br>
><br>
> Cheers,<br>
> Frank</p>
<p dir="ltr">Do we need the UID and GID? Can we just store whether the file is \
readable, writable, and executable? That would be three bools rather than two \
ints.<br> </p>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic