From kfm-devel Tue Sep 17 11:32:13 2013 From: Mark Date: Tue, 17 Sep 2013 11:32:13 +0000 To: kfm-devel Subject: Re: UDSEntry compression ideas Message-Id: X-MARC-Message: https://marc.info/?l=kfm-devel&m=137941758301483 MIME-Version: 1 Content-Type: multipart/mixed; boundary="--047d7b6dcc9649e9d604e692adcd" --047d7b6dcc9649e9d604e692adcd Content-Type: text/plain; charset=ISO-8859-1 On Tue, Sep 17, 2013 at 12:57 AM, Frank Reininghaus < frank78ac@googlemail.com> wrote: > Hi Mark, > > I'm too tired for a longer reply already, but here is a quick idea > that I just had and tried immediately. > > 2013/9/16 Mark: > > Hi All, > > > > please read this as a brainstorm post with some ideas. I have no code > > working in this area yet. > > > > Lazy loading is awesome, but if you can compress the data enough that you > > still have all details without lazy loading is even better i think. For > > example, doing a directory listing on any given folder gives at least the > > following UDSEntry values: > > - Device ID > > - Inode > > - User > > - Group > > > > In the "experimental" case where you list the folder contents that has > just > > a bunch of files created by one user you can say that at least the above > 4 > > properties are the same for all files. However, the experimental > situation > > certainly doesn't work on "real data" where you can have files from > multiple > > users and folders as well. > > Well, I guess that most people have quite a few folders where the > "User" and "Group" are the same for all files ;-) > > The easiest "compression" that I could think of was to force the > UDSEntries to implicitly share all their QStrings for one field if > they are all equal: > > http://paste.kde.org/p52a24b49/ > > Reduces the memory consumption in Details View for 100,000 items from > 160.5 MB to 147 MB here. The patch is a bit hackish though, and it > might have a small performance penalty. Maybe it's worth investigating > that further though. > > I guess any more sophisticated "compression" approach will have to be > implemented in frameworks only. > > Cheers, > Frank > That is a quite effective first approach :) Doing the same for the numeric values is likely going to give an equal memory saving. --047d7b6dcc9649e9d604e692adcd Content-Type: text/html; charset=ISO-8859-1
On Tue, Sep 17, 2013 at 12:57 AM, Frank Reininghaus <frank78ac@googlemail.com> wrote:
Hi Mark,

I'm too tired for a longer reply already, but here is a quick idea
that I just had and tried immediately.

2013/9/16 Mark:
> Hi All,
>
> please read this as a brainstorm post with some ideas. I have no code
> working in this area yet.
>
> Lazy loading is awesome, but if you can compress the data enough that you
> still have all details without lazy loading is even better i think. For
> example, doing a directory listing on any given folder gives at least the
> following UDSEntry values:
> - Device ID
> - Inode
> - User
> - Group
>
> In the "experimental" case where you list the folder contents that has just
> a bunch of files created by one user you can say that at least the above 4
> properties are the same for all files. However, the experimental situation
> certainly doesn't work on "real data" where you can have files from multiple
> users and folders as well.

Well, I guess that most people have quite a few folders where the
"User" and "Group" are the same for all files ;-)

The easiest "compression" that I could think of was to force the
UDSEntries to implicitly share all their QStrings for one field if
they are all equal:

http://paste.kde.org/p52a24b49/

Reduces the memory consumption in Details View for 100,000 items from
160.5 MB to 147 MB here. The patch is a bit hackish though, and it
might have a small performance penalty. Maybe it's worth investigating
that further though.

I guess any more sophisticated "compression" approach will have to be
implemented in frameworks only.

Cheers,
Frank

That is a quite effective first approach :)
Doing the same for the numeric values is likely going to give an equal memory saving.
--047d7b6dcc9649e9d604e692adcd--