[prev in list] [next in list] [prev in thread] [next in thread] 

List:       kfm-devel
Subject:    Re: UDSEntry compression ideas
From:       Mark <markg85 () gmail ! com>
Date:       2013-09-17 11:32:13
Message-ID: CAPd6JnFHr1gYMq3qQxNnV1rjWcgTwwsPtOxCtD82d35rj2xk7A () mail ! gmail ! com
[Download RAW message or body]

On Tue, Sep 17, 2013 at 12:57 AM, Frank Reininghaus <
frank78ac@googlemail.com> wrote:

> Hi Mark,
>
> I'm too tired for a longer reply already, but here is a quick idea
> that I just had and tried immediately.
>
> 2013/9/16 Mark:
> > Hi All,
> >
> > please read this as a brainstorm post with some ideas. I have no code
> > working in this area yet.
> >
> > Lazy loading is awesome, but if you can compress the data enough that you
> > still have all details without lazy loading is even better i think. For
> > example, doing a directory listing on any given folder gives at least the
> > following UDSEntry values:
> > - Device ID
> > - Inode
> > - User
> > - Group
> >
> > In the "experimental" case where you list the folder contents that has
> just
> > a bunch of files created by one user you can say that at least the above
> 4
> > properties are the same for all files. However, the experimental
> situation
> > certainly doesn't work on "real data" where you can have files from
> multiple
> > users and folders as well.
>
> Well, I guess that most people have quite a few folders where the
> "User" and "Group" are the same for all files ;-)
>
> The easiest "compression" that I could think of was to force the
> UDSEntries to implicitly share all their QStrings for one field if
> they are all equal:
>
> http://paste.kde.org/p52a24b49/
>
> Reduces the memory consumption in Details View for 100,000 items from
> 160.5 MB to 147 MB here. The patch is a bit hackish though, and it
> might have a small performance penalty. Maybe it's worth investigating
> that further though.
>
> I guess any more sophisticated "compression" approach will have to be
> implemented in frameworks only.
>
> Cheers,
> Frank
>

That is a quite effective first approach :)
Doing the same for the numeric values is likely going to give an equal
memory saving.

[Attachment #3 (text/html)]

<div dir="ltr">On Tue, Sep 17, 2013 at 12:57 AM, Frank Reininghaus <span \
dir="ltr">&lt;<a href="mailto:frank78ac@googlemail.com" \
target="_blank">frank78ac@googlemail.com</a>&gt;</span> wrote:<br><div \
class="gmail_extra"><div class="gmail_quote">

<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Hi Mark,<br> <br>
I&#39;m too tired for a longer reply already, but here is a quick idea<br>
that I just had and tried immediately.<br>
<br>
2013/9/16 Mark:<br>
<div class="im">&gt; Hi All,<br>
&gt;<br>
&gt; please read this as a brainstorm post with some ideas. I have no code<br>
&gt; working in this area yet.<br>
&gt;<br>
&gt; Lazy loading is awesome, but if you can compress the data enough that you<br>
&gt; still have all details without lazy loading is even better i think. For<br>
&gt; example, doing a directory listing on any given folder gives at least the<br>
&gt; following UDSEntry values:<br>
&gt; - Device ID<br>
&gt; - Inode<br>
&gt; - User<br>
&gt; - Group<br>
&gt;<br>
&gt; In the &quot;experimental&quot; case where you list the folder contents that has \
just<br> &gt; a bunch of files created by one user you can say that at least the \
above 4<br> &gt; properties are the same for all files. However, the experimental \
situation<br> &gt; certainly doesn&#39;t work on &quot;real data&quot; where you can \
have files from multiple<br> &gt; users and folders as well.<br>
<br>
</div>Well, I guess that most people have quite a few folders where the<br>
&quot;User&quot; and &quot;Group&quot; are the same for all files ;-)<br>
<br>
The easiest &quot;compression&quot; that I could think of was to force the<br>
UDSEntries to implicitly share all their QStrings for one field if<br>
they are all equal:<br>
<br>
<a href="http://paste.kde.org/p52a24b49/" \
target="_blank">http://paste.kde.org/p52a24b49/</a><br> <br>
Reduces the memory consumption in Details View for 100,000 items from<br>
160.5 MB to 147 MB here. The patch is a bit hackish though, and it<br>
might have a small performance penalty. Maybe it&#39;s worth investigating<br>
that further though.<br>
<br>
I guess any more sophisticated &quot;compression&quot; approach will have to be<br>
implemented in frameworks only.<br>
<br>
Cheers,<br>
Frank<br>
</blockquote></div><br></div><div class="gmail_extra">That is a quite effective first \
approach :)</div><div class="gmail_extra">Doing the same for the numeric values is \
likely going to give an equal memory saving.</div></div>



[prev in list] [next in list] [prev in thread] [next in thread]