[prev in list] [next in list] [prev in thread] [next in thread]
List: nepomuk
Subject: Re: [Nepomuk] Performance issues in KDE 4.8.1
From: Vishesh Handa <me () vhanda ! in>
Date: 2012-03-09 19:24:29
Message-ID: CAOPTMKB=ZW1+msmhwXQKgkDZ4EMa3pTr=o3dVXMqdQRdEP3QDg () mail ! gmail ! com
[Download RAW message or body]
[Attachment #2 (multipart/alternative)]
On Fri, Mar 9, 2012 at 7:52 PM, Ignacio Serantes <kde@aynoa.net> wrote:
>
> On Fri, Mar 9, 2012 at 9:05 AM, Vishesh Handa <me@vhanda.in> wrote:
>
>>
>>
>> On Fri, Mar 9, 2012 at 1:25 PM, Ignacio Serantes <kde@aynoa.net> wrote:
>>
>>> Hi,
>>>
>>> I'm not sure but seems like creating a resource with Nepomuk.Resource()
>>> is slow than before and needs more CPU. This is a point that
>>> probably Ehrichs could confirm better than me.
>>>
>>
>> Yup. It should be slower, we're now doing error checking. I'm not sure
>> about it needing more CPU than before, though.
>>
>
> If you could write code that do error checking and logical analysis
> without requiring more CPU you will don't have problems to found a job in
> the future. But, I wonder why are you checking errors when you are only
> reading data?
>
We aren't. The data reading code is exactly the same.
>
>
>>
>> An about the synchronous has it's advantages where there is a sort in
>>> client side, there is a lot of resources, you are handling correctly th=
e
>>> fetch on demand and you are connecting to an internet server but, if t=
here
>>> is performance issues returning 144 records from a local database serve=
r
>>> because the operation is synchronous, I think we have a little problem.=
In
>>> fact with so few records probably asynchronous operation will be slow i=
n
>>> total time ;).
>>>
>>
>> Uhh. I lost you. Isn't 144 records a lot?
>>
>
> No, it's nothing, there are practically 0 records.
>
>
>>
>> What happens with the synchronous version is -
>>
>> 1. One blocking query to get all the resource uris of type nao:Tag
>> 2. One query for each nao:Tag in order to load *all* of its properties,
>> and then store them in a cache.
>>
>
> I don't follow you, it's the same you are using synchronous or
> asynchronous communication, you need to do a query to obtain the result s=
et
> you are looking for. For example in the case of tags a good one will be:
>
> SELECT ?uri ?identifier ?prefLabel
> WHERE {
> ?uri rdf:type nao:Tag .
> ?uri nao:identifier ?identifier .
> ?uri nao:userVisible 1 .
> OPTIONAL { ?uri nao:prefLabel . } .
> }
>
> because you are reading three important fields with only one query and,
> off course, LIMIT and OFFSET are your friends. The method used to transmi=
t
> this information could be synchronous or asynchronous :?.
>
Sorry. I should have been clearer.
Here is what is happening right now -
1. One synchronous blocking query is executed to get all the tag uris,
which are of the form nepomuk:/res/some-uuid
2. For each uri
One synchronous blocking query is performed which is something like -
select ?p ?o where { <nepomuk:/res/the-uuid> ?p ?o. }
3. This all is stored in the cache
That's why it is slow.
>
>> Considering that we do not need all of the properties and we only requir=
e
>> a label. One query would be a lot better, even if it is synchronous.
>>
>>
>>>
>>> How many queries are required to create a resource? Is cache working as
>>> expected? I know there is a cache because I have issues, expected cache
>>> issues :), developing nepoogle so it's still working as expected? Obvio=
usly
>>> there is something different in the last version of KDE.
>>>
>>
>> What kind of cache issues are you experiencing? Cause with 4.8.1 we have
>> added extra code to make sure that the cache is always up to date.
>>
>
> As I wrote, expected cache issues basically related with filex:/ protocol
> when I plug and unplug an external HD. In this case cache results are
> outdated and I found this a logical behavior easy to solve using
> Nepomuk.ResourceManager.instance().clearCache().
>
Hmm. We didn't think of that.
I'll take a look at it.
>
>
>>
>>
>>>
>>> On Fri, Mar 9, 2012 at 6:52 AM, Vishesh Handa <me@vhanda.in> wrote:
>>>
>>>>
>>>>
>>>> On Fri, Mar 9, 2012 at 1:39 AM, Ignacio Serantes <kde@aynoa.net> wrote=
:
>>>>
>>>>> Hi,
>>>>>
>>>>> Just installed KDE 4.8.1 I found a severe performance issue with my
>>>>> plasmoid Drop2TagIcon.
>>>>>
>>>>> Some investigation reveals that the problem is the call to
>>>>> Nepomuk.Tag.allTags() and, as I have 5 Drop2TagIcon in each of my fou=
r
>>>>> activities, the final result is that Plasma was totally frozen.
>>>>>
>>>>> Sadly this seems not restricted to Python, in irc J=F6rg Ehrichs has
>>>>> similar issues with Conquirere and a few minutes ago, when I'm trying=
to
>>>>> tag files using Dolphin in the old fashion way, I discover that the p=
roblem
>>>>> is here too.up
>>>>>
>>>>
>>>> Yup. Nepomuk::Tag::allTags() is synchronous, and ideally, shouldn't be
>>>> used.
>>>>
>>>> I've noticed the performance impact as well.
>>>>
>>>>
>>>>> More information:
>>>>> - I have only 144 tags.
>>>>> - Over 7-8 seconds passed between I click in Dolphin "Add Tags..." an=
d
>>>>> the dialog is displayed.
>>>>> - Meanwhile over 33% of my dual core CPU is used by the
>>>>> nepomukservicestub that launches virtuoso.
>>>>> - The next query:
>>>>>
>>>>> SELECT ?uri ?label
>>>>>
>>>>> WHERE {
>>>>>
>>>>> ?uri rdf:type nao:Tag .
>>>>>
>>>>> OPTIONAL { ?uri nao:prefLabel ?label . } .
>>>>>
>>>>> } ORDER BY ?label
>>>>>
>>>>> took over 150-170 milliseconds using NepSak and consuming 6% for an
>>>>> instant.
>>>>>
>>>>
>>>> That seems decent. I was going to use the QueryLibrary, but maybe I
>>>> could directly use sparql.
>>>>
>>>>
>>>>>
>>>>> --
>>>>> Best wishes,
>>>>> Ignacio
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Nepomuk mailing list
>>>>> Nepomuk@kde.org
>>>>> https://mail.kde.org/mailman/listinfo/nepomuk
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> Best wishes,
>>> Ignacio
>>>
>>>
>>>
>>> _______________________________________________
>>> Nepomuk mailing list
>>> Nepomuk@kde.org
>>> https://mail.kde.org/mailman/listinfo/nepomuk
>>>
>>>
>>
>
>
> --
> Best wishes,
> Ignacio
>
>
>
> _______________________________________________
> Nepomuk mailing list
> Nepomuk@kde.org
> https://mail.kde.org/mailman/listinfo/nepomuk
>
>
[Attachment #5 (text/html)]
<br><br><div class="gmail_quote">On Fri, Mar 9, 2012 at 7:52 PM, Ignacio Serantes \
<span dir="ltr"><<a href="mailto:kde@aynoa.net">kde@aynoa.net</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"> <br><div class="gmail_quote"><div class="im">On Fri, \
Mar 9, 2012 at 9:05 AM, Vishesh Handa <span dir="ltr"><<a \
href="mailto:me@vhanda.in" target="_blank">me@vhanda.in</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex">
<br><br><div class="gmail_quote"><div>On Fri, Mar 9, 2012 at 1:25 PM, Ignacio \
Serantes <span dir="ltr"><<a href="mailto:kde@aynoa.net" \
target="_blank">kde@aynoa.net</a>></span> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">
Hi,<div><br></div><div><div>I'm not sure but seems like creating a resource with \
Nepomuk.Resource() is slow than before and needs more CPU. This is a point that \
probably Ehrichs could confirm better than me.</div> </div></blockquote>
<div><br>Yup. It should be slower, we're now doing error checking. I'm not \
sure about it needing more CPU than before, \
though.<br></div></div></blockquote><div><br></div></div><div>If you could write code \
that do error checking and logical analysis without requiring more CPU you will \
don't have problems to found a job in the future. But, I wonder why are you \
checking errors when you are only reading data?</div> </div></blockquote><div><br>We \
aren't. The data reading code is exactly the same.<br> <br></div><blockquote \
class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"> <div class="gmail_quote"><div class="im">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div class="gmail_quote"><div><br></div><div><blockquote \
class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">
<div>An about the synchronous has it's advantages where there is a sort in client \
side, there is a lot of resources, you are handling correctly the fetch on demand and \
you are connecting to an internet server but, if there is performance issues \
returning 144 records from a local database server because the operation is \
synchronous, I think we have a little problem. In fact with so few records probably \
asynchronous operation will be slow in total time ;).</div>
</blockquote></div><div><br>Uhh. I lost you. Isn't 144 records a \
lot?<br></div></div></blockquote><div><br></div></div><div>No, it's nothing, \
there are practically 0 records.</div><div class="im"><div> </div> <blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div class="gmail_quote"><div><br>What happens with the \
synchronous version is -<br><br>1. One blocking query to get all the resource uris of \
type nao:Tag<br>
2. One query for each nao:Tag in order to load *all* of its properties, and then \
store them in a cache.<br></div></div></blockquote><div><br></div></div><div>I \
don't follow you, it's the same you are using synchronous or asynchronous \
communication, you need to do a query to obtain the result set you are looking for. \
For example in the case of tags a good one will be:</div>
<div><br></div><div>SELECT ?uri ?identifier ?prefLabel</div></div><div><div \
class="gmail_quote"><div><div class="im"><div>WHERE {</div><div> ?uri rdf:type \
nao:Tag . </div></div><div> ?uri nao:identifier ?identifier .</div> <div> ?uri \
nao:userVisible 1 .</div>
<div> OPTIONAL { ?uri nao:prefLabel . } .</div>
<div>}</div></div><div><br></div><div>because you are reading three important fields \
with only one query and, off course, LIMIT and OFFSET are your friends. The method \
used to transmit this information could be synchronous or asynchronous :?.</div> \
</div></div></blockquote><div><br>Sorry. I should have been clearer.<br><br>Here is \
what is happening right now -<br><br>1. One synchronous blocking query is executed to \
get all the tag uris, which are of the form nepomuk:/res/some-uuid<br> 2. For each \
uri <br> One synchronous blocking query is performed which is something like -<br> \
select ?p ?o where { <nepomuk:/res/the-uuid> ?p ?o. }<br>3. This all is stored \
in the cache<br><br>That's why it is slow.<br> <br></div><blockquote \
class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex"><div><div class="gmail_quote"><div class="im">
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 \
.8ex;border-left:1px #ccc solid;padding-left:1ex"><div \
class="gmail_quote"><div><br>Considering that we do not need all of the properties \
and we only require a label. One query would be a lot better, even if it is \
synchronous.<br>
<br></div><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><br></div><div>How many queries are required to create a resource? Is cache \
working as expected? I know there is a cache because I have issues, expected cache \
issues :), developing nepoogle so it's still working as expected? Obviously there \
is something different in the last version of KDE.</div>
</blockquote></div><div><br>What kind of cache issues are you experiencing? Cause \
with 4.8.1 we have added extra code to make sure that the cache is always up to \
date.<br></div></div></blockquote><div><br></div></div><div> As I wrote, expected \
cache issues basically related with filex:/ protocol when I plug and unplug an \
external HD. In this case cache results are outdated and I found this a logical \
behavior easy to solve using Nepomuk.ResourceManager.instance().clearCache().</div> \
</div></div></blockquote><div><br>Hmm. We didn't think of that.<br><br>I'll \
take a look at it.<br> <br></div><blockquote class="gmail_quote" style="margin:0pt \
0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <div><div \
class="gmail_quote"><div><div></div><div class="h5">
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"><div class="gmail_quote"><div> \
<br></div><div><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><br></div><div><div><div></div><div><div class="gmail_quote">On Fri, Mar 9, 2012 \
at 6:52 AM, Vishesh Handa <span dir="ltr"><<a href="mailto:me@vhanda.in" \
target="_blank">me@vhanda.in</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><br><br><div class="gmail_quote"><div>On Fri, Mar 9, 2012 at \
1:39 AM, Ignacio Serantes <span dir="ltr"><<a href="mailto:kde@aynoa.net" \
target="_blank">kde@aynoa.net</a>></span> wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div> Hi,<div><br></div><div>Just installed KDE 4.8.1 I found \
a severe performance issue with my plasmoid \
Drop2TagIcon.</div><div><br></div><div>Some investigation reveals that the problem is \
the call to Nepomuk.Tag.allTags() and, as I have 5 Drop2TagIcon in each of my four \
activities, the final result is that Plasma was totally frozen.</div>
<div><br></div></div><div>Sadly this seems not restricted to Python, in irc Jörg \
Ehrichs has similar issues with Conquirere and a few minutes ago, when I'm trying \
to tag files using Dolphin in the old fashion way, I discover that the problem is \
here too.up</div>
</blockquote><div><br>Yup. Nepomuk::Tag::allTags() is synchronous, and ideally, \
shouldn't be used.<br><br>I've noticed the performance impact as \
well.<br><br></div><div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><br></div><div>More information:</div><div>- I have only 144 tags.</div><div>- \
Over 7-8 seconds passed between I click in Dolphin "Add Tags..." and the \
dialog is displayed. </div><div>- Meanwhile over 33% of my dual core CPU is used by \
the nepomukservicestub that launches virtuoso.</div>
<div>- The next query:</div><div><br></div><div>SELECT ?uri ?label</div><div>
<p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px">WHERE \
{</p> <p style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px"> \
?uri rdf:type nao:Tag .</p> <p \
style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px"> OPTIONAL \
{ ?uri nao:prefLabel ?label . } .</p> <p \
style="margin-top:0px;margin-bottom:0px;margin-left:0px;margin-right:0px">} ORDER BY \
?label</p></div><div><br></div><div>took over 150-170 milliseconds using NepSak and \
consuming 6% for an instant.</div></blockquote></div>
<div>
<br>That seems decent. I was going to use the QueryLibrary, but maybe I could \
directly use sparql.<br> <br></div><blockquote class="gmail_quote" style="margin:0pt \
0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div><br></div><span><font color="#888888"><font color="#888888">
<div>-- <br>Best wishes,<div>Ignacio</div><div><br></div><br>
</div>
</font><br>_______________________________________________<br>
Nepomuk mailing list<br>
<a href="mailto:Nepomuk@kde.org" target="_blank">Nepomuk@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/nepomuk" \
target="_blank">https://mail.kde.org/mailman/listinfo/nepomuk</a><br> \
<br></font></span></blockquote></div><br> </blockquote></div><br><br \
clear="all"><div><br></div></div></div><font color="#888888">-- <br>Best \
wishes,<div>Ignacio</div><div><br></div><br> </font></div>
<br>_______________________________________________<br>
Nepomuk mailing list<br>
<a href="mailto:Nepomuk@kde.org" target="_blank">Nepomuk@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/nepomuk" \
target="_blank">https://mail.kde.org/mailman/listinfo/nepomuk</a><br> \
<br></blockquote></div></div></div><br> </blockquote></div></div></div><font \
color="#888888"><br><br clear="all"><div><br></div>-- <br>Best \
wishes,<div>Ignacio</div><div><br></div><br> </font></div>
<br>_______________________________________________<br>
Nepomuk mailing list<br>
<a href="mailto:Nepomuk@kde.org">Nepomuk@kde.org</a><br>
<a href="https://mail.kde.org/mailman/listinfo/nepomuk" \
target="_blank">https://mail.kde.org/mailman/listinfo/nepomuk</a><br> \
<br></blockquote></div><br>
_______________________________________________
Nepomuk mailing list
Nepomuk@kde.org
https://mail.kde.org/mailman/listinfo/nepomuk
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic