[prev in list] [next in list] [prev in thread] [next in thread]
List: kde-commits
Subject: Re: [nepomuk-core/feature/virtuosoInference] /: New resource visibility handling.
From: Vishesh Handa <handa.vish () gmail ! com>
Date: 2012-03-31 19:32:16
Message-ID: CAOPTMKCerjzDeyPMAvSKOX7xPCdVVWmt4rLxCjtNmzq4ez19NQ () mail ! gmail ! com
[Download RAW message or body]
On Sun, Apr 1, 2012 at 12:30 AM, Sebastian Trueg <trueg@kde.org> wrote:
> Git commit 9476df60687ecb133ebc9dd2f9c893edc1ab097a by Sebastian Trueg.
> Committed on 31/03/2012 at 20:46.
> Pushed by trueg into branch 'feature/virtuosoInference'.
>
> New resource visibility handling.
>
> Until now the query api would use the nao:userVisible values stored for
> each resource to determine if a resource should be returned.
> This requires a lot of nao:userVisible value maintenance and makes for
> slower queries.
> The new solution is much simpler: only types have nao:userVisible values
> and the query api simply checks if the result candidate has a type
> which is visible.
> In contrast to the old one the new solution does not allow to hide a
> sub-class of a visible class. But since we never did that anyway and
> the resulting queries are faster and it allows us to remove a lot
> maintenance code (which eats CPU cycles and disk space) the change is
> worth it.
>
I'm not sure about this. I kinda liked being able to hide certain resources
from "normal" view. I always thought it could be used to hold Artem's
decision resources. But then it wasn't really used.
>
> M +8 -13 libnepomukcore/query/query.cpp
> M +52 -0 services/storage/ontologyloader.cpp
> M +2 -1 services/storage/ontologyloader.h
>
>
> http://commits.kde.org/nepomuk-core/9476df60687ecb133ebc9dd2f9c893edc1ab097a
>
> diff --git a/libnepomukcore/query/query.cpp
> b/libnepomukcore/query/query.cpp
> index 943705d..6e242f4 100644
> --- a/libnepomukcore/query/query.cpp
> +++ b/libnepomukcore/query/query.cpp
> @@ -1,6 +1,6 @@
> /*
> This file is part of the Nepomuk KDE project.
> - Copyright (C) 2008-2010 Sebastian Trueg <trueg@kde.org>
> + Copyright (C) 2008-2012 Sebastian Trueg <trueg@kde.org>
>
> This library is free software; you can redistribute it and/or
> modify it under the terms of the GNU Library General Public
> @@ -450,22 +450,17 @@ QString Nepomuk::Query::Query::toSparqlQuery(
> SparqlFlags sparqlFlags ) const
> QueryBuilderData qbd( d.constData(), sparqlFlags );
>
>
> - // We restrict results to user visible types only. There is no need
> for this with file queries as normally all files are visible.
> //
> - // Here we use an optimizations:
> - // The storage service creates "nao:userVisible 1" entries for all
> visible resources. This contains two optimizations:
> - // - We use an integer instead of a boolean because Virtuoso does not
> support booleans and, thus, Soprano converts
> - // booleans into a fake type which is stored as a string. This
> makes comparision much slower.
> - // - Instead of using crappy inference via "?r a ?t . ?t
> nao:userVisible true" we can directly check the resources.
> - // - Instead of restricting the visibility to 1 we trick the Virtuoso
> query optimizer so it won't try to use the visibility
> - // as priority. We do this with a filter that does not test for
> equality. Using "?r nao:userVisible 1" has a drastic
> - // performance impact which the filter has not.
> + // We restrict results to user visible types only. There is no need
> for this with file queries as normally all files are visible.
> + // We need the additional type pattern to ensure that the variable ?r
> is actually available to the filter (in case there is only
> + // UNIONs before.
> //
> QString userVisibilityRestriction;
> if( !(queryFlags()&NoResultRestrictions) && !d->m_isFileQuery ) {
> - userVisibilityRestriction = QString::fromLatin1("?r %1 %2 .
> FILTER(%2>0) . ")
> -
> .arg(Soprano::Node::resourceToN3(Soprano::Vocabulary::NAO::userVisible()),
> - qbd.uniqueVarName());
> + userVisibilityRestriction = QString::fromLatin1("?r a %1 . FILTER
> EXISTS { ?r a [ %2 %3 ] . } . ")
> + .arg(qbd.uniqueVarName(),
> +
> Soprano::Node::resourceToN3(Soprano::Vocabulary::NAO::userVisible()),
> +
> Soprano::Node::literalToN3(Soprano::LiteralValue(true)));
> }
>
>
> diff --git a/services/storage/ontologyloader.cpp
> b/services/storage/ontologyloader.cpp
> index c06a297..a09665b 100644
> --- a/services/storage/ontologyloader.cpp
> +++ b/services/storage/ontologyloader.cpp
> @@ -27,6 +27,7 @@
> #include <Soprano/PluginManager>
> #include <Soprano/StatementIterator>
> #include <Soprano/Parser>
> +#include <Soprano/QueryResultIterator>
>
> #include <KConfig>
> #include <KConfigGroup>
> @@ -42,14 +43,23 @@
> #include <kpluginfactory.h>
> #include <kpluginloader.h>
>
> +#include <Soprano/Vocabulary/RDFS>
> +#include <Soprano/Vocabulary/NAO>
> +#include <Soprano/Vocabulary/XMLSchema>
>
> using namespace Soprano;
> +using namespace Soprano::Vocabulary;
> +
> +namespace {
> + const char* s_typeVisibilityGraph = "nepomuk:/ctx/typevisibility";
> +}
>
> class Nepomuk::OntologyLoader::Private
> {
> public:
> Private( OntologyLoader* p )
> : forceOntologyUpdate( false ),
> + someOntologyUpdated( false ),
> q( p ) {
> }
>
> @@ -59,6 +69,9 @@ public:
> bool forceOntologyUpdate;
> QStringList desktopFilesToUpdate;
>
> + // true if at least one ontology has been updated
> + bool someOntologyUpdated;
> +
> void updateOntology( const QString& filename );
>
> private:
> @@ -110,6 +123,8 @@ void Nepomuk::OntologyLoader::Private::updateOntology(
> const QString& filename )
> }
>
> if( update ) {
> + someOntologyUpdated = true;
> +
> QString mimeType = df.readEntry( "MimeType", QString() );
>
> const Soprano::Parser* parser
> @@ -179,6 +194,7 @@ Nepomuk::OntologyLoader::~OntologyLoader()
>
> void Nepomuk::OntologyLoader::updateLocalOntologies()
> {
> + d->someOntologyUpdated = false;
> d->desktopFilesToUpdate = KGlobal::dirs()->findAllResources(
> "xdgdata-ontology", "*.ontology",
> KStandardDirs::Recursive|KStandardDirs::NoDuplicates );
> if(d->desktopFilesToUpdate.isEmpty())
> kError() << "No ontology files found! Make sure the
> shared-desktop-ontologies project is installed and XDG_DATA_DIRS is set
> properly.";
> @@ -201,6 +217,16 @@ void Nepomuk::OntologyLoader::updateNextOntology()
> else {
> d->forceOntologyUpdate = false;
> d->updateTimer.stop();
> +
> + // update graph visibility if something has changed or if we
> never did it
> + const QUrl visibilityGraph =
> QUrl::fromEncoded(s_typeVisibilityGraph);
> + if(d->someOntologyUpdated ||
> + !d->model->executeQuery(QString::fromLatin1("ask where { graph
> %1 { ?s ?p ?o . } }")
> +
> .arg(Soprano::Node::resourceToN3(visibilityGraph)),
> +
> Soprano::Query::QueryLanguageSparql).boolValue()) {
> + updateTypeVisibility();
> + }
> +
> emit ontologyLoadingFinished(this);
> }
> }
> @@ -229,6 +255,7 @@ void
> Nepomuk::OntologyLoader::slotGraphRetrieverResult( KJob* job )
> // TODO: find a way to check if the imported version of the
> ontology
> // is newer than the already installed one
> if ( d->model->updateOntology( graphRetriever->statements(),
> QUrl()/*graphRetriever->url()*/ ) ) {
> + updateTypeVisibility();
> emit ontologyUpdated( QString::fromAscii(
> graphRetriever->url().toEncoded() ) );
> }
> else {
> @@ -237,4 +264,29 @@ void
> Nepomuk::OntologyLoader::slotGraphRetrieverResult( KJob* job )
> }
> }
>
> +void Nepomuk::OntologyLoader::updateTypeVisibility()
> +{
> + const QUrl visibilityGraph = QUrl::fromEncoded(s_typeVisibilityGraph);
> +
> + // 1. remove all visibility values we added ourselves
> + d->model->removeContext(visibilityGraph);
> +
> + // 2. make rdfs:Resource non-visible (this is required since with KDE
> 4.9 we introduced a new
> + // way of visibility handling which relies on types alone rather
> than visibility values on
> + // resources. Any visible type will make all sub-types visible,
> too. If rdfs:Resource were
> + // visible everything would be.
> + d->model->removeAllStatements(RDFS::Resource(), NAO::userVisible(),
> Soprano::Node());
> +
> + // 3. Set each type visible which is not rdfs:Resource and does not
> have a non-visible parent
> + d->model->executeQuery(QString::fromLatin1("insert into %1 { "
> + "?t %2 'true'^^%3 . "
> + "} where { "
> + "?t a rdfs:Class . "
> + "filter not exists { ?tt
> %2 'false'^^%3 . ?t rdfs:subClassOf ?tt . } }")
> +
> .arg(Soprano::Node::resourceToN3(visibilityGraph),
> +
> Soprano::Node::resourceToN3(NAO::userVisible()),
> +
> Soprano::Node::resourceToN3(XMLSchema::boolean())),
> + Soprano::Query::QueryLanguageSparql);
> +}
> +
> #include "ontologyloader.moc"
> diff --git a/services/storage/ontologyloader.h
> b/services/storage/ontologyloader.h
> index 4e0ce26..d714249 100644
> --- a/services/storage/ontologyloader.h
> +++ b/services/storage/ontologyloader.h
> @@ -1,5 +1,5 @@
> /* This file is part of the KDE Project
> - Copyright (c) 2007-2010 Sebastian Trueg <trueg@kde.org>
> + Copyright (c) 2007-2012 Sebastian Trueg <trueg@kde.org>
>
> This library is free software; you can redistribute it and/or
> modify it under the terms of the GNU Library General Public
> @@ -91,6 +91,7 @@ namespace Nepomuk {
> // a little async updating
> void updateNextOntology();
> void slotGraphRetrieverResult( KJob* job );
> + void updateTypeVisibility();
>
> private:
> class Private;
>
>
[Attachment #3 (text/html)]
<br><br><div class="gmail_quote">On Sun, Apr 1, 2012 at 12:30 AM, Sebastian Trueg \
<span dir="ltr"><<a href="mailto:trueg@kde.org">trueg@kde.org</a>></span> \
wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px \
#ccc solid;padding-left:1ex"> Git commit 9476df60687ecb133ebc9dd2f9c893edc1ab097a by \
Sebastian Trueg.<br> Committed on 31/03/2012 at 20:46.<br>
Pushed by trueg into branch 'feature/virtuosoInference'.<br>
<br>
New resource visibility handling.<br>
<br>
Until now the query api would use the nao:userVisible values stored for<br>
each resource to determine if a resource should be returned.<br>
This requires a lot of nao:userVisible value maintenance and makes for<br>
slower queries.<br>
The new solution is much simpler: only types have nao:userVisible values<br>
and the query api simply checks if the result candidate has a type<br>
which is visible.<br>
In contrast to the old one the new solution does not allow to hide a<br>
sub-class of a visible class. But since we never did that anyway and<br>
the resulting queries are faster and it allows us to remove a lot<br>
maintenance code (which eats CPU cycles and disk space) the change is<br>
worth it.<br></blockquote><div><br>I'm not sure about this. I kinda liked being \
able to hide certain resources from "normal" view. I always thought it \
could be used to hold Artem's decision resources. But then it wasn't really \
used.<br> </div><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
M +8 -13 libnepomukcore/query/query.cpp<br>
M +52 -0 services/storage/ontologyloader.cpp<br>
M +2 -1 services/storage/ontologyloader.h<br>
<br>
<a href="http://commits.kde.org/nepomuk-core/9476df60687ecb133ebc9dd2f9c893edc1ab097a" \
target="_blank">http://commits.kde.org/nepomuk-core/9476df60687ecb133ebc9dd2f9c893edc1ab097a</a><br>
<br>
diff --git a/libnepomukcore/query/query.cpp b/libnepomukcore/query/query.cpp<br>
index 943705d..6e242f4 100644<br>
--- a/libnepomukcore/query/query.cpp<br>
+++ b/libnepomukcore/query/query.cpp<br>
@@ -1,6 +1,6 @@<br>
/*<br>
This file is part of the Nepomuk KDE project.<br>
- Copyright (C) 2008-2010 Sebastian Trueg <<a \
href="mailto:trueg@kde.org">trueg@kde.org</a>><br> + Copyright (C) 2008-2012 \
Sebastian Trueg <<a href="mailto:trueg@kde.org">trueg@kde.org</a>><br> <br>
This library is free software; you can redistribute it and/or<br>
modify it under the terms of the GNU Library General Public<br>
@@ -450,22 +450,17 @@ QString Nepomuk::Query::Query::toSparqlQuery( SparqlFlags \
sparqlFlags ) const<br> QueryBuilderData qbd( d.constData(), sparqlFlags );<br>
<br>
<br>
- // We restrict results to user visible types only. There is no need for this \
with file queries as normally all files are visible.<br> //<br>
- // Here we use an optimizations:<br>
- // The storage service creates "nao:userVisible 1" entries for all \
visible resources. This contains two optimizations:<br>
- // - We use an integer instead of a boolean because Virtuoso does not support \
booleans and, thus, Soprano converts<br>
- // booleans into a fake type which is stored as a string. This makes \
comparision much slower.<br>
- // - Instead of using crappy inference via "?r a ?t . ?t nao:userVisible \
true" we can directly check the resources.<br>
- // - Instead of restricting the visibility to 1 we trick the Virtuoso query \
optimizer so it won't try to use the visibility<br>
- // as priority. We do this with a filter that does not test for equality. \
Using "?r nao:userVisible 1" has a drastic<br>
- // performance impact which the filter has not.<br>
+ // We restrict results to user visible types only. There is no need for this \
with file queries as normally all files are visible.<br> + // We need the \
additional type pattern to ensure that the variable ?r is actually available to the \
filter (in case there is only<br> + // UNIONs before.<br>
//<br>
QString userVisibilityRestriction;<br>
if( !(queryFlags()&NoResultRestrictions) && !d->m_isFileQuery ) \
{<br>
- userVisibilityRestriction = QString::fromLatin1("?r %1 %2 . \
FILTER(%2>0) . ")<br>
- .arg(Soprano::Node::resourceToN3(Soprano::Vocabulary::NAO::userVisible()),<br>
- qbd.uniqueVarName());<br>
+ userVisibilityRestriction = QString::fromLatin1("?r a %1 . FILTER \
EXISTS { ?r a [ %2 %3 ] . } . ")<br> + \
.arg(qbd.uniqueVarName(),<br> + \
Soprano::Node::resourceToN3(Soprano::Vocabulary::NAO::userVisible()),<br> + \
Soprano::Node::literalToN3(Soprano::LiteralValue(true)));<br> }<br>
<br>
<br>
diff --git a/services/storage/ontologyloader.cpp \
b/services/storage/ontologyloader.cpp<br> index c06a297..a09665b 100644<br>
--- a/services/storage/ontologyloader.cpp<br>
+++ b/services/storage/ontologyloader.cpp<br>
@@ -27,6 +27,7 @@<br>
#include <Soprano/PluginManager><br>
#include <Soprano/StatementIterator><br>
#include <Soprano/Parser><br>
+#include <Soprano/QueryResultIterator><br>
<br>
#include <KConfig><br>
#include <KConfigGroup><br>
@@ -42,14 +43,23 @@<br>
#include <kpluginfactory.h><br>
#include <kpluginloader.h><br>
<br>
+#include <Soprano/Vocabulary/RDFS><br>
+#include <Soprano/Vocabulary/NAO><br>
+#include <Soprano/Vocabulary/XMLSchema><br>
<br>
using namespace Soprano;<br>
+using namespace Soprano::Vocabulary;<br>
+<br>
+namespace {<br>
+ const char* s_typeVisibilityGraph = "nepomuk:/ctx/typevisibility";<br>
+}<br>
<br>
class Nepomuk::OntologyLoader::Private<br>
{<br>
public:<br>
Private( OntologyLoader* p )<br>
: forceOntologyUpdate( false ),<br>
+ someOntologyUpdated( false ),<br>
q( p ) {<br>
}<br>
<br>
@@ -59,6 +69,9 @@ public:<br>
bool forceOntologyUpdate;<br>
QStringList desktopFilesToUpdate;<br>
<br>
+ // true if at least one ontology has been updated<br>
+ bool someOntologyUpdated;<br>
+<br>
void updateOntology( const QString& filename );<br>
<br>
private:<br>
@@ -110,6 +123,8 @@ void Nepomuk::OntologyLoader::Private::updateOntology( const \
QString& filename )<br> }<br>
<br>
if( update ) {<br>
+ someOntologyUpdated = true;<br>
+<br>
QString mimeType = df.readEntry( "MimeType", QString() );<br>
<br>
const Soprano::Parser* parser<br>
@@ -179,6 +194,7 @@ Nepomuk::OntologyLoader::~OntologyLoader()<br>
<br>
void Nepomuk::OntologyLoader::updateLocalOntologies()<br>
{<br>
+ d->someOntologyUpdated = false;<br>
d->desktopFilesToUpdate = KGlobal::dirs()->findAllResources( \
"xdgdata-ontology", "*.ontology", \
KStandardDirs::Recursive|KStandardDirs::NoDuplicates );<br> \
if(d->desktopFilesToUpdate.isEmpty())<br>
kError() << "No ontology files found! Make sure the \
shared-desktop-ontologies project is installed and XDG_DATA_DIRS is set \
properly.";<br> @@ -201,6 +217,16 @@ void \
Nepomuk::OntologyLoader::updateNextOntology()<br> else {<br>
d->forceOntologyUpdate = false;<br>
d->updateTimer.stop();<br>
+<br>
+ // update graph visibility if something has changed or if we never did \
it<br> + const QUrl visibilityGraph = \
QUrl::fromEncoded(s_typeVisibilityGraph);<br> + if(d->someOntologyUpdated \
||<br> + !d->model->executeQuery(QString::fromLatin1("ask where \
{ graph %1 { ?s ?p ?o . } }")<br> + \
.arg(Soprano::Node::resourceToN3(visibilityGraph)),<br> + \
Soprano::Query::QueryLanguageSparql).boolValue()) {<br> + \
updateTypeVisibility();<br> + }<br>
+<br>
emit ontologyLoadingFinished(this);<br>
}<br>
}<br>
@@ -229,6 +255,7 @@ void Nepomuk::OntologyLoader::slotGraphRetrieverResult( KJob* job \
)<br>
// TODO: find a way to check if the imported version of the ontology<br>
// is newer than the already installed one<br>
if ( d->model->updateOntology( graphRetriever->statements(), \
QUrl()/*graphRetriever->url()*/ ) ) {<br> + updateTypeVisibility();<br>
emit ontologyUpdated( QString::fromAscii( \
graphRetriever->url().toEncoded() ) );<br> }<br>
else {<br>
@@ -237,4 +264,29 @@ void Nepomuk::OntologyLoader::slotGraphRetrieverResult( KJob* \
job )<br> }<br>
}<br>
<br>
+void Nepomuk::OntologyLoader::updateTypeVisibility()<br>
+{<br>
+ const QUrl visibilityGraph = QUrl::fromEncoded(s_typeVisibilityGraph);<br>
+<br>
+ // 1. remove all visibility values we added ourselves<br>
+ d->model->removeContext(visibilityGraph);<br>
+<br>
+ // 2. make rdfs:Resource non-visible (this is required since with KDE 4.9 we \
introduced a new<br> + // way of visibility handling which relies on types \
alone rather than visibility values on<br> + // resources. Any visible type \
will make all sub-types visible, too. If rdfs:Resource were<br> + // visible \
everything would be.<br> + d->model->removeAllStatements(RDFS::Resource(), \
NAO::userVisible(), Soprano::Node());<br> +<br>
+ // 3. Set each type visible which is not rdfs:Resource and does not have a \
non-visible parent<br> + \
d->model->executeQuery(QString::fromLatin1("insert into %1 { "<br> + \
"?t %2 'true'^^%3 . "<br> + \
"} where { "<br> + "?t a \
rdfs:Class . "<br> + "filter \
not exists { ?tt %2 'false'^^%3 . ?t rdfs:subClassOf ?tt . } }")<br> + \
.arg(Soprano::Node::resourceToN3(visibilityGraph),<br> + \
Soprano::Node::resourceToN3(NAO::userVisible()),<br> + \
Soprano::Node::resourceToN3(XMLSchema::boolean())),<br> + \
Soprano::Query::QueryLanguageSparql);<br> +}<br>
+<br>
#include "ontologyloader.moc"<br>
diff --git a/services/storage/ontologyloader.h \
b/services/storage/ontologyloader.h<br> index 4e0ce26..d714249 100644<br>
--- a/services/storage/ontologyloader.h<br>
+++ b/services/storage/ontologyloader.h<br>
@@ -1,5 +1,5 @@<br>
/* This file is part of the KDE Project<br>
- Copyright (c) 2007-2010 Sebastian Trueg <<a \
href="mailto:trueg@kde.org">trueg@kde.org</a>><br> + Copyright (c) 2007-2012 \
Sebastian Trueg <<a href="mailto:trueg@kde.org">trueg@kde.org</a>><br> <br>
This library is free software; you can redistribute it and/or<br>
modify it under the terms of the GNU Library General Public<br>
@@ -91,6 +91,7 @@ namespace Nepomuk {<br>
// a little async updating<br>
void updateNextOntology();<br>
void slotGraphRetrieverResult( KJob* job );<br>
+ void updateTypeVisibility();<br>
<br>
private:<br>
class Private;<br>
<br>
</blockquote></div><br>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic