[prev in list] [next in list] [prev in thread] [next in thread] 

List:       solr-user
Subject:    Re: How might one search for dupe IDs other than faceting on the ID field?
From:       Dotan Cohen <dotancohen () gmail ! com>
Date:       2013-07-31 5:29:26
Message-ID: CAKDXFkMPY7Nf2SeVDe+MXGBUGojP4fe1yFqOVy8+AufPqWLSRQ () mail ! gmail ! com
[Download RAW message or body]

On Tue, Jul 30, 2013 at 11:00 PM, Mikhail Khludnev
<mkhludnev@griddynamics.com> wrote:
> Dotan,
> 
> Could you please provide more line of the stack trace?

Sure, thanks:
<response><lst name="error"><str
name="msg">java.lang.OutOfMemoryError: Java heap space</str><str
name="trace">java.lang.RuntimeException: java.lang.OutOfMemoryError:
Java heap space
    at org.apache.solr.servlet.SolrDispatchFilter.sendError(SolrDispatchFilter.java:670)
  at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:380)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
  at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1307)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:453)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:560)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
  at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
  at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006)
  at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
  at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
  at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:365)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485)
  at org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
  at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926)
  at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988)
  at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
    at org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
  at org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
  at java.lang.Thread.run(Thread.java:679)
Caused by: java.lang.OutOfMemoryError: Java heap space
</str><int name="code">500</int></lst></response>


> I have no idea why it made worse at 4.3. I know that 4.3 can use facets
> backed on DocValues, which are modest for the heap. But from what I saw,
> but can be wrong it's disabled from numeric facets. Hence, I can suggest to
> reindex id as string docvalues and hope for them. However, it's doubtful to
> reindex everything without strong guaranties.

We also had issues with 4.2, though I really don't remember the
details. Some simple queries such as 'q=ubuntu' would take tens of
seconds whereas on 4.1 it was almost instantaneous. In fact, even in
4.3 I feel that things have slowed down terribly (3000 ms on simple
queries whereas 4.1 would do it in tens or maximum a few hundred). Of
course, the index is constantly growing so that may be a factor. Note
that in both cases the index and configuration was carryover from 4.1
so that may have been an issue. Moving back from 4.2 to 4.1 I bit the
bullet and deleted the extant documents. I no longer have that luxury
now.


> Also, I checked source code of
> http://wiki.apache.org/solr/TermsComponentand found that it can be
> really memory modest (ie without sort nor limit).
> Be aware that df-s returned by that component are unaware of deleted
> document, hence expungeDeletes before.
> 

Thank you, I will look into that.

-- 
Dotan Cohen

http://gibberish.co.il
http://what-is-what.com


[prev in list] [next in list] [prev in thread] [next in thread]