[prev in list] [next in list] [prev in thread] [next in thread]
List: solr-user
Subject: RE: How to describe 2 entities in dataConfig for the DataImporter?
From: "Julio Castillo" <jcastillo () edgenuity ! com>
Date: 2008-05-31 19:11:03
Message-ID: F00373137D6341E59F0B9B61906CC4AC () minbar
[Download RAW message or body]
Not a problem Shalin,
On the contrary, thanks for all your hard work.
I will try it as soon as possible.
** julio
-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
Sent: Saturday, May 31, 2008 10:26 AM
To: solr-user@lucene.apache.org
Subject: Re: How to describe 2 entities in dataConfig for the DataImporter?
Hi Julio,
I've fixed the bug, can you please replace the exiting
TemplateTransformer.java in the SOLR-469.patch and use the attached
TemplateTransformer.java file. We'll add the changes to our next patch.
Sorry for all the trouble.
On Sat, May 31, 2008 at 10:31 PM, Noble Paul ??????? ??????
<noble.paul@gmail.com> wrote:
> julio,
> Looks like it is a bug.
> We can give u a new TemplateTransformer.java which we will incorporate
> in the next patch --Noble
>
> On Sat, May 31, 2008 at 12:24 AM, Julio Castillo
> <jcastillo@edgenuity.com> wrote:
>> I'm sorry Shalin, but I still get the same Null Pointer exception.
>> This is my complete dataconfig.xml (I remove the parallel entity to
>> narrow down the scope of the problem).
>> <dataConfig>
>> <document name="doc-1">
>> <entity name="vets" pk="id"
>> query="select id as idAlias,first_name,last_name FROM vets"
>> deltaQuery="SELECT id as idAlias FROM vets WHERE
>> last_modified > '${dataimporter.last_index_time}'"
>> transformer="TemplateTransformer">
>> <field column="id" name="id"
>> template="vets-${vets.idAlias}"/>
>> <field column="first_name" name="userName"/>
>> <field column="last_name" name="userName"/>
>> </entity>
>> </document>
>> </dataConfig>
>>
>> Thanks again.
>>
>> ** julio
>>
>> -----Original Message-----
>> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
>> Sent: Friday, May 30, 2008 11:38 AM
>> To: solr-user@lucene.apache.org
>> Subject: Re: How to describe 2 entities in dataConfig for the
DataImporter?
>>
>> The surname is used just as an example of a field.
>>
>> The NullPointerException is because the same field "id" tries to use
>> it's own value in a template. The template cannot contain the same
>> field on which it is being applied. I'd suggest that you get the id
>> aliased to another name, for example using a query "select id as
>> idAlias from vets" and then
>> use:
>> <field column="id" template="vets-${vets.idAlias}" />
>>
>> That should work, let me know if you face a problem.
>>
>> On Fri, May 30, 2008 at 10:40 PM, Julio Castillo
>> <jcastillo@edgenuity.com>
>> wrote:
>>> Thanks for all the leads.
>>> I did get however a null pointer exception while implementing it:
>>>
>>> May 30, 2008 9:57:50 AM
>>> org.apache.solr.handler.dataimport.EntityProcessorBase
>>> applyTransformer
>>> WARNING: transformer threw error
>>> java.lang.NullPointerException
>>> at
>>> org.apache.solr.handler.dataimport.TemplateTransformer.transformRow(
>>> Te
>>> mplate
>>> Transformer.java:55)
>>> at
>>> org.apache.solr.handler.dataimport.EntityProcessorBase.applyTransfor
>>> me
>>> r(Enti
>>> tyProcessorBase.java:186)
>>>
>>> Looking at the source code, it appears that the resolverMap is null.
>>> The resolver returned null given the entityName.
>>>
>>> Looking at the documentation, there is the reference to a
eparent.surname.
>>> The example says:
>>> <field column="namedesc" template="hello${e.name},${eparent.surname}"
>>> />
>>>
>>> I'm afraid, I don't know what an eparent.surname is. This is my
>>> current dataconfig.xml configuration excerpt:
>>>
>>> <entity name="vets" pk="id" query="..." deltaQuery="..."
>>> transformer="TemplateTransformer">
>>> <field column="id" name="id" template="vets-${vets.id}"/> ...
>>>
>>> Am I missing a surname? Whatever that may be?
>>>
>>> Thanks
>>>
>>> ** julio
>>>
>>> -----Original Message-----
>>> From: Noble Paul ??????? ?????? [mailto:noble.paul@gmail.com]
>>> Sent: Thursday, May 29, 2008 11:10 PM
>>> To: solr-user@lucene.apache.org
>>> Subject: Re: How to describe 2 entities in dataConfig for the
>> DataImporter?
>>>
>>> Sorry I forgot to mention that.
>>> http://wiki.apache.org/solr/DataImportHandler#head-a6916b30b5d7605a9
>>> 90
>>> fb03c4
>>> ff461b3736496a9
>>> --Noble
>>>
>>> On Fri, May 30, 2008 at 11:37 AM, Shalin Shekhar Mangar
>>> <shalinmangar@gmail.com> wrote:
>>>> You need to enable TemplateTransformer for your entity. For example:
>>>> <entity name="owners" pk="id" query="...."
>>>> transformer="TemplateTransformer">
>>>>
>>>> On Fri, May 30, 2008 at 11:31 AM, Julio Castillo
>>>> <jcastillo@edgenuity.com> wrote:
>>>>> Noble,
>>>>> I tried the template setting for the "id" field, but I didn't
>>>>> notice any different behavior. I also didn't see where this would be
reflected.
>>>>> I looked at the fields and the debug output for the dataImporter
>>>>> and couldn't see any reference to a modified id name (per the
>>>>> template instructions).
>>>>>
>>>>> The behavior in the end seemed to be the same. Did I miss anything?
>>>>> I assume that the <uniqueKey>id</uniqueKey> setting in the
>>>>> schema.xml remains the same?
>>>>>
>>>>> Thanks again
>>>>>
>>>>> ** julio
>>>>>
>>>>> -----Original Message-----
>>>>> From: Noble Paul ??????? ?????? [mailto:noble.paul@gmail.com]
>>>>> Sent: Thursday, May 29, 2008 9:46 PM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>> DataImporter?
>>>>>
>>>>> Consider constructing the id concatenating an extra string for
>>>>> each document . You can construct that field using the
TeplateTransformer.
>>>>> in the entity owners keep the id as
>>>>>
>>>>> <field column="id" name="id" template="owners-${owners.id}"/> and
>>>>> in vets <field column="id" name="id" template="vets-${vets.id}"/>
>>>>>
>>>>> or anything else which can make it unique
>>>>>
>>>>> --Noble
>>>>>
>>>>> On Fri, May 30, 2008 at 10:05 AM, Shalin Shekhar Mangar
>>>>> <shalinmangar@gmail.com> wrote:
>>>>>> That will happen only if id is the uniqueKey in Solr and the id
>>>>>> coming from both your tables have same values. In that case, they
>>>>>> will overwrite each other. You will need a separate uniqueKey (on
>>>>>> other than id field).
>>>>>>
>>>>>> On Fri, May 30, 2008 at 6:34 AM, Julio Castillo
>>>>>> <jcastillo@edgenuity.com>
>>>>> wrote:
>>>>>>> Thanks Shalin,
>>>>>>> I tried putting everything under the same document (two
>>>>>>> different unrelated entities), and got a bit further.
>>>>>>>
>>>>>>> My problem now appears to be both of them stepping on each other
>>>>>>> due to
>>>>> "id"
>>>>>>> conflicts. Currently my id is defined in my schema as <field
>>>>>>> name="id" type="long" indexed="true" stored="true"
>>>>>>> required="true"/>
>>>>>>>
>>>>>>> Do I have to create a new "id" field?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> ** julio
>>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
>>>>>>> Sent: Thursday, May 29, 2008 11:40 AM
>>>>>>> To: solr-user@lucene.apache.org
>>>>>>> Subject: Re: How to describe 2 entities in dataConfig for the
>>>>> DataImporter?
>>>>>>>
>>>>>>> Hi Julio,
>>>>>>>
>>>>>>> The first data-config is correct.
>>>>>>>
>>>>>>> You're running DataImportHandler in debug mode which creates
>>>>>>> only the first 10 documents by default. You can also add count=N
>>>>>>> to index only the first N documents. But this is intended only
>>>>>>> for debugging purposes. If you want to do a full-import just use
>>>>>>> /dataimport?command=full-import. By default, a full-import
>>>>>>> automatically
>>>>> cleans and commits.
>>>>>>>
>>>>>>> Hope that helps.
>>>>>>>
>>>>>>> On Thu, May 29, 2008 at 11:42 PM, Julio Castillo
>>>>>>> <jcastillo@edgenuity.com>
>>>>>>> wrote:
>>>>>>>> I have 2 dB tables unrelated to each other that I want to index.
>>>>>>>>
>>>>>>>> I have tried 2 approaches for specifying them in my
>>>>>>>> data-config.xml
>>>>> file.
>>>>>>>> None of them seem to work (it seems I can only get data for the
>>>>>>>> first one listed).
>>>>>>>>
>>>>>>>> CASE 1)
>>>>>>>> <dataConfig>
>>>>>>>> <document name="test">
>>>>>>>> <entity name="owners" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> <entity name="vets" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> CASE 2)
>>>>>>>> <dataConfig>
>>>>>>>> <document name="test-1">
>>>>>>>> <entity name="owners" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> <document name="test-2">
>>>>>>>> <entity name="vets" pk="id" query="....">
>>>>>>>> <field column="id" name="id"/>
>>>>>>>> <field column=name" name="userName"/>
>>>>>>>> </entity>
>>>>>>>> </document>
>>>>>>>> </dataConfig>
>>>>>>>>
>>>>>>>> I ran the dataImporter as follows
>>>>>>>> http://localhost:2455/solr/dataimport?command=full-import&debug
>>>>>>>> =o
>>>>>>>> n
>>>>>>>> &v
>>>>>>>> er
>>>>>>>> bose=t
>>>>>>>> rue&clean=true&commit=true
>>>>>>>>
>>>>>>>> After running Case 1 the output only shows documents under
>>>>> entity:owners.
>>>>>>>> And an empty reference to entity:vets (no data).
>>>>>>>>
>>>>>>>> After running Case 2, the output only shows documents under
>>>>> entity:owners.
>>>>>>>> No references to entity:vets.
>>>>>>>>
>>>>>>>> In either case, I have commented out the entity:owners and I do
>>>>>>>> see the entity:vets data then.
>>>>>>>>
>>>>>>>> Thanks for your assistance
>>>>>>>>
>>>>>>>> Julio Castillo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shalin Shekhar Mangar.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shalin Shekhar Mangar.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> --Noble Paul
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shalin Shekhar Mangar.
>>>>
>>>
>>>
>>>
>>> --
>>> --Noble Paul
>>>
>>>
>>
>>
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
>>
>
>
>
> --
> --Noble Paul
>
--
Regards,
Shalin Shekhar Mangar.
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic