[prev in list] [next in list] [prev in thread] [next in thread]
List: lucene-dev
Subject: Re: [Data Import Handler] proposal: make FileListEntityProcessor streaming
From: Marco Bolis <mbolis1984 () gmail ! com>
Date: 2020-07-13 14:13:15
Message-ID: CAHSnVzXkF03Db6O0dzPNeJvu26z6sFG8e8LFn2uoS29Ey0AcmQ () mail ! gmail ! com
[Download RAW message or body]
Thank you very much.
I opened PRs to both projects.
Regards,
Marco
Il giorno ven 10 lug 2020 alle ore 05:04 Noble Paul <noble.paul@gmail.com>
ha scritto:
> The project will go live anytime from now. It means a user can use it
> on any release newer than Solr 8.6 . Even if you provide a fix in the
> current 8.x branch, it will not be available before Solr 8.7 release.
> OTOH, DIH plugin will have bug fix releases independent of Solr
> releases and every user will be able to upgrade their plugin without
> upgrading their Solr.
>
> So, please give PRs to both the external plugin and to Solr
>
> On Fri, Jul 10, 2020 at 2:58 AM Marco Bolis <mbolis1984@gmail.com> wrote:
> >
> > I see.
> > How is the transition going to work Eric?
> > I understand the community supported project is going to take over from
> Solr 9.0, is that correct? Is DIH code on the Lucene side going to freeze
> soon?
> > Thank you for the heads up.
> >
> > Regards,
> > Marco
> >
> >
> > Il giorno gio 9 lug 2020 alle ore 18:49 Eric Pugh <
> epugh@opensourceconnections.com> ha scritto:
> > >
> > > Another thought….
> > >
> > > Since DIH is moving to a community supported (
> https://github.com/rohitbemax/dataimporthandler) plugin for Solr, maybe
> you want to focus your efforts on that project?
> > >
> > > One of the reasons for moving DIH into it's own plugin it to open the
> door to more contributions from the community, and this is a good example!
> > >
> > >
> > >
> > > On Jul 9, 2020, at 12:09 PM, Erick Erickson <erickerickson@gmail.com>
> wrote:
> > >
> > > If you've created a JIRA login, there should be a button on the JIRA
> about "attach files". It's perfectly OK to attach a diff file to the JIRA.
> It's preferred to just label it SOLR-#####.patch. Successive versions of
> the patch should have the exact same name, the old ones are grayed out
> making it easy to know what the most recent one is without losing the old
> versions. No big deal though.
> > >
> > > If you're familiar with GIT and have your own fork somewhere, it's just
> the usual process of creating a Pull Request from your GitHub repo. If you
> mention the JIRA when you create the PR by starting the title with
> "SOLR-#####: any comments you want to make", it'll automagically be linked
> to the JIRA you created. I've personally found this a bit confusing because
> the title you edit is not the first screen when you hit the "create PR"
> button. If the automagic linking doesn't work, just paste a link to the PR
> in the comments.
> > >
> > > Don't stress over it, if making a PR is bothersome, just attach a diff
> file. Either one is fine. Code reviews are easier with a PR, but depending
> on the size of the patch the utility of easy reviews may be only marginally
> beneficial.
> > >
> > > Best,
> > > Erick
> > >
> > > On Jul 9, 2020, at 11:23 AM, Marco Bolis <mbolis1984@gmail.com> wrote:
> > >
> > > Thanks for the answers.
> > >
> > > Excuse me, I'm new to this: how am I supposed to attach the patch / PR
> to the issue?
> > > Is it ok to add a diff as attachment?
> > > Should I open the PR and link to it from the issue?
> > >
> > > Thank you very much, regards,
> > > Marco
> > >
> > > Il giorno gio 9 lug 2020 alle ore 17:06 Erick Erickson <
> erickerickson@gmail.com> ha scritto:
> > > Marco:
> > >
> > > Thanks for volunteering your fix!
> > >
> > > The best way is to raise a JIRA, see:
> https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-JIRAtips(ourissue/bugtracker)
> and attach a patch or pull request. From there we can discuss/give
> feedback/add to the repo, etc.
> > >
> > > Best,
> > > Erick
> > >
> > > On Jul 9, 2020, at 9:56 AM, Marco Bolis <mbolis1984@gmail.com> wrote:
> > >
> > > Hello,
> > >
> > > I just wrote a patch to make FileListEntityProcessor work by streaming,
> using Java 8 Stream and NIO2, instead of buffering the entire file list in
> memory.
> > > I had to do it because I had a very large list of files (upwards of 1M)
> and kept going OOM.
> > >
> > > I wish I could contribute this patch, if it is deemed useful.
> > >
> > > Regards,
> > > Marco
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: dev-help@lucene.apache.org
> > >
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: dev-help@lucene.apache.org
> > >
> > >
> > > _______________________
> > > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467
> > http://www.opensourceconnections.com | My Free/Busy
> > > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed
> > > This e-mail and all contents, including attachments, is considered to
> be Company Confidential unless explicitly stated otherwise, regardless of
> whether attachments are marked as such.
> > >
>
>
> --
> -----------------------------------------------------
> Noble Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
[Attachment #3 (text/html)]
<div dir="ltr">Thank you very much.<div>I opened PRs to both \
projects.</div><div>Regards,</div><div>Marco</div></div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 10 lug 2020 alle \
ore 05:04 Noble Paul <<a \
href="mailto:noble.paul@gmail.com">noble.paul@gmail.com</a>> ha \
scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">The project will go \
live anytime from now. It means a user can use it<br> on any release newer than Solr \
8.6 . Even if you provide a fix in the<br> current 8.x branch, it will not be \
available before Solr 8.7 release.<br> OTOH, DIH plugin will have bug fix releases \
independent of Solr<br> releases and every user will be able to upgrade their plugin \
without<br> upgrading their Solr.<br>
<br>
So, please give PRs to both the external plugin and to Solr<br>
<br>
On Fri, Jul 10, 2020 at 2:58 AM Marco Bolis <<a href="mailto:mbolis1984@gmail.com" \
target="_blank">mbolis1984@gmail.com</a>> wrote:<br> ><br>
> I see.<br>
> How is the transition going to work Eric?<br>
> I understand the community supported project is going to take over from Solr \
9.0, is that correct? Is DIH code on the Lucene side going to freeze soon?<br> > \
Thank you for the heads up.<br> ><br>
> Regards,<br>
> Marco<br>
><br>
><br>
> Il giorno gio 9 lug 2020 alle ore 18:49 Eric Pugh <<a \
href="mailto:epugh@opensourceconnections.com" \
target="_blank">epugh@opensourceconnections.com</a>> ha scritto:<br> >><br>
>> Another thought….<br>
>><br>
>> Since DIH is moving to a community supported (<a \
href="https://github.com/rohitbemax/dataimporthandler" rel="noreferrer" \
target="_blank">https://github.com/rohitbemax/dataimporthandler</a>) plugin for Solr, \
maybe you want to focus your efforts on that project?<br> >><br>
>> One of the reasons for moving DIH into it's own plugin it to open the door \
to more contributions from the community, and this is a good example!<br> \
>><br> >><br>
>><br>
>> On Jul 9, 2020, at 12:09 PM, Erick Erickson <<a \
href="mailto:erickerickson@gmail.com" target="_blank">erickerickson@gmail.com</a>> \
wrote:<br> >><br>
>> If you've created a JIRA login, there should be a button on the JIRA about \
"attach files". It's perfectly OK to attach a diff file to the JIRA. It's preferred \
to just label it SOLR-#####.patch. Successive versions of the patch should have the \
exact same name, the old ones are grayed out making it easy to know what the most \
recent one is without losing the old versions. No big deal though.<br> >><br>
>> If you're familiar with GIT and have your own fork somewhere, it's just the \
usual process of creating a Pull Request from your GitHub repo. If you mention the \
JIRA when you create the PR by starting the title with "SOLR-#####: any comments you \
want to make", it'll automagically be linked to the JIRA you created. I've personally \
found this a bit confusing because the title you edit is not the first screen when \
you hit the "create PR" button. If the automagic linking doesn't work, just paste a \
link to the PR in the comments.<br> >><br>
>> Don't stress over it, if making a PR is bothersome, just attach a diff file. \
Either one is fine. Code reviews are easier with a PR, but depending on the size of \
the patch the utility of easy reviews may be only marginally beneficial.<br> \
>><br> >> Best,<br>
>> Erick<br>
>><br>
>> On Jul 9, 2020, at 11:23 AM, Marco Bolis <<a \
href="mailto:mbolis1984@gmail.com" target="_blank">mbolis1984@gmail.com</a>> \
wrote:<br> >><br>
>> Thanks for the answers.<br>
>><br>
>> Excuse me, I'm new to this: how am I supposed to attach the patch / PR \
to the issue?<br> >> Is it ok to add a diff as attachment?<br>
>> Should I open the PR and link to it from the issue?<br>
>><br>
>> Thank you very much, regards,<br>
>> Marco<br>
>><br>
>> Il giorno gio 9 lug 2020 alle ore 17:06 Erick Erickson <<a \
href="mailto:erickerickson@gmail.com" target="_blank">erickerickson@gmail.com</a>> \
ha scritto:<br> >> Marco:<br>
>><br>
>> Thanks for volunteering your fix!<br>
>><br>
>> The best way is to raise a JIRA, see: <a \
href="https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-JIRAtips(ourissue/bugtracker)" \
rel="noreferrer" target="_blank">https://cwiki.apache.org/confluence/display/solr/HowToContribute#HowToContribute-JIRAtips(ourissue/bugtracker)</a> \
and attach a patch or pull request. From there we can discuss/give feedback/add to \
the repo, etc.<br> >><br>
>> Best,<br>
>> Erick<br>
>><br>
>> On Jul 9, 2020, at 9:56 AM, Marco Bolis <<a \
href="mailto:mbolis1984@gmail.com" target="_blank">mbolis1984@gmail.com</a>> \
wrote:<br> >><br>
>> Hello,<br>
>><br>
>> I just wrote a patch to make FileListEntityProcessor work by streaming, \
using Java 8 Stream and NIO2, instead of buffering the entire file list in \
memory.<br> >> I had to do it because I had a very large list of files (upwards \
of 1M) and kept going OOM.<br> >><br>
>> I wish I could contribute this patch, if it is deemed useful.<br>
>><br>
>> Regards,<br>
>> Marco<br>
>><br>
>><br>
>><br>
>> ---------------------------------------------------------------------<br>
>> To unsubscribe, e-mail: <a href="mailto:dev-unsubscribe@lucene.apache.org" \
target="_blank">dev-unsubscribe@lucene.apache.org</a><br> >> For additional \
commands, e-mail: <a href="mailto:dev-help@lucene.apache.org" \
target="_blank">dev-help@lucene.apache.org</a><br> >><br>
>><br>
>><br>
>> ---------------------------------------------------------------------<br>
>> To unsubscribe, e-mail: <a href="mailto:dev-unsubscribe@lucene.apache.org" \
target="_blank">dev-unsubscribe@lucene.apache.org</a><br> >> For additional \
commands, e-mail: <a href="mailto:dev-help@lucene.apache.org" \
target="_blank">dev-help@lucene.apache.org</a><br> >><br>
>><br>
>> _______________________<br>
>> Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | \
<a href="http://www.opensourceconnections.com" rel="noreferrer" \
target="_blank">http://www.opensourceconnections.com</a> | My Free/Busy<br> >> \
Co-Author: Apache Solr Enterprise Search Server, 3rd Ed<br> >> This e-mail and \
all contents, including attachments, is considered to be Company Confidential unless \
explicitly stated otherwise, regardless of whether attachments are marked as \
such.<br> >><br>
<br>
<br>
-- <br>
-----------------------------------------------------<br>
Noble Paul<br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href="mailto:dev-unsubscribe@lucene.apache.org" \
target="_blank">dev-unsubscribe@lucene.apache.org</a><br> For additional commands, \
e-mail: <a href="mailto:dev-help@lucene.apache.org" \
target="_blank">dev-help@lucene.apache.org</a><br> <br>
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic