[prev in list] [next in list] [prev in thread] [next in thread]
List: lucene-dev
Subject: Re: [JENKINS] Lucene-MMAPv2-Windows (64bit/jdk-17.0.3) - Build # 131 - Unstable!
From: Patrick Zhai <zhai7631 () gmail ! com>
Date: 2022-07-28 6:41:41
Message-ID: CA+vuXU7_vdYEQ1uY0VNYN_OEUTUzkjfU1p_YXJ2q9zcX0NvJtQ () mail ! gmail ! com
[Download RAW message or body]
I executed the same test on my windows pc and macbook, seems on the windows
there's high possibility of failing while on the mac it always pass. It
also seems on windows it is quite slow, after discussion with Mike I think
probably on windows the merging is slower so that more files are kept open,
and thus reaches the limit.
I'll try increase the limit and see how it goes
On Mon, Jul 25, 2022 at 10:35 AM Patrick Zhai <zhai7631@gmail.com> wrote:
> Seems the error was thrown from HandleLimitFS.java, which enforce an
> artificial limitation on the number of files opened it seems. But the weird
> thing is why I only see such errors on Windows builds? I'll try to dig more
> on this error.
>
> On Sun, Jul 24, 2022 at 8:41 PM Policeman Jenkins Server <
> jenkins@thetaphi.de> wrote:
>
> > Build: https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/131/
> > Java: 64bit/jdk-17.0.3 -XX:-UseCompressedOops -XX:+UseG1GC
> >
> > 1 tests failed.
> > FAILED:
> > org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader
> >
> > Error Message:
> > java.nio.file.FileSystemException:
> > C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\core\build\tmp\tests-tmp\l \
> > ucene.index.TestIndexWriterMergePolicy_93D10D9F5BC86FBA-001\index-MMapDirectory-014\_149_1_Lucene90_0.dvm:
> > Too many open files
> >
> > Stack Trace:
> > java.nio.file.FileSystemException:
> > C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\core\build\tmp\tests-tmp\l \
> > ucene.index.TestIndexWriterMergePolicy_93D10D9F5BC86FBA-001\index-MMapDirectory-014\_149_1_Lucene90_0.dvm:
> > Too many open files
> > at
> > org.apache.lucene.tests.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:67)
> > at
> > org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)
> > at
> > org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)
> > at
> > org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)
> > at java.base/java.nio.file.Files.newOutputStream(Files.java:228)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)
> > at
> > org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:385)
> > at
> > org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)
> > at
> > org.apache.lucene.tests.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:717)
> > at
> > org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)
> > at
> > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)
> > at
> > org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)
> > at
> > org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.<init>(Lucene90DocValuesConsumer.java:91)
> > at
> > org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat.fieldsConsumer(Lucene90DocValuesFormat.java:148)
> > at
> > org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:231)
> > at
> > org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:162)
> > at
> > org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:104)
> > at
> > org.apache.lucene.index.ReadersAndUpdates.handleDVUpdates(ReadersAndUpdates.java:397)
> > at
> > org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:624)
> > at
> > org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251)
> > at
> > org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3734)
> > at
> > org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591)
> > at
> > org.apache.lucene.index.IndexWriter$4.getReader(IndexWriter.java:6301)
> > at
> > org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:488)
> > at
> > org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:420)
> > at
> > org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:793)
> > at
> > org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader(TestIndexWriterMergePolicy.java:737)
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> > Method)
> > at
> > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
> > at
> > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.base/java.lang.reflect.Method.invoke(Method.java:568)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)
> > at
> > org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
> > at
> > org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> > org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
> > at
> > org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> > org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)
> > at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)
> > at
> > com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)
> > at
> > org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> > org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
> > at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> > com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
> > at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> > org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
> > at
> > org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
> > at
> > org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
> > at
> > org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
> > at
> > org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
> > at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> > at
> > com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
> > at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)
> > at
> > com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)
> > at java.base/java.lang.Thread.run(Thread.java:833)
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: builds-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: builds-help@lucene.apache.org
>
>
[Attachment #3 (text/html)]
<div dir="ltr">I executed the same test on my windows pc and macbook, seems on the \
windows there's high possibility of failing while on the mac it always pass. It \
also seems on windows it is quite slow, after discussion with Mike I think probably \
on windows the merging is slower so that more files are kept open, and thus reaches \
the limit.<div><br></div><div>I'll try increase the limit and see how it \
goes</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On \
Mon, Jul 25, 2022 at 10:35 AM Patrick Zhai <<a \
href="mailto:zhai7631@gmail.com">zhai7631@gmail.com</a>> \
wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px \
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Seems \
the error was thrown from HandleLimitFS.java, which enforce an artificial limitation \
on the number of files opened it seems. But the weird thing is why I only see such \
errors on Windows builds? I'll try to dig more on this error.</div><br><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sun, Jul 24, 2022 at 8:41 PM \
Policeman Jenkins Server <<a href="mailto:jenkins@thetaphi.de" \
target="_blank">jenkins@thetaphi.de</a>> wrote:<br></div><blockquote \
class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid \
rgb(204,204,204);padding-left:1ex">Build: <a \
href="https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/131/" rel="noreferrer" \
target="_blank">https://jenkins.thetaphi.de/job/Lucene-MMAPv2-Windows/131/</a><br>
Java: 64bit/jdk-17.0.3 -XX:-UseCompressedOops -XX:+UseG1GC<br>
<br>
1 tests failed.<br>
FAILED: org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader<br>
<br>
Error Message:<br>
java.nio.file.FileSystemException: \
C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\core\build\tmp\tests-tmp\lucen \
e.index.TestIndexWriterMergePolicy_93D10D9F5BC86FBA-001\index-MMapDirectory-014\_149_1_Lucene90_0.dvm: \
Too many open files<br> <br>
Stack Trace:<br>
java.nio.file.FileSystemException: \
C:\Users\jenkins\workspace\Lucene-MMAPv2-Windows\lucene\core\build\tmp\tests-tmp\lucen \
e.index.TestIndexWriterMergePolicy_93D10D9F5BC86FBA-001\index-MMapDirectory-014\_149_1_Lucene90_0.dvm: \
Too many open files<br>
at org.apache.lucene.tests.mockfile.HandleLimitFS.onOpen(HandleLimitFS.java:67)<br>
at org.apache.lucene.tests.mockfile.HandleTrackingFS.callOpenHook(HandleTrackingFS.java:82)<br>
at org.apache.lucene.tests.mockfile.HandleTrackingFS.newOutputStream(HandleTrackingFS.java:163)<br>
at org.apache.lucene.tests.mockfile.FilterFileSystemProvider.newOutputStream(FilterFileSystemProvider.java:198)<br>
at java.base/java.nio.file.Files.newOutputStream(Files.java:228)<br>
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:392)<br>
at org.apache.lucene.store.FSDirectory$FSIndexOutput.<init>(FSDirectory.java:385)<br>
at org.apache.lucene.store.FSDirectory.createOutput(FSDirectory.java:220)<br>
at org.apache.lucene.tests.store.MockDirectoryWrapper.createOutput(MockDirectoryWrapper.java:717)<br>
at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43)<br>
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)<br>
at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:41)<br>
at org.apache.lucene.codecs.lucene90.Lucene90DocValuesConsumer.<init>(Lucene90DocValuesConsumer.java:91)<br>
at org.apache.lucene.codecs.lucene90.Lucene90DocValuesFormat.fieldsConsumer(Lucene90DocValuesFormat.java:148)<br>
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:231)<br>
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:162)<br>
at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:104)<br>
at org.apache.lucene.index.ReadersAndUpdates.handleDVUpdates(ReadersAndUpdates.java:397)<br>
at org.apache.lucene.index.ReadersAndUpdates.writeFieldUpdates(ReadersAndUpdates.java:624)<br>
at org.apache.lucene.index.ReaderPool.writeAllDocValuesUpdates(ReaderPool.java:251)<br>
at org.apache.lucene.index.IndexWriter.writeReaderPool(IndexWriter.java:3734)<br>
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:591)<br>
at org.apache.lucene.index.IndexWriter$4.getReader(IndexWriter.java:6301)<br>
at org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:488)<br>
at org.apache.lucene.tests.index.RandomIndexWriter.getReader(RandomIndexWriter.java:420)<br>
at org.apache.lucene.index.TestIndexWriterMergePolicy.stressUpdateSameDocumentWithMergeOnX(TestIndexWriterMergePolicy.java:793)<br>
at org.apache.lucene.index.TestIndexWriterMergePolicy.testStressUpdateSameDocumentWithMergeOnGetReader(TestIndexWriterMergePolicy.java:737)<br>
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native \
Method)<br>
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)<br>
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)<br>
at java.base/java.lang.reflect.Method.invoke(Method.java:568)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1754)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:942)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:978)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:992)<br>
at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)<br>
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)<br>
at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)<br>
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)<br>
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)<br>
at org.junit.rules.RunRules.evaluate(RunRules.java:20)<br>
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)<br>
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)<br>
at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:819)<br>
at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:470)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:951)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:836)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:887)<br>
at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:898)<br>
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)<br>
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)<br>
at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)<br>
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)<br>
at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)<br>
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)<br>
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)<br>
at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)<br>
at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)<br>
at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)<br>
at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)<br>
at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)<br>
at org.junit.rules.RunRules.evaluate(RunRules.java:20)<br>
at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)<br>
at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:370)<br>
at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:826)<br>
at java.base/java.lang.Thread.run(Thread.java:833)<br>
<br>
---------------------------------------------------------------------<br>
To unsubscribe, e-mail: <a href="mailto:builds-unsubscribe@lucene.apache.org" \
target="_blank">builds-unsubscribe@lucene.apache.org</a><br> For additional commands, \
e-mail: <a href="mailto:builds-help@lucene.apache.org" \
target="_blank">builds-help@lucene.apache.org</a></blockquote></div> \
</blockquote></div>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic