[prev in list] [next in list] [prev in thread] [next in thread]
List: tomcat-user
Subject: Re: AW: AW: AW: AW: Filehandle left open when using sendfile
From: Niranjan Rao <nhrdls () gmail ! com>
Date: 2022-06-21 21:46:09
Message-ID: d16ce8f8-03a6-136a-5d30-fed7b23dc5b2 () gmail ! com
[Download RAW message or body]
On 6/20/22 23:59, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Mark,
>
> > -----Ursprüngliche Nachricht-----
> > Von: Mark Thomas <markt@apache.org>
> > Gesendet: Montag, 20. Juni 2022 22:13
> > An: users@tomcat.apache.org
> > Betreff: Re: AW: AW: AW: Filehandle left open when using sendfile
> >
> > On 20/06/2022 11:39, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > > Hello Mark,
> > >
> > > thanks for your reply!
> > >
> > > > -----Ursprüngliche Nachricht-----
> > > > Von: Mark Thomas <markt@apache.org>
> > > > Gesendet: Montag, 20. Juni 2022 12:06
> > > > An: users@tomcat.apache.org
> > > > Betreff: Re: AW: AW: Filehandle left open when using sendfile
> > > >
> > > > On 16/06/2022 19:58, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > > >
> > > > <snip/>
> > > >
> > > > > In the meantime I stumbled upon this bug-Report:
> > > > > https://bugs.java.com/bugdatabase/view_bug.do?bug_id=4715154
> > > > > So maybe the problem lies even deeper.
> > > > > Similar description here:
> > > > > https://cemerick.com/blog/2006/08/30/memory-mapping-files-in-java-
> > > > caus
> > > > > es-problems.html
> > > > >
> > > > > Some ppl suggest to use java.lang.ref.Cleaner or don't use Memory-
> > > > Mapped files under Windows.
> > > > > I don't know if there are other solutions.
> > > > Your research looks to be exhaustive. I can't find any better ideas.
> > > >
> > > > Using the java.lang.ref.Cleaner looks to be a viable option. We know
> > > > when the mapped file is no longer being used. However, that requires
> > > > Java 12 onwards.
> > > >
> > > > This is only going to be required if the file locking is an issue. In
> > > > read-only scenarios or when using an OS other than Windows it won't be
> > an issue.
> > > > So, what do we want to do?
> > > >
> > > > 1. Disable sendfile for HTTP/2 if running on Windows?
> > > >
> > > > 2. Document the potential issues with sendfile + HTTP/2 + Windows if
> > > > resources are not read-only?
> > > >
> > > > 3. Use the JreCompat mechanism to clear the references if possible:
> > > > - if running on Windows
> > > > - on all OSes
> > > > - if enabled via configuration
> > > >
> > > > Something else?
> > > >
> > > > Mark
> > > >
> > > > ---------------------------------------------------------------------
> > > > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > > > For additional commands, e-mail: users-help@tomcat.apache.org
> > > I did some further searching on this topic.
> > > Several posts disregard using java.lang.ref.Cleaner because if the buffer is
> > used afterwards, it will crash the VM. But if used carefully it works.
> >
> > If we use this option, it should be possible to use it appropriately carefully.
> >
> > > About your suggestions:
> > > 2) Documenting would be helpful, if lock can't be prevented.
> > > I also found documentation at e.g.
> > https://docs.oracle.com/javase/9/docs/api/java/nio/channels/FileChannel.h
> > tml#map-java.nio.channels.FileChannel.MapMode-long-long-
> > > " The buffer and the mapping that it represents will remain valid until
> > the buffer itself is garbage-collected."
> >
> > Which is essentially the problem. Using the Cleaner would clean up the
> > reference sooner.
> >
> > > 3) As JreCompat is a bit risky, enabling via config sounds safe to me.
> > JreCompat is perfectly safe. The jdk.internal.misc.Unsafe API is where the
> > risk is and this is primarily the risk of the crash mentioned above that we
> > should be able to avoid.
> >
> > > Some other (theoretical?) options:
> > > 4) In an older version of Tomcat native lib there seemed to be a native
> > Implementation of MMap: https://tomcat.apache.org/tomcat-10.0-
> > doc/api/org/apache/tomcat/jni/Mmap.html
> > > I read that this was an alternative to the Java memory mapped
> > > file. But it was removed in newer versions. Maybe it can be
> > > resurrected for this case and used if native lib is available(?)
> > Sorry, no. We are moving away from the native library. Eventually we will just
> > use project Panama to wrap OpenSSL. Until then, we are removing
> > everything that isn't required to support the use of OPenSSl with NIO and
> > NIO2.
> >
> > The primary reason for this is stability.
> >
> > > 5) Instead of FileChannel.map maybe a normal ByteBuffer with
> > > FileChannel.read(buffer) can be used (?)
> > That is worth considering. The other sendfile implementations don't use a
> > memory mapped file.
> >
> > I'll start a discussion on the dev list.
> >
> > > One remaining question:
> > > I didn't find FileChannel.map in the other connectors. Is
> > Http2AsyncUpgradeHandler the only occurrence?
> >
> > In the main code base, yes. There is another usage in the test code but that is
> > less of a concern.
> >
> > Mark
> >
> Just two thoughts / remarks:
> 3) New java versions provide java.lang.ref.Cleaner. In older java versions a \
> similar class was sun.misc.Cleaner (though the usage looks a bit strange)
> 5) The memory mapped file approach is quite memory efficient as the file gets \
> virtually mapped into the address space of the java process without loading all the \
> data into (real) memory. As far as I understood, only the used parts of the file \
> gets loaded by Windows OS. When reading the file into a Buffer, the file is read \
> completely. This shouldn't make a big difference with files around some KB or MB. \
> However, for large files with several GB they might be handled well via memory \
> mapped files but not with a ByteBuffer. I am not sure, if sendfile is popular and \
> used for this use-cases. So maybe a configuration toggle between "memory-optimized" \
> and "lock-optimized" might be useful.
> Thanks!
> Thomas
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
I've not understood the problem well, but we did face file handle leaks
problem few years ago for a long running application that used to open
large number of files. It was not a tomcat application.
What we discovered was even though we were closing the streams created
by FileOutputStream, JVM was not releasing the handles. Code was using
opening stream using try(InputStream is = new FileInputStream(path));
We used to run out of file handles regularly. Even explicit close call
did not help.
I don't have links now, but research at that time indicated, it had to
do with timing and when some thread of JVM gets a chance to do cleanup.
For a busy application thread very rarely got executed and file handles
keep piling up.
We replaced all the streams with Files.newInputStream or
Files.newOutputStream and never faced the issue. Is it worth replacing
input stream on line 869 to use Path and equivalent? Those streams get
closed more reliably - at least in my experience.
Regards,
Niranjan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic