[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: OperatingSystemMXBean unaware of container memory limits
From: Kirk Pepperdine <kirk.pepperdine () gmail ! com>
Date: 2019-06-23 5:54:48
Message-ID: 8A24A29A-4D4E-4283-BE48-34F6CC7DED90 () gmail ! com
[Download RAW message or body]
Hi Mario,
I completely agree, the OS Bean should be consistent with the memory. That said I \
favor introducing new MXBeans to achieve the desired effects. If there is no \
container than the container MXBean could simply not be instantiated.
I'm looking for more visibility especially for the purposes of supporting diagnostic \
tooling. If visibility is shut down it's just makes it that much harder to perform \
any reasonable level of high level diagnostics.
Kind regards,
Kirk
> On Jun 21, 2019, at 5:27 AM, Mario Torre <neugens.limasoftware@gmail.com> wrote:
>
> Hi Kirk,
>
> I think I understand what you mean, however then the OS Bean should be consistent \
> regarding CPU information as well.
> I think I remember why this was fixed the way it is now was because of incorrect \
> behavior during GC configuration, or something along those lines, so I guess we \
> would need two APIs after all to give tooling a way to sneak into the actual \
> hardware properties.
> I would guess, however, that from the point of view of a containerised VM its OS is \
> the container not the bare metal (or virtualized metal perhaps), so tooling would \
> need to check specifically for a separate bean.
> That could be indicated via a property in the OS Bean (if it's not the case \
> already).
> Nevertheless, I think consistency in the OS Bean should be achieved.
>
> Cheers,
> Mario
>
> On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine <kirk.pepperdine@gmail.com \
> <mailto:kirk.pepperdine@gmail.com>> wrote: Hi Mario,
>
> I don't believe the MBean returns the wrong information. Is it not that the calls \
> by-pass the container? Would it not be more appropriate to add a container aware \
> mean? From a tooling perspective it's a mistake to completely seal the JVM away \
> from the hardware as it makes certain diagnostics more difficult to perform.
> Kind regards,
> Kirk
>
>
> > On Jun 21, 2019, at 6:06 AM, Mario Torre <neugens.limasoftware@gmail.com \
> > <mailto:neugens.limasoftware@gmail.com>> wrote:
> > The way I understood the bug report is a two fold approach, i.e. fix the os bean \
> > and *possibly* add a container one or extend it to add more data.
> > I agree with you and Andrew that the current OS Bean returns wrong information, \
> > this should be fixed in any case I think.
> > Bob, do you have some design ideas to share regarding the new interface? I think \
> > this may be an interesting project to pick up, even for students wanting to write \
> > a thesis, as it seems time is a limiting factor here for you to work on that?
> > Cheers,
> > Mario
> >
> > On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <sgehwolf@redhat.com \
> > <mailto:sgehwolf@redhat.com>> wrote: Hi Bob,
> >
> > On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> > > Hi Andrew,
> > >
> > > I am aware of the limitations of the OperatingSystemMXBean and was
> > > hoping to address these limitations during the implementation of
> > > https://bugs.openjdk.java.net/browse/JDK-8199944 \
> > > <https://bugs.openjdk.java.net/browse/JDK-8199944>.
> > > It would be helpful if you feel this is important to add some votes
> > > to this issue.
> >
> > It seems strange that the getAvailableProcessors() returns the
> > container limit, while the memory limits are for the physical host. If
> > anything, shouldn't they agree (both physical host or both container
> > limits)?
> >
> > When I briefly looked into it initially it seems to be a side-effect of
> > what is being used by the JDK code implementation-wise. IIRC
> > getAvailableProcessors() uses a runtime call. Memory reporting has it's
> > own logic[1], thus not reporting the container limit.
> >
> > This seems weird from a consistency perspective. Thoughts?
> >
> > If I understand you correctly, you are arguing that container reporting
> > should go into its own MX bean. On the other hand, CPU reporting for
> > the OS MX bean is container aware already. That makes me believe we
> > should rather make this consistent before evaluating a new MX bean.
> >
> > Thanks,
> > Severin
> >
> > [1] http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365 \
> > <http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365>
> >
> >
> > > Bob.
> > >
> > >
> > > > On Jun 20, 2019, at 9:43 AM, Andrew Azores <aazores@redhat.com \
> > > > <mailto:aazores@redhat.com>> wrote:
> > > >
> > > > Hi all,
> > > >
> > > > Apologies if this is not the most appropriate list, in which case
> > > > please direct me where to go.
> > > >
> > > > I've noticed a surprising result from the
> > > > com.sun.management.OperatingSystemMXBean implementation when
> > > > running in
> > > > a containerized (specifically, using Docker on Linux) environment.
> > > > The
> > > > bean appears to be container-aware for processors, in that running
> > > > with
> > > > Docker option `--cpus 1.0` for example, on a multicore system, will
> > > > cause both java.lang.Runtime#availableProcessors and
> > > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > > > return 1. However, the Docker option `--memory 100M` (or any other
> > > > limit value) is not reflected in the value returned by
> > > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > > > ,
> > > > and instead the returned value is still the total physical memory
> > > > of
> > > > the host machine - of which only a small portion may actually be
> > > > available to the "Operating System" of the JVM. Similarly for the
> > > > methods regarding free physical memory, total swap, and free swap.
> > > >
> > > > I have attached a patch which adds a small reproducer to the
> > > > existing
> > > > MemoryAwareness test.
> > > >
> > > > This seems like a bug to me, since if the imposed container limit
> > > > on
> > > > processors as a resource is included as part of the "Operating
> > > > System"
> > > > resource reporting, then surely memory resources should be reported
> > > > the
> > > > same way. As I said, I found the current behaviour quite
> > > > surprising.
> > > >
> > > > --
> > > > Andrew Azores
> > > > Software Engineer, OpenJDK Team
> > > > Red Hat
> > > > <jdk-osmxbean-container-memory-test-01.patch>
> > >
> > >
> >
> > --
> > pgp key: http://subkeys.pgp.net/ <http://subkeys.pgp.net/> PGP Key ID: 80F240CF
> > Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF
> >
> > Java Champion - Blog: http://neugens.wordpress.com \
> > <http://neugens.wordpress.com/> - Twitter: @neugens Proud GNU Classpath \
> > developer: http://www.classpath.org/ <http://www.classpath.org/>
> > OpenJDK: http://openjdk.java.net/projects/caciocavallo/ \
> > <http://openjdk.java.net/projects/caciocavallo/>
> > Please, support open standards:
> > http://endsoftpatents.org/ <http://endsoftpatents.org/>
>
> --
> pgp key: http://subkeys.pgp.net/ <http://subkeys.pgp.net/> PGP Key ID: 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF
>
> Java Champion - Blog: http://neugens.wordpress.com <http://neugens.wordpress.com/> \
> - Twitter: @neugens Proud GNU Classpath developer: http://www.classpath.org/ \
> <http://www.classpath.org/>
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/ \
> <http://openjdk.java.net/projects/caciocavallo/>
> Please, support open standards:
> http://endsoftpatents.org/ <http://endsoftpatents.org/>
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class="">Hi Mario,<div class=""><br \
class=""></div><div class="">I completely agree, the OS Bean should be consistent \
with the memory. That said I favor introducing new MXBeans to achieve the desired \
effects. If there is no container than the container MXBean could simply not be \
instantiated.</div><div class=""><br class=""></div><div class="">I'm looking for \
more visibility especially for the purposes of supporting diagnostic tooling. If \
visibility is shut down it's just makes it that much harder to perform any reasonable \
level of high level diagnostics.</div><div class=""><br class=""></div><div \
class="">Kind regards,</div><div class="">Kirk</div><div class=""><br \
class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Jun 21, \
2019, at 5:27 AM, Mario Torre <<a href="mailto:neugens.limasoftware@gmail.com" \
class="">neugens.limasoftware@gmail.com</a>> wrote:</div><br \
class="Apple-interchange-newline"><div class=""><div class=""><div class=""><div \
dir="auto" class="">Hi Kirk,</div><div dir="auto" class=""><br class=""></div><div \
dir="auto" class="">I think I understand what you mean, however then the OS Bean \
should be consistent regarding CPU information as well.</div></div></div><div \
dir="auto" class=""><br class=""></div><div dir="auto" class="">I think I remember \
why this was fixed the way it is now was because of incorrect behavior during GC \
configuration, or something along those lines, so I guess we would need two APIs \
after all to give tooling a way to sneak into the actual hardware \
properties.</div><div dir="auto" class=""><br class=""></div><div dir="auto" \
class="">I would guess, however, that from the point of view of a containerised VM \
its OS is the container not the bare metal (or virtualized metal perhaps), so tooling \
would need to check specifically for a separate bean.</div><div dir="auto" \
class=""><br class=""></div><div dir="auto" class="">That could be indicated via a \
property in the OS Bean (if it's not the case already).</div><div dir="auto" \
class=""><br class=""></div><div dir="auto" class="">Nevertheless, I think \
consistency in the OS Bean should be achieved.</div><div dir="auto" class=""><br \
class=""></div><div dir="auto" class="">Cheers,</div><div dir="auto" \
class="">Mario </div><div class=""><div class=""><br class=""><div \
class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri 21. Jun 2019 at 13:23, \
Kirk Pepperdine <<a href="mailto:kirk.pepperdine@gmail.com" target="_blank" \
class="">kirk.pepperdine@gmail.com</a>> wrote:<br class=""></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex"><div \
style="word-wrap:break-word;line-break:after-white-space" class="">Hi Mario,<div \
class=""><br class=""></div><div class="">I don't believe the MBean returns the wrong \
information. Is it not that the calls by-pass the container? Would it not be more \
appropriate to add a container aware mean? From a tooling perspective it's a mistake \
to completely seal the JVM away from the hardware as it makes certain diagnostics \
more difficult to perform.</div><div class=""><br class=""></div><div class="">Kind \
regards,</div><div class="">Kirk</div></div><div \
style="word-wrap:break-word;line-break:after-white-space" class=""><div class=""><br \
class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On \
Jun 21, 2019, at 6:06 AM, Mario Torre <<a \
href="mailto:neugens.limasoftware@gmail.com" target="_blank" \
class="">neugens.limasoftware@gmail.com</a>> wrote:</div><br \
class="m_-6207319854354579326m_6668840894600592225Apple-interchange-newline"><div \
class=""><div class=""><div dir="auto" class="">The way I understood the bug report \
is a two fold approach, i.e. fix the os bean and *possibly* add a container one or \
extend it to add more data.</div></div><div dir="auto" class=""><br \
class=""></div><div dir="auto" class="">I agree with you and Andrew that the current \
OS Bean returns wrong information, this should be fixed in any case I \
think.</div><div dir="auto" class=""><br class=""></div><div dir="auto" class="">Bob, \
do you have some design ideas to share regarding the new interface? I think this may \
be an interesting project to pick up, even for students wanting to write a thesis, as \
it seems time is a limiting factor here for you to work on that?</div><div dir="auto" \
class=""><br class=""></div><div dir="auto" class="">Cheers,</div><div dir="auto" \
class="">Mario</div><div class=""><br class=""><div class="gmail_quote"><div \
dir="ltr" class="gmail_attr">On Fri 21. Jun 2019 at 10:53, Severin Gehwolf <<a \
href="mailto:sgehwolf@redhat.com" target="_blank" \
class="">sgehwolf@redhat.com</a>> wrote:<br class=""></div><blockquote \
class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc \
solid;padding-left:1ex">Hi Bob,<br class=""> <br class="">
On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:<br class="">
> Hi Andrew,<br class="">
> <br class="">
> I am aware of the limitations of the OperatingSystemMXBean and was<br class="">
> hoping to address these limitations during the implementation of<br class="">
> <a href="https://bugs.openjdk.java.net/browse/JDK-8199944" rel="noreferrer" \
target="_blank" class="">https://bugs.openjdk.java.net/browse/JDK-8199944</a>.<br \
class=""> > <br class="">
> It would be helpful if you feel this is important to add some votes<br class="">
> to this issue.<br class="">
<br class="">
It seems strange that the getAvailableProcessors() returns the<br class="">
container limit, while the memory limits are for the physical host. If<br class="">
anything, shouldn't they agree (both physical host or both container<br class="">
limits)?<br class="">
<br class="">
When I briefly looked into it initially it seems to be a side-effect of<br class="">
what is being used by the JDK code implementation-wise. IIRC<br class="">
getAvailableProcessors() uses a runtime call. Memory reporting has it's<br class="">
own logic[1], thus not reporting the container limit.<br class="">
<br class="">
This seems weird from a consistency perspective. Thoughts?<br class="">
<br class="">
If I understand you correctly, you are arguing that container reporting<br class="">
should go into its own MX bean. On the other hand, CPU reporting for<br class="">
the OS MX bean is container aware already. That makes me believe we<br class="">
should rather make this consistent before evaluating a new MX bean.<br class="">
<br class="">
Thanks,<br class="">
Severin<br class="">
<br class="">
[1] <a href="http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365" \
rel="noreferrer" target="_blank" \
class="">http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365</a><br \
class=""> <br class="">
<br class="">
> Bob.<br class="">
> <br class="">
> <br class="">
> > On Jun 20, 2019, at 9:43 AM, Andrew Azores <<a \
href="mailto:aazores@redhat.com" target="_blank" \
class="">aazores@redhat.com</a>><br class=""> > > wrote:<br class="">
> > <br class="">
> > Hi all,<br class="">
> > <br class="">
> > Apologies if this is not the most appropriate list, in which case<br \
class=""> > > please direct me where to go.<br class="">
> > <br class="">
> > I've noticed a surprising result from the<br class="">
> > com.sun.management.OperatingSystemMXBean implementation when<br class="">
> > running in<br class="">
> > a containerized (specifically, using Docker on Linux) environment.<br \
class=""> > > The<br class="">
> > bean appears to be container-aware for processors, in that running<br \
class=""> > > with<br class="">
> > Docker option `--cpus 1.0` for example, on a multicore system, will<br \
class=""> > > cause both java.lang.Runtime#availableProcessors and<br class="">
> > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /<br \
class=""> > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors \
to<br class=""> > > return 1. However, the Docker option `--memory 100M` (or \
any other<br class=""> > > limit value) is not reflected in the value returned \
by<br class=""> > > \
com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize<br class=""> > \
> ,<br class=""> > > and instead the returned value is still the total \
physical memory<br class=""> > > of<br class="">
> > the host machine - of which only a small portion may actually be<br \
class=""> > > available to the "Operating System" of the JVM. Similarly for \
the<br class=""> > > methods regarding free physical memory, total swap, and \
free swap.<br class=""> > > <br class="">
> > I have attached a patch which adds a small reproducer to the<br class="">
> > existing<br class="">
> > MemoryAwareness test.<br class="">
> > <br class="">
> > This seems like a bug to me, since if the imposed container limit<br \
class=""> > > on<br class="">
> > processors as a resource is included as part of the "Operating<br class="">
> > System"<br class="">
> > resource reporting, then surely memory resources should be reported<br \
class=""> > > the<br class="">
> > same way. As I said, I found the current behaviour quite<br class="">
> > surprising.<br class="">
> > <br class="">
> > -- <br class="">
> > Andrew Azores<br class="">
> > Software Engineer, OpenJDK Team<br class="">
> > Red Hat<br class="">
> > <jdk-osmxbean-container-memory-test-01.patch><br class="">
> <br class="">
> <br class="">
<br class="">
</blockquote></div></div>-- <br class=""><div dir="ltr" \
class="m_-6207319854354579326m_6668840894600592225gmail_signature" \
data-smartmail="gmail_signature">pgp key: <a href="http://subkeys.pgp.net/" \
target="_blank" class="">http://subkeys.pgp.net/</a> PGP Key ID: 80F240CF<br \
class="">Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF<br \
class=""><br class="">Java Champion - Blog: <a href="http://neugens.wordpress.com/" \
target="_blank" class="">http://neugens.wordpress.com</a> - Twitter: @neugens<br \
class="">Proud GNU Classpath developer: <a href="http://www.classpath.org/" \
target="_blank" class="">http://www.classpath.org/</a><br class="">OpenJDK: <a \
href="http://openjdk.java.net/projects/caciocavallo/" target="_blank" \
class="">http://openjdk.java.net/projects/caciocavallo/</a><br class=""><br \
class="">Please, support open standards:<br class=""><a \
href="http://endsoftpatents.org/" target="_blank" \
class="">http://endsoftpatents.org/</a><br class=""></div> \
</div></blockquote></div><br class=""></div></div></blockquote></div></div> </div>-- \
<br class=""><div dir="ltr" class="gmail_signature" \
data-smartmail="gmail_signature">pgp key: <a href="http://subkeys.pgp.net/" \
target="_blank" class="">http://subkeys.pgp.net/</a> PGP Key ID: 80F240CF<br \
class="">Fingerprint: BA39 9666 94EC 8B73 27FA FC7C 4086 63E3 80F2 40CF<br \
class=""><br class="">Java Champion - Blog: <a href="http://neugens.wordpress.com/" \
target="_blank" class="">http://neugens.wordpress.com</a> - Twitter: @neugens<br \
class="">Proud GNU Classpath developer: <a href="http://www.classpath.org/" \
target="_blank" class="">http://www.classpath.org/</a><br class="">OpenJDK: <a \
href="http://openjdk.java.net/projects/caciocavallo/" target="_blank" \
class="">http://openjdk.java.net/projects/caciocavallo/</a><br class=""><br \
class="">Please, support open standards:<br class=""><a \
href="http://endsoftpatents.org/" target="_blank" \
class="">http://endsoftpatents.org/</a><br class=""></div> \
</div></blockquote></div><br class=""></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic