[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openjdk-serviceability-dev
Subject:    Re: jcmd - executing multiple commands at the same safepoint?
From:       Frederic Parain <frederic.parain () oracle ! com>
Date:       2018-05-29 19:54:59
Message-ID: 3D31B965-68BE-4CBD-8952-907EBDA71BCE () oracle ! com
[Download RAW message or body]

Hi Thomas,


> On May 26, 2018, at 04:07, Thomas Stüfe <thomas.stuefe@gmail.com> wrote:
> 
<snip>
>> 
>> I think that it is a fundamental piece of the design. Adding a new diagnostic
>> command to group several commands under a unique safepoint:
>>  1 - requires explicit action from the user (and acknowledgment of the risks)
>>  2 - doesn't touch the current syntax/semantic at all
>> 
>> Another reason I'm pushing for a specific diagnostic command rather than
>> changing the syntax is remote execution. jcmd is not the only way to invoke
>> a diagnostic command, they can also be invoke remotely using JMX and the
>> com.sun.management.DiagnosticCommand dynamic MBean. Through the
>> platform MBean, users do not have access to the jcmd syntax, they see
>> each diagnostic command as a MBean operation (providing that they have
>> been registered with the DCmd_Source_MBean flag). The initial support
>> for remote execution was taking a single String and processed it just like
>> a string passed by jcmd, but people developing consoles for remote
>> management complained that this was not convenient for them, and asked
>> for something they could use more programmatically.
>> If the safepoint bundling is implemented in the syntax, JMX clients won't
>> be able to use the feature. If we implement it with a new diagnostic command,
>> it will be visible from the MBean and they will be able to use it.
> 
> That is interesting. Do you have examples for such programs? I was
> looking for ways of how to use jcmd (or equivalent) remotely.

If you're looking for a console application to invoke diagnostic command
on a remote JVM, Mission Control has a dedicated support for that in the
"Diagnostic Commands" tab.
If you're looking for exemples of source code doing remote invocations,
there're several unit tests for the DiagnosticCommand MBean in the directory
test/jdk/com/sun/management/DiagnosticCommandMBean.

> 
>> 
>> At last, creating a specific diagnostic command for safepoint bundling would
>> show the way on how to group commands for consistency. Safepoints is not
>> the only way to get consistency. For some JVM internals reasons, you might
>> want to group commands under a single lock, like the Threads_lock. Adding
>> a diagnostic command for each type of bundling seems more extensible.
>> 
> 
> You make good points.
> 
> Adding a new command has also the advantage that it does not affect
> downward compatibility and hence would not require a CSR. CSR seems to
> slow down things a bit (I have a very simple CSR for jcmd open
> undecided since 12 days:
> https://bugs.openjdk.java.net/browse/JDK-8203043). So, while I think
> CSR are a necessary process, I am happy to avoid them if possible.
> 
> Lets spin this out a bit more:
> 
> -> However the user specifies "please execute these commands at a
> single safepoint", if one of the enclosed commands cannot be run
> inside a safepoint the whole bundle should fail, agreed? So,
> specifying e.g. JFR.start as part of such a command bundle would cause
> the whole bundle to not being executed.

Yes. The new command N would ask for the execution of a list of
commands { A, B, C, … }, and would fail if:
  - any of the specified commands { A, B, C, … } is not a registered
    command
  - any of the specified commands { A, B, C, … } doesn't support
    execution at safepoint
  - any of the specified commands { A, B, C, … } is not exported to
    the DCmdSource used to invoke N (*)

(*) Not all commands are exported to all interfaces (jcmd,JMX), for
instance, some commands related to the JMX agent are cannot
be invoked through the JMX. Even if the new command can be
invoked from all interfaces, it must not allow to circumvent this
restriction. This should not be an issue today, because all
JMX agent commands are calling back Java code, and are not
suitable for execution at a safepoint. But this would ensure the
consistency of the model in the long term (and the test is cheap).

> 
> -> If we introduce a command like you propose (lets call it "VM.run"
> for the sake of the argument) we still need to decide how to delimit
> the commands at the command line. That brings us back to the syntax
> question: separation by a explicit delimiter, if possible one that
> does not clash with common unix shells?
> 
> Lets try colon ':' :
> 
> jcmd VM.run VM.uptime : VM.metaspace show-loaders show-classes
> by-chunktype : VM.classloaders show-classes : VM.info
> 
> Of course, we can wrap all arguments to jcmd in quotes, then the
> delimiter sign does not affect things as much, unless we use quotes :)
> We could e.g. use semicolon:
> 
> jcmd ' VM.run VM.uptime ; VM.metaspace show-loaders show-classes
> by-chunktype ; VM.classloaders show-classes ; VM.info ‘
> 

I thought about colon and semi-colon too, because they are the most
obvious delimiters. However, they are already as delimiters in classpaths
(semi-colon on Unix, colon on Windows), so parsing to find the right
delimiters would be tricky.

I'd start with ‘!' or ‘+' in a first time, to make some progress on the other
aspects of the command, and buy some time to discuss the delimiter
issue in parallel.

The last resort would be to force each command in the bundle to
be specified with quotes (arguments included). But it would make the
syntax very heavy, I'd prefer a simpler delimiter.


> Or, we go back to my original idea of letting the command keyword be
> the indicator for a new command start. Honestly, at this point I have
> no preference. Both approaches seem reasonable.
> 
> The only niggle I have with an explicit command is the syntax, which I
> find a bit awkward. My thinking before reading your response was that
> I'd prefer a jcmd flag to specify the same behavior: that all commands
> should be executed at a single safepoint. E.g. [-s/--stacked]
> 
> jcmd -s ' VM.uptime ; VM.metaspace show-loaders show-classes
> by-chunktype ; VM.classloaders show-classes ; VM.info '
> 
> But I see the disadvantage: depending on how it is implemented this
> would require consistent changes in both jcmd and hotspot, and not
> work with other clients using MBeans. Whereas the hypothetical
> "VM.run" would work out of the box for any old jcmd version and any
> other client.
> 
> Of course one could keep the "VM.run" idea and still give jcmd a
> "-s/--stacked" option, as a shallow convenience feature, which just
> translates to "VM.run" and runs that.
> 
> -> A variation of your "have special commands to run a command bundle
> in one particular context" theme could be having only one command,
> e.g. "VM.run", specifying the conditions under which to run the bundle
> as sub options. E.g. "VM.run -s <commands>"  for "run commands at
> safepoint", "VM.run -t <commands>" for "run commands under
> Threads_lock". That would even allow to combine these contexts. I do
> not know if that makes sense.
> 
> - Another advantage of having an explicit command just occurring to me
> is that it enables more evolved command files, where one can
> intersperse command bundles and single commands. For instance, I could
> give this to a customer:
> 
> <command file>
> 1  VM.uptime
> 2  VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
> VM.classloaders details VM.classloader_stats
> 3  GC.run
> 4  VM.run_commands VM.metaspace show-loaders show-classes by-chunktype
> VM.classloaders details VM.classloader_stats
> 
> to execute a command bundle before and after triggering a GC.

This would  work fine . The parser of the jcmd string will split the
string based on ‘\n' characters, then each line of command will be executed
sequentially, and lines containing the VM.run_command will then create the
safepoint and trigger the execution of the different commands specified in 
its argument list.

> 
>> We have a three days week-end in the US, but I should be able to code
>> something next week.
>> 
> 
> Oh, you want to take over the implementation, sure! Fine by me.

I'm not promising an implementation, I haven't worked on this code
since I left the serviceability team. I need to code at least a skeleton
of the command to refresh my memory about all aspects of the
diagnostic command framework. 

> 
>> Thank you for your ideas and your interest in diagnostic commands.
> 
> Sure. Why I am interested in this: We strongly rely on tools similar
> to jcmd for our support. We have quite capable tools in our (licensed,
> non-open) VM port, which are similar to jcmd in many aspects. So we
> try to bring some of the capabilities upstream, in order to ease the
> merge costs for us and to help the OpenJDK community.

The diagnostic command framework was designed to support an
extensible set of commands. Feel free to extend, and thank you
for contributing to the OpenJDK community.

Fred

>> 
>> 
>>> 
>>>> 
>>>> A new separator, more convenient than newline, would be require to have
>>>> a single line syntax.
>>>> 
>>>> My 2 cents,
>>>> 
>>> 
>>> Certainly worth more than 2 cents :) Thanks for your thoughts!
>>> 
>>> ..Thomas
>>> 
>>>> Fred
>>>> 
>>>> 
>>>>> 
>>>>>> 
>>>>>> Erik
>>>>>> 
>>>>>> Hi Thomas,
>>>>>> 
>>>>>> very positive suggestion.
>>>>>> 
>>>>>> One observation:
>>>>>> There already seems to be some different interpretation of the command
>>>>>> line
>>>>>> in different Java versions:
>>>>>> For instance when I try to dump a flight recording in 1.8.0_152-ea-b05 :
>>>>>> I
>>>>>> can use:
>>>>>> jcmd 33014 JFR.dump filename="recording1.jfr" recording=1
>>>>>> but in build 9+181 , the same command must be:
>>>>>> jcmd 33014 JFR.dump filename="recording1.jfr",recording=1
>>>>>> (the comma to separate sub-options you mentioned earlier)
>>>>>> 
>>>>>> Therefore the suggestion without curly brackets, giving several commands
>>>>>> after one another seems good with regard to backwards compatibility.
>>>>>> 
>>>>>> Motivation: hawt.io uses the MBean
>>>>>> com.sun.management:type=DiagnosticCommand
>>>>>> to access the same functionality as jcmd. Variations in option syntax
>>>>>> across
>>>>>> Java versions is already an issue (only affecting sub-options though, as
>>>>>> each command is a separate JMX operation). So syntax compatibility is
>>>>>> highly
>>>>>> appreciated :-)
>>>>>> 
>>>>>> Martin
>>>>>> 
>>>>>> lør. 12. mai 2018 kl. 20:11 skrev Kirk Pepperdine
>>>>>> <kirk.pepperdine@gmail.com>:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>>> On May 10, 2018, at 11:26 AM, Thomas Stüfe <thomas.stuefe@gmail.com>
>>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On Thu, May 10, 2018 at 9:13 AM, Kirk Pepperdine
>>>>>>>> <kirk.pepperdine@gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> The stacking at the safe point would be a huge win. Right now thread
>>>>>>>>> dump consistency can really confuse people as the tooling will show
>>>>>>>>> two
>>>>>>>>> threads owning the same lock at seemingly the same time. Reality, it's
>>>>>>>>> just
>>>>>>>>> that people didn't realize you get a safe point for each thread
>>>>>>>>> therefor
>>>>>>>>> there is motion in the system as you're collecting the data.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> I am a bit confused. What tooling are you talking about?
>>>>>>> 
>>>>>>> 
>>>>>>> jstack. I've not seen it with jcmd. But I often see 2 threads holding
>>>>>>> the
>>>>>>> same lock at the "same time" which is of course non-sense. I can dig one
>>>>>>> up
>>>>>>> for you if you're still confused.
>>>>>>> 
>>>>>>> 
>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> As an aside, it's amazing how many dev's aren't aware of jcmd. Just
>>>>>>>>> yesterday after my session at Devoxx I had someone ask me about using
>>>>>>>>> jfr
>>>>>>>>> from the command line, many in that session had not seen jcmd before.
>>>>>>>>> The
>>>>>>>>> feedback was, wow, this is very useful and I wished I had of known
>>>>>>>>> about it
>>>>>>>>> earlier.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> Yes, jcmd is quite useful. I also really like the simple design, which
>>>>>>>> is by default up- and downward compatible (so you can talk to any VM,
>>>>>>>> new, old, it should not matter). That is just good design. We - Sap -
>>>>>>>> work to extend jcmd, to help our support folks. E.g. the whole new
>>>>>>>> VM.metaspace command chain.
>>>>>>> 
>>>>>>> 
>>>>>>> And simple it is….well done!!!
>>>>>>> 
>>>>>>> — Kirk
>>>>>>> 
>>>>>> 
>>>> 
>> 

[prev in list] [next in list] [prev in thread] [next in thread]