[prev in list] [next in list] [prev in thread] [next in thread]
List: openjdk-serviceability-dev
Subject: Re: JEP draft: Disallow the Dynamic Loading of Agents by Default
From: Kirk Pepperdine <kirk.pepperdine () gmail ! com>
Date: 2023-04-29 18:41:15
Message-ID: BE145154-0EF0-4A98-8053-E493681F4214 () gmail ! com
[Download RAW message or body]
Hi Dan,
I was initially concerned with this proposal and that it was being introduced into an \
LTS has only added to that. You are quite correct that it's not what we wish to be \
true that should be driving us. So yes, LTS's are no different than other releases \
except that isn't what our users are saying as they jump from LTS to LTS.
While I fear my being concerned by a pending change falls from a tendency towards \
"resistance to change", I also know that changes in the tooling chain historically \
have oversized effects on an organizations ability to migrate to the latest and \
greatest. I'd feel a lot better about things if only I had the data to help me temper \
my level of concern. To that end I've been speaking to different projects to \
understand the impact this change might have on them. The feedback from our biggest \
project that will be directly impacted by this change have (loosely) said, dynamic \
attach offers us a simple model to deliver monitoring but we could switch to direct \
attach. The problem here is that this project is a bolt on to many other applications \
and while this project could change, it's not clear (for many of the reasons you've \
pointed out below) that it's customers could. Complicating this is that I believe \
that only a handful of devs/devops/ops people know this in the works.
Finally, a precedent for delaying the taking away capabilities from users was \
established with the release of modularity in Java 9. As you may recall, Java 9 broke \
so many things that deploying the full on application of modularity was helpful. I \
would think that a similar approach here where the full on change was applied after \
the LTS release would also help adaptation.
Kind regards,
Kirk
PS, this email is targeted for serviceability but has been crossposted so that others \
not following that list will be aware of pending changes.
> On Apr 28, 2023, at 7:30 PM, Dan Heidinga <heidinga@redhat.com> wrote:
>
> Hi Ron,
>
> Thanks for writing up the JEP draft outlining the proposal to disallow dynamic \
> loading of agents by default. The Red Hat Java team has continued to discuss this \
> proposal internally and with our stakeholders.
> While there is a general agreement (or at least acceptance) with the overall \
> direction of this proposal, we still see the concerns raised by Andrew [0] as \
> needing to be addressed.
> So let's start with the high-order bit: timing.
>
> The JEP draft states it "intends to adopt the proposal [to disable agents by \
> default] six years later, in 2023, for JDK 21." We would like to see this targeted \
> to JDK 22 instead as the change has not been widely evangelized yet and comes as a \
> surprise to many, both to those actively developing OpenJDK and to those monitoring \
> its development.
> We owe it to the community to give this proposal enough bake time, especially given \
> that JDK 21 is an LTS release. Though the official position is that LTS releases \
> are no different than any other release, the actions of JDK consumers make them \
> special. Users will be surprised by this change when they migrate from JDK 17 to \
> 21. If we delay till JDK 22, we give the ecosystem time to adapt to the change \
> before the next LTS.
> Additionally, users who have tested with the -XX:-EnableDynamicAgentLoading option \
> will have false expectations if they validated their use of jcmd to load the agent \
> as the behaviour was not correct prior to JDK 21 [1].
> The next concern is one you call out in the draft that "Java's excellent \
> serviceability has long been a source of pride for the platform." We agree!
> Java currently has an excellent, prime position in Observability capabilities. For \
> better or for worse, there are many Observability tools that have relied on dynamic \
> attach to provide the necessary monitoring for workloads
> It's important we give Java's monitoring tools sufficient time to migrate \
> comfortably without shocking the ecosystem by changing the default in an LTS. By \
> delaying the change till JDK 22, we give the ecosystem 2 years to migrate and to \
> prepare users for the change.
> Additionally, this provides the time for Java's profiling tools to adapt as well. \
> And for the "ad-hoc troubleshooting" tools - like Byteman - to retrain their users. \
> Finally, while it's easy to agree with the principle that "the application owner \
> is given the final say on whether to allow dynamic loading of agents", the argument \
> can (and should!) be made that those application owners have made that final \
> decision by deploying the libraries and tools that use dynamic attach. A JVM \
> command line argument is not the only way to express their consent for code to run \
> on their system.
> For many production uses, the reality is more complicated than a single \
> "application owner". Take a Kubernetes cluster for example.
> Who is the application owner when deploying to a Kubernetes cluster? The dev team \
> that develops the application? The operations team that manages the cluster? The \
> monitoring team that monitors the applications? The Support team that troubleshoots \
> issues with the deployment?
> We should be careful not to understate or misrepresent the complex web of "owners" \
> that are currently able (and, for business reasons, need) to apply agents \
> dynamically. Downplaying the complexity many organizations experience when dealing \
> with changes to command line options (as an example) weakens the argument for \
> changing today's status quo.
> We also know that in many cases customers and users may not be in a position to \
> modify startup scripts (e.g. even to add in an extra parameter) as to do so may \
> invalidate support contracts, etc.
> Dynamically attached agents have been a "superpower" of Java and their use has \
> greatly benefited the ecosystem as they've helped service and support use cases \
> that otherwise aren't possible, as they helped propel Java to the forefront of the \
> Observability tooling, and allowed many other useful libraries to be developed.
> Let's delay flipping the default until JDK 22 to give the breadth of the ecosystem \
> time to absorb this change.
> –Dan
>
>
> [0] https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html \
> <https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html> [1] \
> https://bugs.openjdk.org/browse/JDK-8304438 \
> <https://bugs.openjdk.org/browse/JDK-8304438>
[Attachment #3 (unknown)]
<html><head><meta http-equiv="Content-Type" content="text/html; \
charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; \
line-break: after-white-space;" class="">Hi Dan,<div class=""><br class=""></div><div \
class="">I was initially concerned with this proposal and that it was being \
introduced into an LTS has only added to that. You are quite correct that it's not \
what we wish to be true that should be driving us. So yes, LTS's are no different \
than other releases except that isn't what our users are saying as they jump from LTS \
to LTS.</div><div class=""><br class=""></div><div class="">While I fear my being \
concerned by a pending change falls from a tendency towards "resistance to change", I \
also know that<font color="#000000" class=""> changes in the tooling chain \
historically have oversized effects on an organizations ability to migrate to the \
latest and greatest. I'd feel a lot better about things if only I had the data to \
help me temper my level of concern. To that end I've been speaking to \
different projects to understand the impact this change might have on \
them. The feedback from our biggest project that will </font>be \
directly impacted by this change have (loosely) said, dynamic attach offers us a \
simple model to deliver monitoring but we could switch to direct attach. The problem \
here is that this project is a bolt on to many other applications and while this \
project could change, it's not clear (for many of the reasons you've pointed out \
below) that it's customers could. Complicating this is that I believe that only a \
handful of devs/devops/ops people know this in the works.</div><div class=""><br \
class=""></div><div class="">Finally, a precedent for delaying the taking away \
capabilities from users was established with the release of modularity in Java 9. As \
you may recall, Java 9 broke so many things that deploying the full on application of \
modularity was helpful. I would think that a similar approach here where the full on \
change was applied after the LTS release would also help adaptation.</div><div \
class=""><br class=""></div><div class="">Kind regards,</div><div \
class="">Kirk</div><div class=""><br class=""></div><div class="">PS, this email is \
targeted for serviceability but has been crossposted so that others not following \
that list will be aware of pending changes.</div><div class=""><br \
class=""></div><div class=""><div><br class=""><blockquote type="cite" class=""><div \
class="">On Apr 28, 2023, at 7:30 PM, Dan Heidinga <<a \
href="mailto:heidinga@redhat.com" class="">heidinga@redhat.com</a>> \
wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" \
class="">Hi Ron,<br class=""><br class="">Thanks for writing up the JEP draft \
outlining the proposal to disallow dynamic loading of agents by default. The \
Red Hat Java team has continued to discuss this proposal internally and with our \
stakeholders.<br class=""><br class="">While there is a general agreement (or at \
least acceptance) with the overall direction of this proposal, we still see the \
concerns raised by Andrew [0] as needing to be addressed.<br class=""><br class="">So \
let's start with the high-order bit: timing.<br class=""><br class="">The JEP draft \
states it "intends to adopt the proposal [to disable agents by default] six years \
later, in 2023, for JDK 21." We would like to see this targeted to JDK 22 \
instead as the change has not been widely evangelized yet and comes as a surprise to \
many, both to those actively developing OpenJDK and to those monitoring its \
development.<br class=""><br class="">We owe it to the community to give this \
proposal enough bake time, especially given that JDK 21 is an LTS release. \
Though the official position is that LTS releases are no different than any other \
release, the actions of JDK consumers make them special. Users will be \
surprised by this change when they migrate from JDK 17 to 21. If we delay till \
JDK 22, we give the ecosystem time to adapt to the change before the next LTS.<br \
class=""><br class="">Additionally, users who have tested with the \
-XX:-EnableDynamicAgentLoading option will have false expectations if they validated \
their use of jcmd to load the agent as the behaviour was not correct prior to JDK 21 \
[1].<br class=""><br class="">The next concern is one you call out in the draft that \
"Java's excellent serviceability has long been a source of pride for the platform." \
We agree! <br class=""><br class="">Java currently has an excellent, \
prime position in Observability capabilities. For better or for worse, there are many \
Observability tools that have relied on dynamic attach to provide the necessary \
monitoring for workloads<br class=""><br class="">It's important we give Java's \
monitoring tools sufficient time to migrate comfortably without shocking the \
ecosystem by changing the default in an LTS. By delaying the change till JDK \
22, we give the ecosystem 2 years to migrate and to prepare users for the change.<br \
class=""><br class="">Additionally, this provides the time for Java's profiling tools \
to adapt as well. And for the "ad-hoc troubleshooting" tools - like Byteman - \
to retrain their users.<br class=""><br class="">Finally, while it's easy to agree \
with the principle that "the application owner is given the final say on whether to \
allow dynamic loading of agents", the argument can (and should!) be made that those \
application owners have made that final decision by deploying the libraries and tools \
that use dynamic attach. A JVM command line argument is not the only way to \
express their consent for code to run on their system.<br class=""><br class="">For \
many production uses, the reality is more complicated than a single "application \
owner". Take a Kubernetes cluster for example. <br class=""><br class="">Who is \
the application owner when deploying to a Kubernetes cluster? The dev team that \
develops the application? The operations team that manages the cluster? \
The monitoring team that monitors the applications? The Support team that \
troubleshoots issues with the deployment? <br class=""><br class="">We should \
be careful not to understate or misrepresent the complex web of "owners" that are \
currently able (and, for business reasons, need) to apply agents dynamically. \
Downplaying the complexity many organizations experience when dealing with changes to \
command line options (as an example) weakens the argument for changing today's status \
quo.<br class=""><br class="">We also know that in many cases customers and users may \
not be in a position to modify startup scripts (e.g. even to add in an extra \
parameter) as to do so may invalidate support contracts, etc.<br class=""><br \
class="">Dynamically attached agents have been a "superpower" of Java and their use \
has greatly benefited the ecosystem as they've helped service and support use cases \
that otherwise aren't possible, as they helped propel Java to the forefront of the \
Observability tooling, and allowed many other useful libraries to be developed.<br \
class=""><br class="">Let's delay flipping the default until JDK 22 to give the \
breadth of the ecosystem time to absorb this change.<br class=""><br \
class="">–Dan<br class=""> <br class=""><br class="">[0] <a \
href="https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html" \
class="">https://mail.openjdk.org/pipermail/serviceability-dev/2023-March/047084.html</a><br \
class="">[1] <a href="https://bugs.openjdk.org/browse/JDK-8304438" \
class="">https://bugs.openjdk.org/browse/JDK-8304438</a><br class=""></div> \
</div></blockquote></div><br class=""></div></body></html>
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic