[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-dev
Subject: Re: [DISCUSS] CEP-3: Guardrails
From: Andrés_de_la_Peña <adelapena () apache ! org>
Date: 2021-11-10 20:34:01
Message-ID: CANSRBLBbiGsv8MzMnpQrCFQcXsf2NT+mJEmJPVxKaUXQiO0JEg () mail ! gmail ! com
[Download RAW message or body]
>
> I would love to see added something like "all guard rails must work in a
> rolling fashion where some nodes do not have the latest guard rails", but
> also cool leaving this to JIRA.
Good point, I have added a line about the rolling behaviour.
The cassandra.yaml file is pretty big already and configuration of
> guardrails seems like a good candidate to modularize here. Could also be
> done during implementation (or discussed then), so don't hold up voting on
> me. :)
The configuration is supposed to go under a specific section in
cassandra.yaml. There was a colon character missed in the example, I have
fixed it to reflect that all guardrail parameters are grouped in a single
section. Nevertheless, we can always further discuss the details on how we
organize the config during implementation.
On Wed, 10 Nov 2021 at 20:16, Joshua McKenzie <jmckenzie@apache.org> wrote:
> Kind of a nit, but I advocate for us having a guardrails.yaml and updating
> this line:
> >
> > *cassandra.yaml:* allows configuring individual guardrails at startup,
> > being globally disabled by default. These guardrails will also be
> > dynamically configurable through JMX and/or virtual tables.
>
> The cassandra.yaml file is pretty big already and configuration of
> guardrails seems like a good candidate to modularize here. Could also be
> done during implementation (or discussed then), so don't hold up voting on
> me. :)
>
> On Wed, Nov 10, 2021 at 3:06 PM David Capwell <dcapwell@apple.com.invalid>
> wrote:
>
> > I am cool with voting. I would love to see added something like "all
> > guard rails must work in a rolling fashion where some nodes do not have the
> > latest guard rails", but also cool leaving this to JIRA.
> >
> > > On Nov 10, 2021, at 11:17 AM, Ekaterina Dimitrova <
> > e.dimitrova@gmail.com> wrote:
> > >
> > > I am personally ready to give you my vote :-)
> > >
> > > On Wed, 10 Nov 2021 at 13:09, Andrés de la Peña <adelapena@apache.org>
> > > wrote:
> > >
> > > > I think the updated CEP incorporates the feedback above, unless I'm
> > missing
> > > > something. Are we ready to start a vote?
> > > >
> > > > On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña <adelapena@apache.org>
> > > > wrote:
> > > >
> > > > > Under "Migrating existing cassandra.yaml warn/fail thresholds", I
> > > > recently
> > > > > > added a few things which are basically guardrails, so should be
> > > > included in
> > > > > > this set; they are configured by track_warnings
> > (coordinator_read_size,
> > > > > > local_read_size, and row_index_size). With track_warnings I setup
> > the
> > > > > > plumbing to have read queries trigger warnings (or abort the query)
> > to
> > > > the
> > > > > > client exists (under "Event logging" you mention "and also to the
> > client
> > > > > > connection when applicable") and isn't limited to the coordinator
> > > > > > participating in the query (previous limitation for tombstone
> > warnings).
> > > > > > One thing I found which was problematic for track_warnings was that
> > > > > > altering clients is annoying as java and python both ignore the error
> > > > > > message we send (see
> > > > > >
> > > >
> > https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131
> >
> > > > ).
> > > > > > We log client warnings (if enabled) but ignore any detailed error
> > > > message
> > > > > > received from the server; it would be good to talk about client
> > > > > > integrations and how users are informed of issues in more detail.
> > > > >
> > > > >
> > > > > I have updated the CEP to include those thresholds among the ones that
> > > > > could be migrated once we have the guardrails framework ready. I have
> > > > also
> > > > > mentioned the usage of internal messaging to be able to propagate the
> > > > > outcome of guardrails triggered on nodes that are not the coordinator,
> > > > and
> > > > > the need of making changes on drivers.
> > > > >
> > > > > What I meant by "and also to the client connection when applicable" is
> > > > > that some guardrails can be applied to things that are nor necessarily
> > > > > associated to a client connection, such as compaction. I have tried
> > to be
> > > > > more explicit about that.
> > > > >
> > > > >
> > > > > On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña <
> > a.penya.garcia@gmail.com
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Being able to configure guardrails dynamically makes a lot of sense
> > to
> > > > > > me, I have updated the CEP to mention that. I think we don't need to
> > > > decide
> > > > > > yet whether it would be done through JMX and/or virtual tables.
> > > > > >
> > > > > > On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas <scott@paradoxica.net>
> > > > > > wrote:
> > > > > >
> > > > > > > Re: "I think you all know my feels on JMX." –
> > > > > > >
> > > > > > > Super fair - I'd meant to speak in terms of desired outcome ("the
> > > > > > > feature should be dynamically configurable at runtime") rather than
> > > > > > > implementation ("this should be via JMX"). 👍
> > > > > > >
> > > > > > > On Nov 1, 2021, at 1:24 PM, David Capwell
> > <dcapwell@apple.com.INVALID>
> > > > > > > wrote:
> > > > > > >
> > > > > > >
> > > > > > > If anyone wants to bite off making
> > > > > > >
> > > >
> > https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
> >
> > > > > > > <
> > > > > > >
> > > >
> > https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
> >
> > > > >
> > > > > > > support mutability then we get vtable support. I am cool with JMX
> > > > and/or
> > > > > > > vtable, to me its just more important to allow dynamic setting of
> > these
> > > > > > > configs.
> > > > > > >
> > > > > > > On Nov 1, 2021, at 10:36 AM, benedict@apache.org wrote:
> > > > > > >
> > > > > > > having them only configured via yaml seems like a bad outcome
> > > > > > >
> > > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > I would like to see us move towards configuration being driven
> > through
> > > > > > > virtual tables where possible, so that the whole cluster can be
> > managed
> > > > > > > from a single interface. Not sure if this is the right place to bite
> > > > this
> > > > > > > off, but perhaps?
> > > > > > >
> > > > > > > From: Jeff Jirsa <jjirsa@gmail.com>
> > > > > > > Date: Monday, 1 November 2021 at 16:47
> > > > > > > To: Cassandra DEV <dev@cassandra.apache.org>
> > > > > > > Subject: Re: [DISCUSS] CEP-3: Guardrails
> > > > > > > Without bike-shedding too much, guardrails would be great, building
> > > > them
> > > > > > > into a more general purpose framework that limits various dangerous
> > > > > > > things
> > > > > > > would be fantastic. The CEP says that the guardrails should be
> > distinct
> > > > > > > from the capability restrictions (
> > > > > > > https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't
> > > > see
> > > > > > > why
> > > > > > > that needs to be the case. A system-level guardrail and a
> > > > personal-level
> > > > > > > guardrail are both restrictions, they just have different scopes, so
> > > > > > > implement the restriction framework first, and allow the scopes to
> > be
> > > > > > > expanded as needed?
> > > > > > >
> > > > > > > Naming wise, I don't know that I'd actually surface these as
> > > > > > > "guardrails",
> > > > > > > but more as general "limits", and having them only configured via
> > yaml
> > > > > > > seems like a bad outcome
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > https://issues.apache.org/jira/browse/CASSANDRA-8303
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <
> > adelapena@apache.org
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > I'd like to start a discussion about Guardrails proposal:
> > > > > > >
> > > > > > >
> > > > > > >
> > > >
> > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
> >
> > > > > > >
> > > > > > > Guardrails are an easy way to enforce system-wide soft and hard
> > limits
> > > > to
> > > > > > > prevent anti-patterns of bad usage and in the long run make it not
> > > > > > > possible
> > > > > > > to severely degrade the performance of a node/cluster through user
> > > > > > > actions
> > > > > > > such as having too many secondary indexes, too large partitions,
> > almost
> > > > > > > full disks, etc.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: dev-help@cassandra.apache.org
> >
> >
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic