[prev in list] [next in list] [prev in thread] [next in thread]
List: cassandra-commits
Subject: [jira] [Updated] (CASSANDRA-19508) Getting tons of msgs "Failed to get peer certificates for peer /x
From: "Mohammad Aburadeh (Jira)" <jira () apache ! org>
Date: 2024-03-31 22:40:00
Message-ID: JIRA.13574074.1711924482000.101586.1711924800037 () Atlassian ! JIRA
[Download RAW message or body]
[ https://issues.apache.org/jira/browse/CASSANDRA-19508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel \
]
Mohammad Aburadeh updated CASSANDRA-19508:
------------------------------------------
Description:
We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started seeing \
thousands of msgs "Failed to get peer certificates for peer /x.x.x.x:45796". SSL is \
enabled but require_client_auth is disabled. This is causing a huge problem for us \
because cassandra log files are growing very fast as our connections are short live \
connections, we open more than 1K connections per second and they stay live for 1-2 \
seconds. {code:java}
DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026 ServerConnection.java:140 \
- Failed to get peer certificates for peer /172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
at io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
at org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
at org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
at org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
at org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) \
{code}
*Our SSL config:*
{code:java}
client_encryption_options:
enabled: true
keystore: /path/to/keystore
keystore_password: xxxxx
optional: false
require_client_auth: false {code}
We should stop throwing this msg when require_client_auth is set to false. Or at \
least it should be logged in TRACE not DEBUG.
I'm working on preparing a PR.
was:
We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started seeing \
thousands of msgs "Failed to get peer certificates for peer /x.x.x.x:45796". SSL is \
enabled but require_client_auth is disabled. This is causing a huge problem for us \
because cassandra log files are growing very fast as our connections are short live \
connections, we open more than 1K connections per second and they stay live for 1-2 \
seconds.
{code:java}
DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026 ServerConnection.java:140 \
- Failed to get peer certificates for peer /172.31.2.23:45796
javax.net.ssl.SSLPeerUnverifiedException: peer not verified
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
at io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
at org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
at org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
at org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
at org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) \
{code}
*Our SSL config:*
{code:java}
client_encryption_options:
enabled: true
keystore: /path/to/keystore
keystore_password: xxxxx
optional: false
require_client_auth: false {code}
We should stop throwing this msg when require_client_auth is set to false. Or at \
least it should be logged in TRACE not DEBUG.
I'm working on preparing a PR.
> Getting tons of msgs "Failed to get peer certificates for peer /x.x.x.x:45796" when \
> require_client_auth is set to false
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-19508
> URL: https://issues.apache.org/jira/browse/CASSANDRA-19508
> Project: Cassandra
> Issue Type: Bug
> Reporter: Mohammad Aburadeh
> Priority: Urgent
>
> We recently upgraded our production clusters from 3.11.15 to 4.1.4. We started \
> seeing thousands of msgs "Failed to get peer certificates for peer /x.x.x.x:45796". \
> SSL is enabled but require_client_auth is disabled. This is causing a huge \
> problem for us because cassandra log files are growing very fast as our connections \
> are short live connections, we open more than 1K connections per second and they \
> stay live for 1-2 seconds. {code:java}
> DEBUG [Native-Transport-Requests-2] 2024-03-31 21:26:38,026 \
> ServerConnection.java:140 - Failed to get peer certificates for peer \
> /172.31.2.23:45796
> javax.net.ssl.SSLPeerUnverifiedException: peer not verified
> at io.netty.handler.ssl.ReferenceCountedOpenSslEngine$DefaultOpenSslSession.getPeerCertificateChain(ReferenceCountedOpenSslEngine.java:2414)
> at io.netty.handler.ssl.ExtendedOpenSslSession.getPeerCertificateChain(ExtendedOpenSslSession.java:140)
> at org.apache.cassandra.transport.ServerConnection.certificates(ServerConnection.java:136)
> at org.apache.cassandra.transport.ServerConnection.getSaslNegotiator(ServerConnection.java:120)
> at org.apache.cassandra.transport.messages.AuthResponse.execute(AuthResponse.java:76)
> at org.apache.cassandra.transport.Message$Request.execute(Message.java:255)
> at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:166)
> at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:185)
> at org.apache.cassandra.transport.Dispatcher.processRequest(Dispatcher.java:212)
> at org.apache.cassandra.transport.Dispatcher$RequestProcessor.run(Dispatcher.java:109)
> at org.apache.cassandra.concurrent.FutureTask$1.call(FutureTask.java:96)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:142)
> at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) \
> {code}
> *Our SSL config:*
> {code:java}
> client_encryption_options:
> enabled: true
> keystore: /path/to/keystore
> keystore_password: xxxxx
> optional: false
> require_client_auth: false {code}
>
> We should stop throwing this msg when require_client_auth is set to false. Or at \
> least it should be logged in TRACE not DEBUG. I'm working on preparing a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic