[prev in list] [next in list] [prev in thread] [next in thread] 

List:       openbsd-misc
Subject:    Icinga2 endpoints unable to connect to master after update to current package 2.12.1-1
From:       "Theodore Wynnychenko" <tmw () uchicago ! edu>
Date:       2020-11-23 23:10:42
Message-ID: 000001d6c1ed$de428680$9ac79380$ () edu
[Download RAW message or body]

Hello

The other day I updated to current (6.8 GENERIC.MP#188).

I then updated packages.

I have been using Icinga2 since about OpenBSD 5.6, and everything was fine.

A few hours after the update, I got a warning that my /var/log filesystem on
the icinga2 master was full.

Then, I noticed warnings in icinga2 for pretty much every check that state:

	"Error: Function call 'pipe2' failed with error code 24, 'Too many
open files'"

I only have a couple of dozen endpoints, and have never had this issue
before.  I tried increasing the file limits, but that only increased the
time before icinga2 crashed into the limit with too many open files.

I then noticed that icinga2 was now throwing a warning about self signed
certificates.

Specifically, I was getting log messages on the endpoints "New client
connection for identity 'master.my.tld' to [172.xx.xx.99]:5665 (certificate
validation failed: code 19: self signed certificate in certificate chain)".

On the master, I was getting the same, but inverted, error:  "New client
connection for identity 'endpoint.my.tld' from [172.xx.xx.1]:3621
(certificate validation failed: code 19: self signed certificate in
certificate chain)".

So, I decided to add the icinga CA certificate to the list of trusted
certificates in /etc/ssl/cert.pem on both master and endpoint.

Now, when I connect (either from master to endpoint, or the reverse) using
s_client, I see:

openssl s_client -connect endpoint.my.tld:5665

CONNECTED(00000003)
depth=1 CN = Icinga CA
verify return:1
depth=1 CN = Icinga CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
depth=1 CN = Icinga CA
verify return:1
depth=0 CN = endpoint.my.tld
verify return:1
depth=0 CN = endpoint.my.tld
verify return:1
write W BLOCK
---
Certificate chain
 0 s:/CN=endpoint.my.tld
   i:/CN=Icinga CA
 1 s:/CN=Icinga CA
   i:/CN=Icinga CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFAjCCAuqgAwIBAgIUZuaLfnjnY6dLkKTQ3J6GGGcCYnowDQYJKoZIhvcNAQEL
BQAwFDESMBAGA1UEAwwJSWNpbmdhIENBMB4XDTE4MDUxNjE2MzQzN1oXDTMzMDUx
MjE2MzQzN1owJzElMCMGA1UEAwwccGlwcGluLnNhbWJhLnd5bm55Y2hlbmtvLmNv
bTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAJzPMaQdWkhb/YMy242C
jmbRVcwOqZrCRPwscYhAfhfSwNJfSN7k5I9UCFszgvAU3QU3RhEElrJYOQY7UKp1
V7D4MO88S5NMh6rLrjyCuojxVnbwCA4WwIXMA0zNY6EEG8/1LbcfA8utSy1Y1X0c
xb4FJKv3ar02j9A5XleZf0p9bKQezysxB3TT17L4AhWsoE1w/7GCfU715OEch+Dw
WI9TusrJXKLFwHvk1j+ZCjiNM49F7gFDpw3m/Asekt5B3M6L7ZkPM9jI1ThX4etj
kEC1C2371lP9OdFExqScLudHCL8+2IILd3i+/7YxWFTnxhFszyWMDiMll/omcpot
qDbFhQdVrfnTo4lczK42EfhQuDMdESkmvay8UVGyUe90AEaQ38V3w8B0iDzgGQTX
UZNxpurotW2zU1XPhzTOawRlp1POFQ1tnFJzH+iyBCxZZZfZsQchJbMJz/JCGIiW
qkhTN5bAKoAgO6GglTmqktSZmixa38fZ8qIJ8Y0UZsnH5zRQWcbzKr49u3FhH2rH
pC9Dh+zlRj/LiMM4c/UF2LzwuIQ3fjZMEff+q3vzs6cXgs8FICX7uvpB4yqqt4Vn
/ZSRrq++dUAav+0JhTYlW2m+sa/ga4HVAB+zNN7+l92PWSJ0b3DDzZt6CHLp/lWX
zCPDpo7ABbk+xrg+NEoym8ejAgMBAAGjOTA3MAwGA1UdEwEB/wQCMAAwJwYDVR0R
BCAwHoIccGlwcGluLnNhbWJhLnd5bm55Y2hlbmtvLmNvbTANBgkqhkiG9w0BAQsF
AAOCAgEAGh1goJVNy4Ltpv0+x1okod55g7ob6/l2hAwrq0jXBca4zIGncQcdl0jg
+z6TDMiq+2UUoKB80k5D947t2VHtp+d/wuSTNwpzESNplh5GWpqkdpOHcAN1lkku
ZgCUnQH/ZFa4Q2V01rPHSaf1znMpaqaYTjAKwNwZY9qRxjXUZg62/D/y9pfmy+VC
yvZype5rETXjLxr0WN6LABRgda41wiLszMWLAonHRHRVkhdyUYC+brmDNhfByqGJ
L5oHpvCq9Oywk4zKO02y3wrhL1+JHt0TH/5RmaalWQRsB0vJY9699cnLk6DK/+CD
YyHecKplsnwnfvwau6aMlwg6zilCZ+YMl3Jwj0vQeG/h8DyTw6t9HtknlnRfcfQ/
eyTHZtdyH1y1O5v4BQJNt5Ewc7y144IP2g/9Y7g3n7GFlvd68TQjJmI6I4nGsJ+u
iNu+DGzD6Ih9UhFCWMrR57r9utdarBWYV4NJaXldNnqXrL099Hu3CLzdjikabeHE
J56gkXOHQdQi2xIyIgIuMHMF+niJkqAmxyGaMfnBOOv6Gvt7TAMLrt4dIFSgomKe
Rd6xSyCh66H/AYF8b4NlwVCjDmvQf31OtGrl8xVlXDnOeaUHoZsu/ECO9f7E5yuN
od5dNX4mJoSCkESbwQ67IumOLxSEw5extwTNplG/oEqgl6YalJI=
-----END CERTIFICATE-----
subject=/CN=endpoint.my.tld
issuer=/CN=Icinga CA
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 3436 bytes and written 392 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
AF10B5FA058B109699E87151A6BFCE6E9AD4968C04F6E8C1EFE24C8830AE7D5F
    Session-ID-ctx:
    Master-Key:
6509DF9A604E5FB4C7F3BD55DC4666FDD93315CA00AA8E373E8C41BD93E3E1D91961AEA35642
A684F45DA530C4FDF260
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - b1 ef 85 f6 29 57 ee 81-8f d7 af 12 de 8e 19 1e
....)W..........
    0010 - c4 3d b8 5d 68 1e d0 87-9a 88 09 b8 e5 b8 fd 7d
.=.]h..........}
    0020 - 6e 48 ea 63 5f df 83 54-9a d3 b4 3e e6 3a 30 a4
nH.c_..T...>.:0.
    0030 - 83 0b df 4d 3e 7b da a2-47 a0 c2 2b 2c 4e 4e f6
...M>{..G..+,NN.
    0040 - f3 b5 6c 24 da 6a f1 c8-bf 27 08 23 1e 37 21 9a
..l$.j...'.#.7!.
    0050 - 93 dd 87 a5 95 b8 72 3c-14 07 33 a1 e4 23 b7 2d
......r<..3..#.-
    0060 - 16 0e b8 ad c4 f9 be 72-a0 44 1f 09 c9 47 47 8a
.......r.D...GG.
    0070 - a6 97 10 55 77 a3 fe 7c-0f 2f 33 6c 40 9f 5a 76
...Uw..|./3l@.Zv
    0080 - 43 1b 17 21 44 d9 6a 15-82 b0 9e 42 da 14 78 4e
C..!D.j....B..xN
    0090 - 50 5c 19 2a a5 09 61 72-0a f5 11 11 6a 75 4c 67
P\.*..ar....juLg

    Start Time: 1606170242
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)


The same is true if I connect from an endpoint to the master:

openssl s_client -connect master.my.tld:5665
CONNECTED(00000003)
depth=1 CN = Icinga CA
verify return:1
depth=1 CN = Icinga CA
verify error:num=19:self signed certificate in certificate chain
verify return:0
depth=1 CN = Icinga CA
verify return:1
depth=0 CN = master.my.tld
verify return:1
depth=0 CN = master.my.tld
verify return:1
write W BLOCK
---
Certificate chain
 0 s:/CN= master.my.tld
   i:/CN=Icinga CA
 1 s:/CN=Icinga CA
   i:/CN=Icinga CA
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIExTCCAq2gAwIBAgIBAjANBgkqhkiG9w0BAQsFADAUMRIwEAYDVQQDDAlJY2lu
Z2EgQ0EwHhcNMTYwMzI5MTg1MzA1WhcNMzEwMzI2MTg1MzA1WjAmMSQwIgYDVQQD
DBttZXJyeS5zYW1iYS53eW5ueWNoZW5rby5jb20wggIiMA0GCSqGSIb3DQEBAQUA
A4ICDwAwggIKAoICAQC6Day5I9w3MeKVE9kNYtiH/N1prcEpq/v5UtsHXXi6rHAA
8Q9HrvOdEVZq3gNGb60tJ03yQQ7/hcyDuU/Rt0lpRiRpi4+pZx9TlTKCt8zT8zCy
bghFxL7F110yf68emwy90MHzsd+piaEsa/ccXfzKC3IgzA9+OXYLk+D/nWYk+qf+
5PWbo4EjTz/uMRJSJXYoNsHBHg14kwNOFHI/ewiF4GGLO5X2y0uoLHb945Kri2MF
6Dq7Hh1apX0PgHxW8bF17TCBXKqRZ+U9qOL5KFWaaxX3Fo84yIX1n8iHRGymS0fH
Jq54G0B8IwB1blPGdr36zbZREc0+m9ywLcw/DmgvIEKuz6UEIGCo0KwXGWsM57DD
butUGNa58782kl5vCf/Ol8ogLEBSWxA5C9tg8eyBYRn705EkLMkSFD3blV1yLAqr
E5VPFqNhA18O0iALRdUEWwNTFFttL81yLO6c3jnG8FVLhCv4PTA39Wbp/K5gshH3
2KHmeXE0jlaOWHFuTod8PEXSl4Ix/HqXPbXZyMrRufLVx/9Tpl95VkEw6euGCz8k
qumdSQP1SqsDOk5a4sU8s22/B+24k61SwObsCpxulFzOEPqfwAYxRfs3vrKssiys
M4EJeey5+f3Xm/sWxf6dfilqXJAywweIJkhuGg4RLMwinuAEmvcAzGSeMDzVZwID
AQABoxAwDjAMBgNVHRMBAf8EAjAAMA0GCSqGSIb3DQEBCwUAA4ICAQAeFDD7wRUd
MwgDoooHTzD1Bt6jFLHgY/0ICTLRPmDxl/WrT67x+NDgwqcWRTc4H81G6RuGBl6M
q5Hh96gaXq7en1JC9r1pg5IzA9fQE0ijNpp4uH5X/7RGD54ca12KSbA+roOPOHX8
fCXGI3xsWW/4LkRhCM9jfHhkle1G3rBvqBQ/WST2ONuj7EJFcXhBBuF2WavvOobU
zoS3RYMOq7S3SEtFyY1VvMrZpKvvOGcjYyTDaZxuT7sJ0O8qynr1rTd19dLMq1Wo
Jn6rILWNoZsXhFYJhBhAeKLak7qNpNlfeIdtCyvybgE4mksfMOrxd9+IhUdzbLuK
8/w/2rK5DJbI2jghHUR8wXrgBFaGCgFbx/rUamDuY09VQP3bCmH+WfvBid/7+M5s
l3VAXnSwI6Ez7KbbeUowznyruaeRiq74UMGa9qEwdnTd8+emu6clRwqPAT1te1Ji
2Tq0KiHu/2xlqKGOTvzY0Im207YpMGWO5Irnaf5o/EYPEWUFG6ngpryYwgoftZPZ
oL/5quNIFjGmAaspHnYFiHQF2KWhg7QUz5nQhm0V7YSybNsRUiYq+jVbgYFyGYIm
GJQOb71D3BiFlTbOXqn2nNK5qvIZNXMCyO7DQu5wAv291O+a9STqqjnScPHSvqZ+
fVMWrnq4BFolyOA9NowZwTpyI43oLffzOw==
-----END CERTIFICATE-----
subject=/CN= master.my.tld
issuer=/CN=Icinga CA
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 3375 bytes and written 392 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID:
AC7F7986150D6886FDFE2CD6B26B20B527D3857D0FDAF56DB85719C53A00DC04
    Session-ID-ctx:
    Master-Key:
6639461A0943FD8A84812B1E2D8C5C02958271502A0A9CD61D20D8306BF85F11B287F2A437A7
EE7E4E8A4BFE03813C0C
    TLS session ticket lifetime hint: 7200 (seconds)
    TLS session ticket:
    0000 - 55 22 c6 0c 47 72 ed 57-a3 2e 68 32 36 51 3e 7c
U"..Gr.W..h26Q>|
    0010 - 62 56 7d 88 82 f7 5c 6f-cf 38 c3 66 b1 ea cb 0a
bV}...\o.8.f....
    0020 - 7c c8 be bd 5f 8b b6 9a-2d c5 8f 01 80 79 88 52
|..._...-....y.R
    0030 - a2 22 5e 8c b0 57 4a db-16 be 3c b4 a2 9e 49 d7
."^..WJ...<...I.
    0040 - 63 d8 39 d5 a8 ee 2a b2-2f d2 60 25 b2 79 58 90
c.9...*./.`%.yX.
    0050 - 0f 71 f7 6f 68 cc 78 dc-90 7f 1d f1 04 66 7f 00
.q.oh.x......f..
    0060 - 60 bc ae 2a 0e ff 26 03-44 e1 fc b7 c5 a3 99 e6
`..*..&.D.......
    0070 - 8f d8 a8 8f 28 ce aa 92-88 32 16 02 6a c9 81 11
....(....2..j...
    0080 - 64 de f9 c5 f5 05 6f 40-a8 4a eb 16 ac 7b 93 d1
d.....o@.J...{..
    0090 - 4b 93 e2 97 71 0c 3d 67-a1 42 d5 15 aa 94 c2 9c
K...q.=g.B......

    Start Time: 1606170371
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)


But, despite the fact that s_client returns a "OK" when connecting to the
icinga2 port, the icinga2 process continues to fail with the "certificate
validation failed: code 19: self signed certificate in certificate chain"
messages.

I think this is the issue (as a minimally informed observer), and that the
failures in validation leave processes in icinga2 hanging until there are
too many open.

But, I have no idea of how to explore this further.

Please let me know if there is anything I should try, or any other
information that may be helpful in identifying what the issue may be.

As I said, this has been working for (what (6.8-5.6)/2 years), and I made no
changes after the update.

Thanks again
Ted




[prev in list] [next in list] [prev in thread] [next in thread]