[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] How network latency affects ceph performance really with NVME only storage?
From:       Stefan Bauer <sb () plzk ! de>
Date:       2024-05-21 10:02:46
Message-ID: cb102ad0-0a37-4f85-8135-723cdb20b72e () plzk ! de
[Download RAW message or body]

Dear Users,

i recently setup a new ceph 3 node cluster. Network is meshed between 
all nodes (2 x 25G with DAC).
Storage is flash only (Kioxia 3.2 TBBiCS FLASH 3D TLC, KCMYXVUG3T20)

The latency with ping tests between the nodes shows:

# ping 10.1.3.13
PING 10.1.3.13 (10.1.3.13) 56(84) bytes of data.
64 bytes from 10.1.3.13: icmp_seq=1 ttl=64 time=0.145 ms
64 bytes from 10.1.3.13: icmp_seq=2 ttl=64 time=0.180 ms
64 bytes from 10.1.3.13: icmp_seq=3 ttl=64 time=0.180 ms
64 bytes from 10.1.3.13: icmp_seq=4 ttl=64 time=0.115 ms
64 bytes from 10.1.3.13: icmp_seq=5 ttl=64 time=0.110 ms
64 bytes from 10.1.3.13: icmp_seq=6 ttl=64 time=0.120 ms
64 bytes from 10.1.3.13: icmp_seq=7 ttl=64 time=0.124 ms
64 bytes from 10.1.3.13: icmp_seq=8 ttl=64 time=0.140 ms
64 bytes from 10.1.3.13: icmp_seq=9 ttl=64 time=0.127 ms
64 bytes from 10.1.3.13: icmp_seq=10 ttl=64 time=0.143 ms
64 bytes from 10.1.3.13: icmp_seq=11 ttl=64 time=0.129 ms
--- 10.1.3.13 ping statistics ---
11 packets transmitted, 11 received, 0% packet loss, time 10242ms
rtt min/avg/max/mdev = 0.110/0.137/0.180/0.022 ms


On another cluster i have much better values, with 10G SFP+ and 
fibre-cables:

64 bytes from large-ipv6-ip: icmp_seq=42 ttl=64 time=0.081 ms
64 bytes from large-ipv6-ip: icmp_seq=43 ttl=64 time=0.078 ms
64 bytes from large-ipv6-ip: icmp_seq=44 ttl=64 time=0.084 ms
64 bytes from large-ipv6-ip: icmp_seq=45 ttl=64 time=0.075 ms
64 bytes from large-ipv6-ip: icmp_seq=46 ttl=64 time=0.071 ms
64 bytes from large-ipv6-ip: icmp_seq=47 ttl=64 time=0.081 ms
64 bytes from large-ipv6-ip: icmp_seq=48 ttl=64 time=0.074 ms
64 bytes from large-ipv6-ip: icmp_seq=49 ttl=64 time=0.085 ms
64 bytes from large-ipv6-ip: icmp_seq=50 ttl=64 time=0.077 ms
64 bytes from large-ipv6-ip: icmp_seq=51 ttl=64 time=0.080 ms
64 bytes from large-ipv6-ip: icmp_seq=52 ttl=64 time=0.084 ms
64 bytes from large-ipv6-ip: icmp_seq=53 ttl=64 time=0.084 ms
^C
--- long-ipv6-ip ping statistics ---
53 packets transmitted, 53 received, 0% packet loss, time 53260ms
rtt min/avg/max/mdev = 0.071/0.082/0.111/0.006 ms

If i want best performance, does the latency difference matter at all? 
Should i change DAC to SFP-transceivers wwith fibre-cables to improve 
overall ceph performance or is this nitpicking?

Thanks a lot.

Stefan
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
[prev in list] [next in list] [prev in thread] [next in thread]