[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] osd latency but disks do not seem busy
From:       Ml Ml <mliebherr99 () googlemail ! com>
Date:       2022-05-29 14:48:38
Message-ID: CANFxOjDGQ1V-+V5PWdv3M=rKwnZ_K9q7NwJJzSPbwo9ik7KZNQ () mail ! gmail ! com
[Download RAW message or body]

Hello,
i have 15.2.12 and ceph osd perf:

root@cluster5-node01:~# ceph osd perf
osd  commit_latency(ms)  apply_latency(ms)
  4                 830                830
 11                 885                885
 15                 878                878
 10                2317               2317
 16                1453               1453
  9                 508                508
 18                1197               1197
  1                 401                401
 20                1032               1032
  3                 680                680
 17                 383                383
  6                2791               2791
  5                 796                796
 14                1622               1622
 13                 851                851
 12                2141               2141
  8                1535               1535
  2                 457                457

which is kind of killig itself, but iostat -dx 3 does not show busy disks.
However, top shows 100% CPU Load for the osd processes.

Any idea why that is? Here is my status:

root@cluster5-node01:~# ceph -s
  cluster:
    id:     e1153ea5-bb07-4548-83a9-edd8bae3eeec
    health: HEALTH_WARN
            noout flag(s) set
            4 nearfull osd(s)
            Reduced data availability: 12 pgs inactive, 35 pgs peering
            Degraded data redundancy: 14594/14947641 objects degraded
(0.098%), 96 pgs degraded
            1 pool(s) do not have an application enabled
            3 pool(s) nearfull
            10 daemons have recently crashed
            4 slow ops, oldest one blocked for 250 sec, daemons
[osd.14,osd.15,osd.17,osd.18,osd.5,osd.9] have slow ops.

  services:
    mon: 3 daemons, quorum
cluster5-node01,cluster5-node02,cluster5-node03 (age 2h)
    mgr: cluster5-node03(active, since 92m), standbys:
cluster5-node02, cluster5-node01
    osd: 18 osds: 18 up (since 29s), 18 in (since 9M); 39 remapped pgs
         flags noout

  task status:

  data:
    pools:   3 pools, 1143 pgs
    objects: 4.98M objects, 17 TiB
    usage:   53 TiB used, 10 TiB / 63 TiB avail
    pgs:     13.561% pgs not active
             14594/14947641 objects degraded (0.098%)
             137011/14947641 objects misplaced (0.917%)
             437 active+clean+snaptrim_wait
             280 active+clean
             136 active+clean+snaptrim
             80  peering
             54  active+recovering+degraded
             53  activating
             24  active+recovery_wait+degraded
             15  active+remapped+backfill_wait
             11  activating+degraded
             9   active+remapped+backfilling
             8   active+recovering
             8   remapped+peering
             6   active+clean+snaptrim_wait+laggy
             4   active
             3   active+recovering+degraded+wait
             3   active+clean+wait
             3   activating+remapped
             2   active+recovery_wait
             2   active+undersized+degraded
             2   active+undersized+degraded+remapped+backfill_wait
             2   active+remapped
             1   active+clean+laggy

  io:
    client:   1.4 KiB/s wr, 0 op/s rd, 0 op/s wr
    recovery: 990 KiB/s, 1 objects/s


cheers,
mario
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io
[prev in list] [next in list] [prev in thread] [next in thread]