[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs
From:       Igor Fedotov <ifedotov () suse ! de>
Date:       2020-06-24 12:24:05
Message-ID: b867420b-75b7-e9c2-6600-c6b4cc9a7e27 () suse ! de
[Download RAW message or body]

Benoit, wondering what are the write cache settings in your case?

And do you see any difference after disabling it if any?


Thanks,

Igor


On 6/24/2020 3:16 PM, Mark Nelson wrote:
> This isn't the first time I've seen drive cache cause problematic 
> latency issues, and not always from the same manufacturer. 
> Unfortunately it seems like you really have to test the drives you 
> want to use before deploying them them to make sure you don't run into 
> issues.
> 
> 
> Mark
> 
> 
> On 6/24/20 6:36 AM, Stefan Priebe - Profihost AG wrote:
> > HI Ben,
> > 
> > yes we have the same issues and switched to seagate for those reasons.
> > 
> > you can fix at least a big part of it by disabling the write cache of
> > those drives - generally speaking it seems the toshiba firmware is 
> > broken.
> > 
> > I was not able to find a newer one.
> > 
> > Greets,
> > Stefan
> > 
> > Am 24.06.20 um 09:43 schrieb Benoît Knecht:
> > > Hi,
> > > 
> > > We have a Nautilus (14.2.9) Ceph cluster with two types of HDDs:
> > > 
> > > - TOSHIBA MG07ACA14TE   [1]
> > > - HGST HUH721212ALE604  [2]
> > > 
> > > They're all bluestore OSDs with no separate DB+WAL and part of the 
> > > same pool.
> > > 
> > > We noticed that while the HGST OSDs have a commit latency of about 
> > > 15ms, the Toshiba OSDs hover around 150ms (these values come from 
> > > the `ceph_osd_commit_latency_ms` metric in Prometheus).
> > > 
> > > On paper, it seems like those drives have very similar specs, so 
> > > it's not clear to me why we're seeing such a large difference when 
> > > it comes to commit latency.
> > > 
> > > Has anyone had any experience with those Toshiba drives? Or looking 
> > > at the specs, do you spot anything suspicious?
> > > 
> > > And if you're running a Ceph cluster with various disk 
> > > brands/models, have you ever noticed some of them standing out when 
> > > looking at `ceph_osd_commit_latency_ms`?
> > > 
> > > Thanks in advance for your feedback.
> > > 
> > > Cheers,
> > > 
> > > -- 
> > > Ben
> > > 
> > > [1]: 
> > > https://toshiba.semicon-storage.com/content/dam/toshiba-ss/asia-pacific/docs/product/storage/product-manual/eHDD-MG07ACA-Product-Manual.pdf
> > >  [2]: 
> > > https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public \
> > > /western-digital/product/data-center-drives/ultrastar-dc-hc500-series/data-sheet-ultrastar-dc-hc520.pdf
> > >  _______________________________________________
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-leave@ceph.io
> > > 
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-leave@ceph.io
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io
_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io


[prev in list] [next in list] [prev in thread] [next in thread]