[prev in list] [next in list] [prev in thread] [next in thread] 

List:       ceph-users
Subject:    [ceph-users] Re: High ceph_osd_commit_latency_ms on Toshiba MG07ACA14TE HDDs
From:       Reed Dier <reed.dier () focusvq ! com>
Date:       2020-06-24 16:06:09
Message-ID: 5680B679-2A6C-4649-9199-17B203C58DF4 () focusvq ! com
[Download RAW message or body]

[Attachment #2 (multipart/signed)]


Just throwing my hat in here with a small bit of anecdotal experience.

In the early days of experimenting with ceph, I had 24x 8T disk, all behind RAID \
controllers as R0 vd's with no BBU (so controller cache is WT, default value), and \
pdcache (disk write cache) enabled (default value).

We had a lightning strike at our previous data center that killed power, and we ended \
up losing the entire ceph pool (not prod), due mostly in part to the pdcache setting.

We then did an exhaustive failure test following that, further isolating the pdcache \
as the culprit, and not the controllers write cache. The controllers now have BBU's \
to further prevent issues, but WB cache with the BBU did not yield issues, only \
pdcache.

So, all of this to say, in my experience, the on-disk write cache was a huge \
liability for losing writes. This was also in the filestore days, and most of our \
issues were with XFS, but the point remains.

Write cache can be a consistency killer, and I recommend disabling where possible.

Reed

> On Jun 24, 2020, at 10:30 AM, Paul Emmerich <paul.emmerich@croit.io> wrote:
> 
> Has anyone ever encountered a drive with a write cache that actually
> *helped*?
> I haven't.
> 
> As in: would it be a good idea for the OSD to just disable the write cache
> on startup? Worst case it doesn't do anything, best case it improves
> latency.
> 
> Paul
> 
> -- 
> Paul Emmerich
> 
> Looking for help with your Ceph cluster? Contact us at https://croit.io
> 
> croit GmbH
> Freseniusstr. 31h
> 81247 München
> www.croit.io
> Tel: +49 89 1896585 90
> 
> 
> On Wed, Jun 24, 2020 at 3:49 PM Frank R <frankaritchie@gmail.com> wrote:
> 
> > fyi, there is an interesting note on disabling the write cache here:
> > 
> > 
> > https://yourcmc.ru/wiki/index.php?title=Ceph_performance&mobileaction=toggle_view_desktop#Drive_cache_is_slowing_you_down
> >  
> > On Wed, Jun 24, 2020 at 9:45 AM Benoît Knecht <bknecht@protonmail.ch>
> > wrote:
> > > 
> > > Hi Igor,
> > > 
> > > Igor Fedotov wrote:
> > > > for the sake of completeness one more experiment please if possible:
> > > > 
> > > > turn off write cache for HGST drives and measure commit latency once
> > again.
> > > 
> > > I just did the same experiment with HGST drives, and disabling the write
> > cache
> > > on those drives brought the latency down from about 7.5ms to about 4ms.
> > > 
> > > So it seems disabling the write cache across the board would be
> > advisable in
> > > our case. Is it recommended in general, or specifically when the DB+WAL
> > is on
> > > the same hard drive?
> > > 
> > > Stefan, Mark, are you disabling the write cache on your HDDs by default?
> > > 
> > > Cheers,
> > > 
> > > --
> > > Ben
> > > _______________________________________________
> > > ceph-users mailing list -- ceph-users@ceph.io
> > > To unsubscribe send an email to ceph-users-leave@ceph.io
> > _______________________________________________
> > ceph-users mailing list -- ceph-users@ceph.io
> > To unsubscribe send an email to ceph-users-leave@ceph.io
> > 
> _______________________________________________
> ceph-users mailing list -- ceph-users@ceph.io
> To unsubscribe send an email to ceph-users-leave@ceph.io


["smime.p7s" (smime.p7s)]

0	*H
 010
	`He0	*H
 j00 M,;	Ł-:+O0
	*H
010	UUS10U
New Jersey10UJersey City10U
The USERTRUST Network1.0,U%USERTrust RSA Certification Authority0
181102000000Z
301231235959Z010	UGB10UGreater Manchester10USalford10U
Sectigo Limited1>0<U5Sectigo RSA Client Authentication and Secure Email \
CA0"0 	*H
0
<
AF;ڒ+N&)**͜P?m.gJLu"%^HJBJ2*MŚ3|qCf^/ԾMS< \
5AFaTQ$7u7͈(l:=ϋnty!5{ş>q[ҡV  \
":1Av9UF~1˥RDOt=rseA)7lnvϐSd \
0`0U#0SyZ+JT؛f0U	ڔ_+ߨB0U0U00U%0++0U \
 00U 0PUI0G0E C \
A?http://crl.usertrust.com/USERTrustRSACertificationAuthority.crl0v+j0h0? \
+03http://crt.usertrust.com/USERTrustRSAAddTrustCA.crt0%+0http://ocsp.usertrust.com0
 	*H
ADu9 ގm2qhz8㱽1ゴ~!anIB<YT
?iD
F b6]6o3t h:U
!S y=\,i /_	~,ƖcIJ41*[T|)MdP_)IkdPo}y
ِ49	$*6UTC{8^]EuzSp4xwMire}sYHOl03]P;g'NYs+">Z][)r-6F^y-3d
 
d6|nkOKՈ3 \
l@9l6f?<tHfYWzF<4E!Vu+9X/15e")Jpj&כ \
:'3KEZP7#~x8Ǜt@Rg-mƐyB4Z9a72ΦIa7~-"׀o&ɳ6'0R0: \
/	Tʹ(̽P0 	*H
010	UGB10UGreater Manchester10USalford10U
Sectigo Limited1>0<U5Sectigo RSA Client Authentication and Secure Email CA0
190926000000Z
220925235959Z0_10	UUS10U7000510U	Louisiana10UMetairie1,0*U	#Ste \
250 The Heritage Plaza Building1#0!U	111 Veterans Memorial Blvd1&0$U Focus \
Automated Equities, LLC1C0AU:Issued through Focus Automated Equities, LLC E-PKI \
Manager10UCorporate Secure Email10U	Reed Dier1$0"	*H \
	reed.dier@focusvq.com0"0 	*H
0
e3&,nF:J&y \
aϼˏ#sFy[<t͏t곝ך۸*o;ɨB@>X8?[h \
1Itabp0^)k0|pịRD9sg;ȪrOG}s`pȎC*ʇCS'Ih
 ZnkZ'"a0)~+0F500U#0	ڔ_+ߨB0UuavԆ80:0U \
0U00U%0++0@U \
90705+10%0#+https://sectigo.com/CPS0ZUS0Q0O M \
KIhttp://crl.sectigo.com/SectigoRSAClientAuthenticationandSecureEmailCA.crl0+ \
~0|0U+0Ihttp://crt.sectigo.com/SectigoRSAClientAuthenticationandSecureEmailCA.crt0#+0http://ocsp.sectigo.com0 \
U0reed.dier@focusvq.com0 	*H
OT~Fp_ir{PM%i6ap愀(Khy
wJdzX<Ă& \
*:85Y0Ljc7]Q"-e6,2az4'-6<?'Qn_mo!{8yl(zu0[x(TpuW8H9v<O3oRJ5 \
	HûzG#L~26u>%Xm5 ʕn100010	UGB10UGreater \
Manchester10USalford10U Sectigo Limited1>0<U5Sectigo RSA Client \
Authentication and Secure Email CA/	Tʹ(̽P0 	`He 0	*H
	1	*H
0	*H
	1
200624160609Z0/	*H
	1" PJh^TbmWh֖Z0	+710010	UGB10UGreater \
Manchester10USalford10U Sectigo Limited1>0<U5Sectigo RSA Client \
Authentication and Secure Email CA/	Tʹ(̽P0*H 	1 \
010	UGB10UGreater Manchester10USalford10U Sectigo \
Limited1>0<U5Sectigo RSA Client Authentication and Secure Email \
CA/	Tʹ(̽P0 	*H
Skdހ
E5rx;.2$`}8!%f[
Qz~X_}7	Q,>OȤ㮋eK4;CJix:q&,Ir^	㗦8.G
ߦK4M-*U FFc9[Y=(ѝ%|U|o 4e$0CҴ֭	r'm$a)
ɓto-!O&V@\e1A;ols_#Q
d"E



_______________________________________________
ceph-users mailing list -- ceph-users@ceph.io
To unsubscribe send an email to ceph-users-leave@ceph.io


[prev in list] [next in list] [prev in thread] [next in thread]

Configure | About | News | Add a list | Sponsored by KoreLogic