[prev in list] [next in list] [prev in thread] [next in thread] 

List:       centos
Subject:    Re: [CentOS] mdadm raid-check
From:       Valeri Galtsev <galtsev () kicp ! uchicago ! edu>
Date:       2020-11-16 15:28:04
Message-ID: 6E4E3341-BFD7-4DB8-B28C-2D4B9CE0089A () kicp ! uchicago ! edu
[Download RAW message or body]



> On Nov 16, 2020, at 2:48 AM, hw <hw@gc-24.de> wrote:
> 
> On Sat, 2020-11-14 at 21:55 -0600, Valeri Galtsev wrote:
> > > On Nov 14, 2020, at 8:20 PM, hw <hw@gc-24.de> wrote:
> > > 
> > > 
> > > Hi,
> > > 
> > > is it required to run /usr/sbin/raid-check once per week?  Centos 7 does
> > > this.  Maybe it's sufficient to run it monthly?  IIRC Debian did it monthly.
> > 
> > On hardware RAIDs I do RAID verification once a week. Once a Month a
> > not often enough in my book. That RAID verification effectively
> > reads all stripes of all drives (and verifies that content of
> > redundant drives is consistent), thus preventing a "time bomb", when
> > a drive left alone for too long, ready to fail in an area which is
> > not accessed, and failing when at some point different drive was
> > replaced and RAID rebuild has to go over all stripes of all
> > drives. Such "multiple failures" are due to poor sysadmin's work:
> > not often enough RAID verification.
> 
> You mean there can be failures which can be detected during a
> raid-check and can still be repaired using the other disk, but they
> can be impossible to repair when a disk has failed?

No, what I meant to say is: the errors could have been detected, and the drive would \
be kicked out of RAID (not errors repaired), and replaced with good drive long ago. \
But if RAID is not being checked often, there is potential that more than redundancy \
number of drives are failed (in different areas) and are waiting to be kicked out, \
and when it happens the failure becomes fatal.

> > If software raid-check does the same, then it makes a lot of sense,
> > and I am more with RedHat's weekly cron job, than with Debian's
> > Monthly.
> 
> How often do partial failures occur during normal operation?

I do not know what you mean by "partial failures". I can imagine:

1. checksum does not match, no reason to suspect any of drives which wrong \
information comes from. If it is RAID-6, in assumption that only one drive provided \
wrong information, wrong drive can be pinpointed, and stripe on it overwritten, the \
event is over without data messed up. If it is RAID-5, there is no way to pinpoint \
wrong drive, if your setting in RAID firmware (I am speaking only about hardware \
RAIDs here) is to overwrite "parity", fair chance is stripe on drive that gave \
correct information is overwritten, and the content on RAID device is damaged.

2. checksum does not match and one of the drives responded with significant delay. If \
there is no other way to pinpoint which drive wrong information came from, drive with \
delay can be fair suspect to be the one (it had to take time to multiple times read \
"bad block" and maybe re-allocate it). With fair certainty (but not 100%), RAID will \
handle the situation without data corruption.

3. One of the drives timed out or reported I/O error. The drive will be kicked out of \
RAID, and it is on operator's side to decide whether to replace it or to attempt to \
rebuild RAID onto the same drive.

> In case
> there was a power failure, it's probably a good idea to do a check
> anyway.

If you care about data on your RAID, you will use battery backup unit, which will \
keep the content of volatile RAM cache without loosing it, so when power has \
returned, the cache can be flashed to the drives. (Without cache hardware RAID \
devices are noticeably slower than with cache enabled). [non-volatile caches and \
supercapacitors are used as well]

However, the drives themselves have volatile memory as cache, that will evaporate \
when power suddenly disappears. To make things worse: drives are designed to lie \
about "transaction complete" (thus manufacturers can declare better specs than those \
of competitors), and "transaction complete" is reported when data is still in drives \
volatile cache, not on the platters. As far as I know, there is no way to query drive \
to get honest answer whether data is already on platters or not. Therefore, hardware \
RAID cards may think some of transactions are completed, but they may never become \
completed in case of power loss.

So, when power suddenly goes… it potentially is a mess on I/O intense box. Even \
with RAID battery backup for cache (or RAID cache disabled), having machine behind \
UPS, and starting clean shutdown when battery in UPS has less than [3 minutes in my \
case, yours may be different] juice left is a good idea.


I hope, this helps.

Valeri

> > Valeri
> > 
> > > I just checked on Fedora 32.  It does not run raid-check at all, at least not
> > > via a cron entry.  /usr/sbin/raid-check is available, though.  Is that an
> > > oversight?  (I started it manually now and will check if it's run once I update
> > > to 33.)
> 
> _______________________________________________
> CentOS mailing list
> CentOS@centos.org
> https://lists.centos.org/mailman/listinfo/centos

_______________________________________________
CentOS mailing list
CentOS@centos.org
https://lists.centos.org/mailman/listinfo/centos


[prev in list] [next in list] [prev in thread] [next in thread]