[prev in list] [next in list] [prev in thread] [next in thread] 

List:       linux-btrfs
Subject:    Re: Extremely slow device removals
From:       Phil Karn <karn () ka9q ! net>
Date:       2020-04-30 20:58:48
Message-ID: de1c1b3e-09ac-bdc8-c2fe-d95c2bffb766 () ka9q ! net
[Download RAW message or body]

On 4/30/20 13:27, Alexandru Dordea wrote:
> Hello,
> 
> I'm encountering the same issue with Raid6 for months :)
> I have a BTRFS raid6 with 15x8TB HDD's and 5 x 14TB. One of the 15 x 8TB crashed. I \
> have removed the faulty drive and if I'm running the delete missing command the \
> sys-load is increasing and to recover 6.66TB will take few months. After 5 days of \
> running the missing data decreased to -6.10. During this period the drivers are \
> almost 100% and the R/W performance is degraded with more than 95%.
I see I have company, and that a more recent kernel has the same problem.
> 
> The R/W performance is not impacted if the process of delete/balance is not \
> running. (Don't know if running balance on a single CPU without multithread is a \
> feature or a bug but it's a shame that the process is keeping only one CPU out of \
> 48 at 100%).

I'm using RAID-1 rather than 6, but for me there's very little CPU
consumption. Nor would I expect there to be, since the work is all in
the copying. I have 8 cores and am running Folding at Home (Covid-19
drug discovery) on 6 of them, but there seems to be plenty of CPU
available; idle time is consistently 12-15%. Still, I tried pausing FAH.
There was no discernable effect on the btrfs remove/copy operation, nor
would I expect there to be since FAH is entirely CPU-bound and the
remove/copy is entirely I/O bound. The 'btrfs remove' command uses
relatively little CPU and always shows as waiting on disk in the 'top'
command. Same for the kernel worker threads.

But just in case, I've scaled FAH back to 3 threads to see what happens.

I'm thinking maybe it's time to go back to dm-raid for RAID-1 and keep
btrfs only for its snapshot feature. Integrating RAID into the file
system seemed like a really good idea at the time, but snapshots alone
are still worth it.

When I ran XFS above dm-raid1, I'd periodically pull one drive, put it
in the safe, replace it with a blank drive and let it rebuild. This gave
me a full image backup, and the rebuild copy went at full disk speed
though it did have to copy unused disk. But based on what I'm seeing
now, that's preferable. A full disk image copy at sequential disk speed
is still much faster than copying only the used blocks in semi-random
order and shaking the hell out of my drives.

I wonder if putting bcache and a SSD between btrfs and my drives would
help...? How about a few hundred GB of RAM (I have only 12)?

--Phil


[prev in list] [next in list] [prev in thread] [next in thread]