[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    HDFS block replication problem
From:       Hendrik Haddorp <hendrik.haddorp () gmx ! net>
Date:       2017-11-27 16:57:39
Message-ID: e4cbce08-934c-621e-2d67-2a5665df3ecd () gmx ! net
[Download RAW message or body]

Hi,

I have a HDFS test cluster with 7 data nodes and one name node. During 
write operations from Solr I get errors on the client side and messages 
like this in the name node logs:

017-11-27 11:16:09,130 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: 
Failed to choose from the next rack (location = /default/rack_02), retry 
choosing ramdomly

I also have entries like this:

2017-11-27 11:16:09,129 DEBUG 
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: 
Failed to choose from local rack (location = /default/rack_01), retry 
with the rack of the next replica (location = /default/rack_02)
org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy$NotEnoughReplicasException: 
[
Node /default/rack_01/10.37.131.184:50010 [
    Storage 
[DISK]DS-35c4ec83-f26f-44fb-a3be-21baf037357e:NORMAL:10.37.131.184:50010 
is not chosen since the node does not have enough DISK space 
(required=134217728, scheduled=14495514624, remaining=14205731794).
]

This seems to indicate that the disk on the data node is full. However 
when running "hdfs dfsadmin -report" I get the following for that node:

Name: 10.37.131.184:50010 (..........)
Hostname: ................
Rack: /default/rack_01
Decommission Status : Normal
Configured Capacity: 47766564864 (44.49 GB)
DFS Used: 4933282704 (4.59 GB)
Non DFS Used: 14173148272 (13.20 GB)
DFS Remaining: 28660133888 (26.69 GB)
DFS Used%: 10.33%
DFS Remaining%: 60.00%
Configured Cache Capacity: 0 (0 B)
Cache Used: 0 (0 B)
Cache Remaining: 0 (0 B)
Cache Used%: 100.00%
Cache Remaining%: 0.00%
Xceivers: 1
Last contact: Mon Nov 27 11:52:59 EST 2017

According to that that should be plenty of space left. The web UI also 
reports:
Admin State       Capacity       Used       Non DFS Used       Remaining Blocks       
Block pool used       Failed Volumes       Version
In Service               44.49 GB   4.59 GB       13.17 GB               26.72 GB 
176852       4.59 GB (10.33%)                0                                     2.7.3

Would be great if somebody could explain why my disk is reported to be full.

thanks,
Hendrik

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org

[prev in list] [next in list] [prev in thread] [next in thread]