[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-user
Subject:    Re: DFS get blocked when writing a file.
From:       Raghu Angadi <rangadi () yahoo-inc ! com>
Date:       2008-03-31 18:11:10
Message-ID: 47F1293E.3070706 () yahoo-inc ! com
[Download RAW message or body]

Iván,

Whether this was expected or an error depends on what happened on the 
client. This could happen and would not be a bug if client was killed 
for some other reason for e.g. But if client is also similarly surprised 
  then its a different case.

You could grep for this block in NameNode log and client. If you are 
still interested in looking into this, I would suggest opening a jira.

Raghu.

Iván de Prado wrote:
> Thanks, 
> 
> I have tried with the trunk version and now the exception "Trying to
> change block file offset of block blk_... to ... but actual size of file
> is ..." has disappeared and the jobs don't seems to get blocked.
> 
> But I have another "Broken Pipe" and "EOF" exceptions in the dfs logs.
> They seems similar to https://issues.apache.org/jira/browse/HADOOP-2042
> ticket. The Jobs ends but not sure if they are executed smoothly. are
> these exceptions normal? As example, the exceptions for the block
> (6801211507359331627) appears in two nodes (I have 2 as replication) and
> looks like:
> 
> hn2: 2008-03-31 05:03:13,736 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 0 forwarding connect ack to upstream firstbadlink is 
> hn2: 2008-03-31 05:03:14,507 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_6801211507359331627 src: /172.16.3.6:38218
> dest: /172.16.3.6:50010
> 
> hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> Exception in receiveBlock for block blk_6801211507359331627
> java.io.EOFException
> hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_6801211507359331627 Interrupted.
> hn2: 2008-03-31 05:04:14,528 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 0 for block blk_6801211507359331627 terminating
> hn2: 2008-03-31 05:04:14,530 INFO org.apache.hadoop.dfs.DataNode:
> writeBlock blk_6801211507359331627 received exception
> java.io.EOFException
> hn2: 2008-03-31 05:04:14,530 ERROR org.apache.hadoop.dfs.DataNode:
> 172.16.3.4:50010:DataXceiver: java.io.EOFException
> hn2:    at java.io.DataInputStream.readInt(DataInputStream.java:375)
> hn2:    at org.apache.hadoop.dfs.DataNode
> $BlockReceiver.receiveBlock(DataNode.java:2243)
> hn2:    at org.apache.hadoop.dfs.DataNode
> $DataXceiver.writeBlock(DataNode.java:1157)
> hn2:    at org.apache.hadoop.dfs.DataNode
> $DataXceiver.run(DataNode.java:938)
> hn2:    at java.lang.Thread.run(Thread.java:619)
> 
> hn4: 2008-03-31 05:03:13,590 INFO org.apache.hadoop.dfs.DataNode:
> Datanode 0 forwarding connect ack to upstream firstbadlink is 
> hn4: 2008-03-31 05:03:14,506 INFO org.apache.hadoop.dfs.DataNode:
> Receiving block blk_6801211507359331627 src: /172.16.3.6:41112
> dest: /172.16.3.6:50010
> 
> hn4: 2008-03-31 05:03:26,825 INFO org.apache.hadoop.dfs.DataNode:
> Exception in receiveBlock for block blk_6801211507359331627
> java.io.EOFException
> 
> hn4: 2008-03-31 05:04:14,524 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder blk_6801211507359331627 1 Exception
> java.net.SocketException: Broken pipe
> hn4:    at java.net.SocketOutputStream.socketWrite0(Native Method)
> hn4:    at
> java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
> hn4:    at
> java.net.SocketOutputStream.write(SocketOutputStream.java:136)
> hn4:    at java.io.DataOutputStream.writeLong(DataOutputStream.java:207)
> hn4:    at org.apache.hadoop.dfs.DataNode
> $PacketResponder.run(DataNode.java:1825)
> hn4:    at java.lang.Thread.run(Thread.java:619)
> hn4: 
> hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
> PacketResponder 1 for block blk_6801211507359331627 terminating
> hn4: 2008-03-31 05:04:14,525 INFO org.apache.hadoop.dfs.DataNode:
> writeBlock blk_6801211507359331627 received exception
> java.io.EOFException
> hn4: 2008-03-31 05:04:14,526 ERROR org.apache.hadoop.dfs.DataNode:
> 172.16.3.6:50010:DataXceiver: java.io.EOFException
> hn4:    at java.io.DataInputStream.readInt(DataInputStream.java:375)
> hn4:    at org.apache.hadoop.dfs.DataNode
> $BlockReceiver.receiveBlock(DataNode.java:2243)
> hn4:    at org.apache.hadoop.dfs.DataNode
> $DataXceiver.writeBlock(DataNode.java:1157)
> hn4:    at org.apache.hadoop.dfs.DataNode
> $DataXceiver.run(DataNode.java:938)
> hn4:    at java.lang.Thread.run(Thread.java:619)
> hn4: 
> 
> Many thanks, 
> 
> Iván de Prado Alonso
> http://ivandeprado.blogspot.com/
[prev in list] [next in list] [prev in thread] [next in thread]