[prev in list] [next in list] [prev in thread] [next in thread] 

List:       hadoop-dev
Subject:    [jira] Commented: (HADOOP-5361) Tasks freeze with "No live nodes
From:       "Matei Zaharia (JIRA)" <jira () apache ! org>
Date:       2009-02-28 2:43:13
Message-ID: 1950432722.1235788993024.JavaMail.jira () brutus
[Download RAW message or body]


    [ https://issues.apache.org/jira/browse/HADOOP-5361?page=com.atlassian.jira.plugin \
.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677631#action_12677631 ] 

Matei Zaharia commented on HADOOP-5361:
---------------------------------------

I think the root cause of this was me using values that were too small for \
dfs.namenode.handler.count, dfs.datanode.max.xcievers and \
ipc.client.connection.maxidletime. After tweaking those I was able to prevent the \
error from happening. However, I still think the way this was handled is wrong.. you \
should be allowed to speculate tasks that fail to load up for this reason, rather \
than having them hang for 8 minutes.

> Tasks freeze with "No live nodes contain current block", job takes long time to \
>                 recover
> ---------------------------------------------------------------------------------------
>  
> Key: HADOOP-5361
> URL: https://issues.apache.org/jira/browse/HADOOP-5361
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.21.0
> Reporter: Matei Zaharia
> 
> Running a recent version of trunk on 100 nodes, I occasionally see some tasks \
> freeze at startup and hang the job. These tasks are not speculatively executed \
> either. Here's sample output from one of them: {noformat}
> 009-02-27 15:19:09,856 WARN org.apache.hadoop.conf.Configuration: DEPRECATED: \
> hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. \
> Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties \
> of core-default.xml, mapred-default.xml and hdfs-default.xml respectively \
> 2009-02-27 15:19:10,229 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: Initializing \
> JVM Metrics with processName=MAP, sessionId= 2009-02-27 15:19:10,486 INFO \
> org.apache.hadoop.mapred.MapTask: numReduceTasks: 0 2009-02-27 15:21:20,952 INFO \
> org.apache.hadoop.hdfs.DFSClient: Could not obtain block \
> blk_2086525142250101885_39076 from any node:  java.io.IOException: No live nodes \
> contain current block 2009-02-27 15:23:23,972 INFO \
> org.apache.hadoop.hdfs.DFSClient: Could not obtain block \
> blk_2086525142250101885_39076 from any node:  java.io.IOException: No live nodes \
> contain current block 2009-02-27 15:25:26,992 INFO \
> org.apache.hadoop.hdfs.DFSClient: Could not obtain block \
> blk_2086525142250101885_39076 from any node:  java.io.IOException: No live nodes \
> contain current block 2009-02-27 15:27:30,012 WARN \
> org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain \
> block: blk_2086525142250101885_39076 file=/user/root/rand2/part-00864 at \
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1664) \
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1492) \
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1619) at \
> java.io.DataInputStream.read(DataInputStream.java:83) at \
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134) at \
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136) at \
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40) at \
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:191) \
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:175) at \
> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) at \
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354) at \
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at \
> org.apache.hadoop.mapred.Child.main(Child.java:155) 2009-02-27 15:27:30,018 WARN \
>                 org.apache.hadoop.mapred.TaskTracker: Error running child
> java.io.IOException: Could not obtain block: blk_2086525142250101885_39076 \
> file=/user/root/rand2/part-00864 at \
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1664) \
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1492) \
> at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1619) at \
> java.io.DataInputStream.read(DataInputStream.java:83) at \
> org.apache.hadoop.util.LineReader.readLine(LineReader.java:134) at \
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:136) at \
> org.apache.hadoop.mapred.LineRecordReader.next(LineRecordReader.java:40) at \
> org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:191) \
> at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:175) at \
> org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:48) at \
> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:354) at \
> org.apache.hadoop.mapred.MapTask.run(MapTask.java:305) at \
> org.apache.hadoop.mapred.Child.main(Child.java:155) {noformat}
> Note how the DFS client fails multiple times to retrieve the block, with a 2 minute \
> wait between each one, without giving up. During this time, the task is *not* \
> speculated. However, once this task finally failed, a new version of it ran \
> successfully. Getting the input file in question with bin/hadoop fs -get also \
> worked fine. There is no mention of the task attempt in question in the NameNode \
> logs but my guess is that something to do with RPC queues is causing its connection \
> to get lost, and the DFSClient does not recover.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[prev in list] [next in list] [prev in thread] [next in thread]