[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-commits
Subject: svn commit: r448334 - in /lucene/hadoop/trunk: CHANGES.txt
From: cutting () apache ! org
Date: 2006-09-20 20:42:44
Message-ID: 20060920204244.9F6681A981A () eris ! apache ! org
[Download RAW message or body]
Author: cutting
Date: Wed Sep 20 13:42:43 2006
New Revision: 448334
URL: http://svn.apache.org/viewvc?view=rev&rev=448334
Log:
HADOOP-261. Record an error message when map outputs are lost. Contributed by Owen.
Modified:
lucene/hadoop/trunk/CHANGES.txt
lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java
lucene/hadoop/trunk/src/webapps/task/getMapOutput.jsp
Modified: lucene/hadoop/trunk/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/CHANGES.txt?view=diff&rev=448334&r1=448333&r2=448334
==============================================================================
--- lucene/hadoop/trunk/CHANGES.txt (original)
+++ lucene/hadoop/trunk/CHANGES.txt Wed Sep 20 13:42:43 2006
@@ -3,35 +3,38 @@
Trunk (unreleased changes)
-1. HADOOP-243. Fix rounding in the display of task and job progress
- so that things are not shown to be 100% complete until they are in
- fact finished. (omalley via cutting)
+ 1. HADOOP-243. Fix rounding in the display of task and job progress
+ so that things are not shown to be 100% complete until they are in
+ fact finished. (omalley via cutting)
-2. HADOOP-438. Limit the length of absolute paths in DFS, since the
- file format used to store pathnames has some limitations.
- (Wendy Chien via cutting)
+ 2. HADOOP-438. Limit the length of absolute paths in DFS, since the
+ file format used to store pathnames has some limitations.
+ (Wendy Chien via cutting)
-3. HADOOP-530. Improve error messages in SequenceFile when keys or
- values are of the wrong type. (Hairong Kuang via cutting)
+ 3. HADOOP-530. Improve error messages in SequenceFile when keys or
+ values are of the wrong type. (Hairong Kuang via cutting)
-4. HADOOP-288. Add a file caching system and use it in MapReduce to
- cache job jar files on slave nodes. (Mahadev Konar via cutting)
+ 4. HADOOP-288. Add a file caching system and use it in MapReduce to
+ cache job jar files on slave nodes. (Mahadev Konar via cutting)
-5. HADOOP-533. Fix unit test to not modify conf directory.
- (Hairong Kuang via cutting)
+ 5. HADOOP-533. Fix unit test to not modify conf directory.
+ (Hairong Kuang via cutting)
-6. HADOOP-527. Permit specification of the local address that various
- Hadoop daemons should bind to. (Philippe Gassmann via cutting)
+ 6. HADOOP-527. Permit specification of the local address that various
+ Hadoop daemons should bind to. (Philippe Gassmann via cutting)
-7. HADOOP-542. Updates to contrib/streaming: reformatted source code,
- on-the-fly merge sort, a fix for HADOOP-540, etc.
- (Michel Tourn via cutting)
+ 7. HADOOP-542. Updates to contrib/streaming: reformatted source code,
+ on-the-fly merge sort, a fix for HADOOP-540, etc.
+ (Michel Tourn via cutting)
-8. HADOOP-545. Remove an unused config file parameter.
- (Philippe Gassmann via cutting)
+ 8. HADOOP-545. Remove an unused config file parameter.
+ (Philippe Gassmann via cutting)
-9. HADOOP-548. Add an Ant property "test.output" to build.xml that
- causes test output to be logged to the console. (omalley via cutting)
+ 9. HADOOP-548. Add an Ant property "test.output" to build.xml that
+ causes test output to be logged to the console. (omalley via cutting)
+
+10. HADOOP-261. Record an error message when map output is lost.
+ (omalley via cutting)
Release 0.6.2 (unreleased)
Modified: lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java
URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java?view=diff&rev=448334&r1=448333&r2=448334
==============================================================================
--- lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java (original)
+++ lucene/hadoop/trunk/src/java/org/apache/hadoop/mapred/TaskTracker.java Wed Sep 20 \
13:42:43 2006 @@ -357,7 +357,7 @@
this.mapOutputFile = new MapOutputFile();
this.mapOutputFile.setConf(conf);
int httpPort = conf.getInt("tasktracker.http.port", 50060);
- String httpBindAddress = conf.get("tasktracker.http.bindAddress", "0.0.0.0");;
+ String httpBindAddress = conf.get("tasktracker.http.bindAddress", "0.0.0.0");
this.server = new StatusHttpServer("task", httpBindAddress, httpPort, true);
int workerThreads = conf.getInt("tasktracker.http.threads", 40);
server.setThreads(1, workerThreads);
@@ -990,11 +990,14 @@
/**
* The map output has been lost.
*/
- public synchronized void mapOutputLost() throws IOException {
+ public synchronized void mapOutputLost(String failure
+ ) throws IOException {
if (runstate == TaskStatus.SUCCEEDED) {
LOG.info("Reporting output lost:"+task.getTaskId());
runstate = TaskStatus.FAILED; // change status to failure
progress = 0.0f;
+ reportDiagnosticInfo("Map output lost, rescheduling: " +
+ failure);
runningTasks.put(task.getTaskId(), this);
mapTotal++;
} else {
@@ -1120,10 +1123,11 @@
/**
* A completed map task's output has been lost.
*/
- public synchronized void mapOutputLost(String taskid) throws IOException {
+ public synchronized void mapOutputLost(String taskid,
+ String errorMsg) throws IOException {
TaskInProgress tip = (TaskInProgress) tasks.get(taskid);
if (tip != null) {
- tip.mapOutputLost();
+ tip.mapOutputLost(errorMsg);
} else {
LOG.warn("Unknown child with bad map output: "+taskid+". Ignored.");
}
Modified: lucene/hadoop/trunk/src/webapps/task/getMapOutput.jsp
URL: http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/webapps/task/getMapOutput.jsp?view=diff&rev=448334&r1=448333&r2=448334
==============================================================================
--- lucene/hadoop/trunk/src/webapps/task/getMapOutput.jsp (original)
+++ lucene/hadoop/trunk/src/webapps/task/getMapOutput.jsp Wed Sep 20 13:42:43 2006
@@ -40,9 +40,10 @@
TaskTracker tracker =
(TaskTracker) application.getAttribute("task.tracker");
Log log = (Log) application.getAttribute("log");
- log.warn("Http server (getMapOutput.jsp): " +
- StringUtils.stringifyException(ie));
- tracker.mapOutputLost(mapId);
+ String errorMsg = "getMapOutput(" + mapId + "," + reduceId + ") failed : "+
+ StringUtils.stringifyException(ie);
+ log.warn(errorMsg);
+ tracker.mapOutputLost(mapId, errorMsg);
throw ie;
} finally {
if (inStream != null) {
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic