[prev in list] [next in list] [prev in thread] [next in thread]
List: hadoop-user
Subject: Re: Too many fetch failures. Help!
From: Abdelrahman Kamel <abdoukamel () gmail ! com>
Date: 2011-09-28 8:57:00
Message-ID: CA+xnphW1_i1ORCxsLZthCGbDHgwd3F6FgCP1uv6WrCQzhEMqZQ () mail ! gmail ! com
[Download RAW message or body]
Thanks again.
I have solved my problem by commenting out "127.0.1.1 hdslave" from
/etc/hosts in all nodes.
On Wed, Sep 28, 2011 at 10:15 AM, praveenesh kumar <praveenesh@gmail.com>wrote:
> Try commenting out "127.0.0.1 localhost" line from /etc/hosts in all our
> systems.
>
>
> On Tue, Sep 27, 2011 at 11:59 PM, Abdelrahman Kamel <abdoukamel@gmail.com
> >wrote:
>
> > Thanks very much for all your fast replies.
> >
> > *Here is my terminal output:*
> >
> > hduser@hdmaster:/usr/local/hadoop$ bin/hadoop jar
> > hadoop-0.20.2-examples.jar
> > wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output
> > 11/09/28 07:38:23 INFO input.FileInputFormat: Total input paths to
> process
> > :
> > 5
> > 11/09/28 07:38:23 INFO mapred.JobClient: Running job:
> job_201109280735_0002
> > 11/09/28 07:38:24 INFO mapred.JobClient: map 0% reduce 0%
> > 11/09/28 07:38:42 INFO mapred.JobClient: map 20% reduce 0%
> > 11/09/28 07:38:44 INFO mapred.JobClient: map 40% reduce 0%
> > 11/09/28 07:38:45 INFO mapred.JobClient: map 60% reduce 0%
> > 11/09/28 07:38:47 INFO mapred.JobClient: map 80% reduce 0%
> > 11/09/28 07:38:51 INFO mapred.JobClient: map 100% reduce 0%
> > 11/09/28 07:38:54 INFO mapred.JobClient: map 100% reduce 13%
> > 11/09/28 07:39:01 INFO mapred.JobClient: map 100% reduce 20%
> >
> > *The terminal is stuck here.*
> >
> > *And here is my JobTracker log:*
> >
> > 2011-09-28 07:35:43,185 INFO org.apache.hadoop.mapred.JobTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting JobTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:43,256 INFO org.apache.hadoop.mapred.JobTracker:
> Scheduler
> > configured with (memSizeForMapSlotOnJT, memSizeForReduceSlotOnJT,
> > limitMaxMemForMapTasks, limitMaxMemForReduceTasks) (-1, -1, -1, -1)
> > 2011-09-28 07:35:43,310 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=JobTracker, port=54311
> > 2011-09-28 07:35:53,431 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:53,510 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50030
> > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50030
> > webServer.getConnectors()[0].getLocalPort() returned 50030
> > 2011-09-28 07:35:53,511 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50030
> > 2011-09-28 07:35:53,511 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:53,816 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50030
> > 2011-09-28 07:35:53,817 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=JobTracker, sessionId=
> > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker
> > up at: 54311
> > 2011-09-28 07:35:53,818 INFO org.apache.hadoop.mapred.JobTracker:
> > JobTracker
> > webserver: 50030
> > 2011-09-28 07:35:53,926 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:35:53,930 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> > Safe mode will be turned off automatically.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:03,934 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:03,935 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 0.0000 has not reached the threshold 0.9990.
> > Safe mode will be turned off automatically.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:13,942 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:13,944 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 25 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:23,947 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:23,949 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 15 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:36:33,952 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:36:33,954 INFO org.apache.hadoop.mapred.JobTracker: problem
> > cleaning system directory:
> hdfs://master:54310/app/hadoop/tmp/mapred/system
> > org.apache.hadoop.ipc.RemoteException:
> > org.apache.hadoop.hdfs.server.namenode.SafeModeException: Cannot delete
> > /app/hadoop/tmp/mapred/system. Name node is in safe mode.
> > The ratio of reported blocks 1.0000 has reached the threshold 0.9990.
> Safe
> > mode will be turned off automatically in 5 seconds.
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.deleteInternal(FSNamesystem.java:1700)
> > at
> >
> >
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.delete(FSNamesystem.java:1680)
> > at
> > org.apache.hadoop.hdfs.server.namenode.NameNode.delete(NameNode.java:517)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:959)
> > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:955)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at javax.security.auth.Subject.doAs(Subject.java:396)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:953)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:740)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
> > at $Proxy4.delete(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at $Proxy4.delete(Unknown Source)
> > at org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:582)
> > at
> >
> >
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:227)
> > at org.apache.hadoop.mapred.JobTracker.<init>(JobTracker.java:1695)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:183)
> > at
> org.apache.hadoop.mapred.JobTracker.startTracker(JobTracker.java:175)
> > at org.apache.hadoop.mapred.JobTracker.main(JobTracker.java:3702)
> > 2011-09-28 07:37:09,922 INFO org.apache.hadoop.mapred.JobTracker:
> Cleaning
> > up the system directory
> > 2011-09-28 07:37:10,107 INFO
> > org.apache.hadoop.mapred.CompletedJobStatusStore: Completed job store is
> > inactive
> > 2011-09-28 07:37:11,461 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54311: starting
> > 2011-09-28 07:37:11,462 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54311: starting
> > 2011-09-28 07:37:11,463 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 4 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 6 on 54311: starting
> > 2011-09-28 07:37:11,470 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 7 on 54311: starting
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.mapred.JobTracker:
> Starting
> > RUNNING
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 5 on 54311: starting
> > 2011-09-28 07:37:11,472 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 8 on 54311: starting
> > 2011-09-28 07:37:11,477 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 9 on 54311: starting
> > 2011-09-28 07:37:11,577 INFO org.apache.hadoop.net.NetworkTopology:
> Adding
> > a
> > new node: /default-rack/hdmaster
> > 2011-09-28 07:37:43,188 INFO org.apache.hadoop.net.NetworkTopology:
> Adding
> > a
> > new node: /default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:23,832 INFO org.apache.hadoop.mapred.JobTracker:
> > Initializing job_201109280735_0002
> > 2011-09-28 07:38:23,838 INFO org.apache.hadoop.mapred.JobInProgress:
> > Initializing job_201109280735_0002
> > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
> Input
> > size for job job_201109280735_0002 = 3430958. Number of splits = 5
> > 2011-09-28 07:38:24,013 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000000 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000000 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000001 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000001 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,280 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000002 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000002 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000003 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000003 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000004 has split on
> > node:/default-rack/hdmaster
> > 2011-09-28 07:38:34,281 INFO org.apache.hadoop.mapred.JobInProgress:
> > tip:task_201109280735_0002_m_000004 has split on
> > node:/default-rack/hdslave.ubuntu-domain
> > 2011-09-28 07:38:35,674 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000006_0' to tip
> > task_201109280735_0002_m_000006, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,710 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000006_0' has completed
> > task_201109280735_0002_m_000006 successfully.
> > 2011-09-28 07:38:38,714 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000000_0' to tip
> > task_201109280735_0002_m_000000, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000000
> > 2011-09-28 07:38:38,717 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000001_0' to tip
> > task_201109280735_0002_m_000001, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:38,718 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000001
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000002_0' to tip
> > task_201109280735_0002_m_000002, for tracker 'tracker_hdmaster:localhost/
> > 127.0.0.1:54873'
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000002
> > 2011-09-28 07:38:40,233 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000003_0' to tip
> > task_201109280735_0002_m_000003, for tracker 'tracker_hdmaster:localhost/
> > 127.0.0.1:54873'
> > 2011-09-28 07:38:40,234 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000003
> > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000000_0' has completed
> > task_201109280735_0002_m_000000 successfully.
> > 2011-09-28 07:38:41,724 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:1 completedMapsInputSize:1423802
> > completedMapsOutputSize:466814
> > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000004_0' to tip
> > task_201109280735_0002_m_000004, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:41,726 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000004
> > 2011-09-28 07:38:41,730 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_r_000000_0' to tip
> > task_201109280735_0002_r_000000, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000003_0' has completed
> > task_201109280735_0002_m_000003 successfully.
> > 2011-09-28 07:38:43,272 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:2 completedMapsInputSize:1817766
> > completedMapsOutputSize:719434
> > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000001_0' has completed
> > task_201109280735_0002_m_000001 successfully.
> > 2011-09-28 07:38:44,746 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:3 completedMapsInputSize:2492333
> > completedMapsOutputSize:986447
> > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000002_0' has completed
> > task_201109280735_0002_m_000002 successfully.
> > 2011-09-28 07:38:46,284 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:4 completedMapsInputSize:3087267
> > completedMapsOutputSize:1204101
> > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000004_0' has completed
> > task_201109280735_0002_m_000004 successfully.
> > 2011-09-28 07:38:50,874 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:5 completedMapsInputSize:3430963
> > completedMapsOutputSize:1367489
> > 2011-09-28 07:39:06,502 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #1 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:41:53,093 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #2 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:47:06,741 INFO org.apache.hadoop.mapred.JobInProgress:
> Failed
> > fetch notification #3 for task attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.JobInProgress: Too
> > many fetch-failures for output of task:
> > attempt_201109280735_0002_m_000003_0
> > ... killing it
> > 2011-09-28 07:47:06,742 INFO org.apache.hadoop.mapred.TaskInProgress:
> Error
> > from attempt_201109280735_0002_m_000003_0: Too many fetch-failures
> > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobTracker: Adding
> > task 'attempt_201109280735_0002_m_000003_1' to tip
> > task_201109280735_0002_m_000003, for tracker
> > 'tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342'
> > 2011-09-28 07:47:06,743 INFO org.apache.hadoop.mapred.JobInProgress:
> > Choosing data-local task task_201109280735_0002_m_000003
> > 2011-09-28 07:47:07,607 INFO org.apache.hadoop.mapred.JobTracker: Removed
> > completed task 'attempt_201109280735_0002_m_000003_0' from
> > 'tracker_hdmaster:localhost/127.0.0.1:54873'
> > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.JobInProgress: Task
> > 'attempt_201109280735_0002_m_000003_1' has completed
> > task_201109280735_0002_m_000003 successfully.
> > 2011-09-28 07:47:13,361 INFO org.apache.hadoop.mapred.ResourceEstimator:
> > completedMapsUpdates:6 completedMapsInputSize:3824927
> > completedMapsOutputSize:1620109
> >
> > *And here is my TaskTracker #1 log:*
> >
> > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50060
> > webServer.getConnectors()[0].getLocalPort() returned 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50060
> > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50060
> > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=TaskTracker, sessionId=
> > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=TaskTracker, port=54873
> > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker up at: localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > tracker tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54873: starting
> > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54873: starting
> > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_1909679810 spawned.
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-622388657 spawned.
> > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_1909679810 given task:
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-622388657 given task:
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000003_0 1.0%
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000003_0 is done.
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000003_0 was 0
> > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000002_0 1.0%
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000002_0 is done.
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000002_0 was 0
> > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
> >
> >
> > *And here is my TaskTracker #2 log:*
> >
> > 2011-09-28 07:35:44,452 INFO org.apache.hadoop.mapred.TaskTracker:
> > STARTUP_MSG:
> > /************************************************************
> > STARTUP_MSG: Starting TaskTracker
> > STARTUP_MSG: host = hdmaster/127.0.1.1
> > STARTUP_MSG: args = []
> > STARTUP_MSG: version = 0.20.2
> > STARTUP_MSG: build =
> > https://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20 -r
> > 911707; compiled by 'chrisdo' on Fri Feb 19 08:07:34 UTC 2010
> > ************************************************************/
> > 2011-09-28 07:35:54,661 INFO org.mortbay.log: Logging to
> > org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via
> > org.mortbay.log.Slf4jLog
> > 2011-09-28 07:35:54,744 INFO org.apache.hadoop.http.HttpServer: Port
> > returned by webServer.getConnectors()[0].getLocalPort() before open() is
> > -1.
> > Opening the listener on 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer:
> > listener.getLocalPort() returned 50060
> > webServer.getConnectors()[0].getLocalPort() returned 50060
> > 2011-09-28 07:35:54,750 INFO org.apache.hadoop.http.HttpServer: Jetty
> bound
> > to port 50060
> > 2011-09-28 07:35:54,750 INFO org.mortbay.log: jetty-6.1.14
> > 2011-09-28 07:35:55,025 INFO org.mortbay.log: Started
> > SelectChannelConnector@0.0.0.0:50060
> > 2011-09-28 07:35:55,064 INFO org.apache.hadoop.metrics.jvm.JvmMetrics:
> > Initializing JVM Metrics with processName=TaskTracker, sessionId=
> > 2011-09-28 07:35:55,082 INFO org.apache.hadoop.ipc.metrics.RpcMetrics:
> > Initializing RPC Metrics with hostName=TaskTracker, port=54873
> > 2011-09-28 07:35:55,127 INFO org.apache.hadoop.ipc.Server: IPC Server
> > Responder: starting
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker up at: localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,129 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > tracker tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 0 on 54873: starting
> > 2011-09-28 07:35:55,131 INFO org.apache.hadoop.ipc.Server: IPC Server
> > listener on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 1 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 2 on 54873: starting
> > 2011-09-28 07:35:55,128 INFO org.apache.hadoop.ipc.Server: IPC Server
> > handler 3 on 54873: starting
> > 2011-09-28 07:37:11,483 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdmaster:localhost/127.0.0.1:54873
> > 2011-09-28 07:37:11,484 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@5a3923
> > 2011-09-28 07:37:11,487 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:37:11,488 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:38:40,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000002_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000003_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,271 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,481 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:40,538 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_1909679810
> > 2011-09-28 07:38:40,540 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_1909679810 spawned.
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-622388657
> > 2011-09-28 07:38:40,576 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-622388657 spawned.
> > 2011-09-28 07:38:41,046 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_1909679810 given task:
> > attempt_201109280735_0002_m_000002_0
> > 2011-09-28 07:38:41,099 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-622388657 given task:
> > attempt_201109280735_0002_m_000003_0
> > 2011-09-28 07:38:43,264 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000003_0 1.0%
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000003_0 is done.
> > 2011-09-28 07:38:43,265 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000003_0 was 0
> > 2011-09-28 07:38:43,266 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000002_0 1.0%
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000002_0 is done.
> > 2011-09-28 07:38:43,317 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000002_0 was 0
> > 2011-09-28 07:38:43,322 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:38:43,362 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_1909679810 exited. Number of tasks it ran: 1
> > 2011-09-28 07:38:43,412 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-622388657 exited. Number of tasks it ran: 1
> > 2011-09-28 07:36:10,343 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/10.0.0.1:54311. Already tried 0 time(s).
> > 2011-09-28 07:36:31,346 INFO org.apache.hadoop.ipc.Client: Retrying
> connect
> > to server: master/10.0.0.1:54311. Already tried 1 time(s).
> > 2011-09-28 07:36:31,581 INFO org.apache.hadoop.mapred.TaskTracker:
> Starting
> > thread: Map-events fetcher for all reduce tasks on
> > tracker_hdslave.ubuntu-domain:localhost/127.0.0.1:49342
> > 2011-09-28 07:36:31,583 INFO org.apache.hadoop.mapred.TaskTracker: Using
> > MemoryCalculatorPlugin :
> > org.apache.hadoop.util.LinuxMemoryCalculatorPlugin@ce16ad
> > 2011-09-28 07:36:31,588 WARN org.apache.hadoop.mapred.TaskTracker:
> > TaskTracker's totalMemoryAllottedForTasks is -1. TaskMemoryManager is
> > disabled.
> > 2011-09-28 07:36:31,589 INFO org.apache.hadoop.mapred.IndexCache:
> > IndexCache
> > created with max memory = 10485760
> > 2011-09-28 07:37:35,032 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000006_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:35,034 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_774270481
> > 2011-09-28 07:37:35,374 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_774270481 spawned.
> > 2011-09-28 07:37:35,773 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_774270481 given task:
> > attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:36,667 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000006_0 0.0% setup
> > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000006_0 is done.
> > 2011-09-28 07:37:36,668 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000006_0 was 0
> > 2011-09-28 07:37:36,670 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:37:36,818 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_774270481 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:38,033 INFO org.apache.hadoop.mapred.TaskTracker:
> > org.apache.hadoop.util.DiskChecker$DiskErrorException: Could not find
> >
> >
> taskTracker/jobcache/job_201109280735_0002/attempt_201109280735_0002_m_000006_0/output/file.out
> > in any of the configured local directories
> > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction
> > (registerTask): attempt_201109280735_0002_m_000000_0 task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:38,045 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000001_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker:
> Received
> > KillTaskAction for task: attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskTracker: About
> to
> > purge task: attempt_201109280735_0002_m_000006_0
> > 2011-09-28 07:37:38,046 INFO org.apache.hadoop.mapred.TaskRunner:
> > attempt_201109280735_0002_m_000006_0 done; removing files.
> > 2011-09-28 07:37:38,047 INFO org.apache.hadoop.mapred.IndexCache: Map ID
> > attempt_201109280735_0002_m_000006_0 not found in cache
> > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,080 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,082 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-1878969571
> > 2011-09-28 07:37:38,086 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-1878969571 spawned.
> > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_-364970930
> > 2011-09-28 07:37:38,129 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_-364970930 spawned.
> > 2011-09-28 07:37:38,590 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-364970930 given task:
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:38,604 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_-1878969571 given task:
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:40,644 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000000_0 1.0%
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000000_0 is done.
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000000_0 was 0
> > 2011-09-28 07:37:40,645 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:37:40,697 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-1878969571 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_m_000004_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:41,067 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 1 and trying to launch
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker:
> > LaunchTaskAction (registerTask): attempt_201109280735_0002_r_000000_0
> > task's
> > state:UNASSIGNED
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: Trying
> > to
> > launch : attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,068 INFO org.apache.hadoop.mapred.TaskTracker: In
> > TaskLauncher, current free slots : 2 and trying to launch
> > attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_m_531367236
> > 2011-09-28 07:37:41,118 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_m_531367236 spawned.
> > 2011-09-28 07:37:41,165 INFO org.apache.hadoop.mapred.JvmManager: In
> > JvmRunner constructed JVM ID: jvm_201109280735_0002_r_774270481
> > 2011-09-28 07:37:41,166 INFO org.apache.hadoop.mapred.JvmManager: JVM
> > Runner
> > jvm_201109280735_0002_r_774270481 spawned.
> > 2011-09-28 07:37:41,635 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_r_774270481 given task:
> > attempt_201109280735_0002_r_000000_0
> > 2011-09-28 07:37:41,660 INFO org.apache.hadoop.mapred.TaskTracker: JVM
> with
> > ID: jvm_201109280735_0002_m_531367236 given task:
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:43,417 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000001_0 1.0%
> > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000001_0 is done.
> > 2011-09-28 07:37:43,418 INFO org.apache.hadoop.mapred.TaskTracker:
> reported
> > output size for attempt_201109280735_0002_m_000001_0 was 0
> > 2011-09-28 07:37:43,419 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 1
> > 2011-09-28 07:37:43,456 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_-364970930 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:46,712 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_m_000004_0 1.0%
> > 2011-09-28 07:37:46,713 INFO org.apache.hadoop.mapred.TaskTracker: Task
> > attempt_201109280735_0002_m_000004_0 is done.2011-09-28 07:37:46,713 INFO
> > org.apache.hadoop.mapred.TaskTracker: reported output size for
> > attempt_201109280735_0002_m_000004_0 was 0
> > 2011-09-28 07:37:46,714 INFO org.apache.hadoop.mapred.TaskTracker:
> > addFreeSlot : current free slots : 2
> > 2011-09-28 07:37:46,847 INFO org.apache.hadoop.mapred.JvmManager: JVM :
> > jvm_201109280735_0002_m_531367236 exited. Number of tasks it ran: 1
> > 2011-09-28 07:37:47,276 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 466814 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000000_0
> > given 466814/466810
> > 2011-09-28 07:37:47,276 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52662, bytes: 466814, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000000_0
> > 2011-09-28 07:37:47,287 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 267013 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000001_0
> > given 267013/267009
> > 2011-09-28 07:37:47,287 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52663, bytes: 267013, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000001_0
> > 2011-09-28 07:37:48,128 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> > 0.12 MB/s) >
> > 2011-09-28 07:37:51,130 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.13333334% reduce > copy (2 of 5 at
> > 0.12 MB/s) >
> > 2011-09-28 07:37:51,176 INFO org.apache.hadoop.mapred.TaskTracker: Sent
> out
> > 163388 bytes for reduce: 0 from map: attempt_201109280735_0002_m_000004_0
> > given 163388/163384
> > 2011-09-28 07:37:51,176 INFO
> > org.apache.hadoop.mapred.TaskTracker.clienttrace: src: 127.0.1.1:50060,
> > dest: 127.0.0.1:52664, bytes: 163388, op: MAPRED_SHUFFLE, cliID:
> > attempt_201109280735_0002_m_000004_0
> > 2011-09-28 07:37:57,132 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:03,135 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:06,136 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:18,140 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:21,142 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:27,144 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:33,146 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:36,148 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >2011-09-28 07:38:42,150 INFO
> > org.apache.hadoop.mapred.TaskTracker:
> attempt_201109280735_0002_r_000000_0
> > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> > 2011-09-28 07:38:48,153 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:51,163 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:38:57,166 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:03,168 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:06,170 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:12,172 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:18,175 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:21,176 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:27,178 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:33,181 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:36,183 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:42,185 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:48,187 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:51,189 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:39:57,191 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:03,193 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:06,195 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:12,197 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:18,199 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:21,201 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0
> > 0.20000002% reduce > copy (3 of 5 at 0.09 MB/s) >
> > 2011-09-28 07:40:27,203 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:33,205 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:36,206 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:42,209 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:51,212 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:40:57,214 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:03,217 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:06,218 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:12,221 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:18,223 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:21,225 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:27,227 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:33,229 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:36,230 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:42,233 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:48,238 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:51,240 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:41:57,242 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:03,244 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:06,246 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:12,248 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:18,250 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:21,252 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:27,254 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:33,256 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:36,258 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:42,261 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:48,263 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:51,265 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:42:57,267 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:03,269 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:06,271 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:12,273 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:18,275 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:27,278 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:33,281 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:36,282 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:42,285 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:48,287 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:51,289 INFO org.apache.hadoop.mapred.TaskTracker:
> > attempt_201109280735_0002_r_000000_0 0.20000002% reduce > copy (3 of 5 at
> > 0.09 MB/s) >
> > 2011-09-28 07:43:57,291 INFO org.apache.hadoop.mapred.TaskTracker:
> >
> >
> > *Great thanks hoping you could help me.*
> >
> >
> > On Mon, Sep 26, 2011 at 6:39 PM, Devaraj k <devaraj.k@huawei.com> wrote:
> >
> > > Hi Bharath,
> > >
> > > There are few reasons to cause this problem. I have listed below some
> > > reasons with solutions. This might help you to solve this. If you post
> > the
> > > logs, the problem can be figured out.
> > >
> > >
> > > Reason 1:
> > > It could be that the mapping in the /etc/hosts file is not present.
> > > The DNS server is down as a result of which the hostnames cannot be
> > > resolved.
> > > The DNS server is in-correctly configured.
> > >
> > > Solution: Setting the slave.host.name property can be one solution.
> > > Appropriate changes need to be done based on the problem.
> > >
> > >
> > > Reason 2: If the map outputs are larger, we may get
> > > java.lang.OutOfMemoryError: Java heap space. Because of this there are
> > too
> > > many fetch failures.
> > >
> > > Solution: The error, java.lang.OutOfMemoryError: Java heap space in
> task
> > > tracker logs can be solved by any of the following methods:
> > >
> > > By decreasing the value configured for
> > > mapred.job.shuffle.input.buffer.percent.
> > > By increasing the heap memory of child JVM options for the property
> > > mapred.child.java.opts.
> > >
> > >
> > >
> > >
> > > Thanks
> > > Devaraj
> > >
> > > ________________________________________
> > > From: bharath vissapragada [bharathvissapragada1990@gmail.com]
> > > Sent: Monday, September 26, 2011 8:54 PM
> > > To: common-user@hadoop.apache.org
> > > Subject: Re: Too many fetch failures. Help!
> > >
> > > Hey,
> > >
> > > Try configuring your cluster with hostnames instead of ips and add
> > > those entries to /etc/hosts and sync it across all the nodes in the
> > > cluster. You need to restart the cluster after making these changes.
> > >
> > > Hope this helps,
> > >
> > > On Mon, Sep 26, 2011 at 8:46 PM, Abdelrahman Kamel <
> abdoukamel@gmail.com
> > >
> > > wrote:
> > > > Hi,
> > > > This is my first post here.
> > > > I'm new to Hadoop.
> > > > I've already installed Hadoop on 2 Ubuntu boxes (one is both master
> and
> > > > slave and the other is only slave).
> > > > When I run a Wordcount example on 5 small txt files, the process
> never
> > > > completes and I get a "Too many fetch failures" error on my terminal.
> > > > If you can help me, I cant post my terminal's output and any log
> files
> > > > needed.
> > > > Great thanks.
> > > >
> > > > --
> > > > Abdelrahman Kamel
> > > >
> > >
> > >
> > >
> > > --
> > > Regards,
> > > Bharath .V
> > > w:http://researchweb.iiit.ac.in/~bharath.v
> > >
> >
> >
> >
> > --
> > Abdelrahman Kamel
> >
>
--
Abdelrahman Kamel
[prev in list] [next in list] [prev in thread] [next in thread]
Configure |
About |
News |
Add a list |
Sponsored by KoreLogic